Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
If I get the same key/value pairs from 2 different mappers running on 2 different datanodes, and if I am using a single reducer, how can I eliminate the duplicate key/value pair and prevent it from entering the reducer?
Should I use a combiner and then check if there are duplicate values for the same key and then eliminate it in the combiner? But the combiner takes as input all key value pairs from single mapper, right?
It is exactly the duty of reducer – to process such duplication. I think there is no way in hadoop to allow it exactly for this reason.
As you pointed in a right way – combiner will not entirely help here, but only reduce the number of such duplications