Question

Is there a way in Hadoop to ensure that every reducer gets only one key that is output by the mapper ?

Answer 1

This question is a bit unclear for me. But I think I have a pretty good idea what you want.

First of all if you do nothing special every time a reduce is called it gets only one single key with a set of one or more values (via an iterator).

My guess is that you want to ensure that every reducer gets exactly one ‘key-value pair’.
There are essentially two ways of doing that:

1. Ensure in the mapper that all keys that are output are unique. So for each key there is only one value.
2. Force the reducer to do this by forcing a group comparator that simply classifies all keys as different.

So if I understand your question correctly. You should implement a GroupComparator that simply states that all keys are different and should therefor be sent to a different reducer call.

---

Because of other answers in this question I’m adding a bit more detail:

There are 3 methods used for comparing keys (I pulled these code samples from a project I did using the 0.18.3 API):

Partitioner

    conf.setPartitionerClass(KeyPartitioner.class);

The partitioner is only to ensure that “things that must be the same end up on the same partition”. If you have 1 computer there is only one partition, so this won’t help much.

Key Comparator

    conf.setOutputKeyComparatorClass(KeyComparator.class);

The key comparator is used to SORT the “key-value pairs” in a group by looking at the key … which must be different somehow.

Group Comparator

    conf.setOutputValueGroupingComparator(GroupComparator.class);

The group comparator is used to group keys that are different, yet must be sent o the same reducer.

HTH