Question

After map and before reduce, there is a sort phase. In streaming mode, how does hadoop know what you key type is, and sort it.

For example

Input file format

1990    1
1991    4
1992    5
...

The result of map has keys 1990,1991,1992…, how hadoop sorts it? (numberic sort or alphabetical sort)

Answer 1

In the StreamJob, the map output key/value pairs are specified as

idResolver.resolve(jobConf_.get("stream.map.output",
  IdentifierResolver.TEXT_ID));
jobConf_.setClass("stream.map.output.reader.class",
  idResolver.getOutputReaderClass(), OutputReader.class);
jobConf_.setMapOutputKeyClass(idResolver.getOutputKeyClass());
jobConf_.setMapOutputValueClass(idResolver.getOutputValueClass());

Unless, stream.map.output property is set to rawbytes or typedbytes, the map output key/value are set as Text.class in IdentifierResolver#resolve.

setOutputKeyClass(Text.class);
setOutputValueClass(Text.class);

In the Text.java, Comparator subclass extends WritableComparator and implements compare method which does compare in Lexicographic order (dictionary or alphabetical order).

Check the StreamJob, IdentifierResolver and Text classes.