Question

Possible Duplicate:
hadoop-streaming example failed to run – Type mismatch in key from map

When I ran Hadoop streaming example, it failed with Type mismatch in key from map

Hadoop version 0.21.0
input file content:

adfad

adfasdflkjlj

Command line: $HADOOP_HOME/bin/hadoop jar $HADOOP_HOME/hadoop-streaming.jar \
-input myInputDirs \
-output myOutputDir \
-mapper org.apache.hadoop.mapred.lib.IdentityMapper \
-reducer /bin/wc

Error I got:

java.lang.Exception: java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable

Please advise. What did I do wrong? Thanks

Answer 1

EDIT: Sorry, didn’t realize this is Streaming. You need to customize how your output is split into Key/Value pairs. The documentation is here: http://hadoop.apache.org/common/docs/current/streaming.html#Customizing+How+Lines+are+Split+into+Key%2FValue+Pairs

The error message gives you all that you need to know. Your mapper is defined as outputting LongWritable keys: `Mapper`, and your Reducer expects Text: `Reducer.`

You need to redefine one or the other of them.