Question

I am learning to work on hadoop cluster. I have worked for some time on hadoop streaming where I coded map-reduce scripts in perl/python and ran the job.
However, I didn’t find any good explanation for running a java map reduce job.
For example:
I have the following program-

http://www.infosci.cornell.edu/hadoop/wordcount.html

Can somebody tell me how shall I actually compile this program and run the job.

Answer 1

Create a directory to hold the compiled class:

mkdir WordCount_classes

Compile your class:

javac -classpath ${HADOOP_HOME}/hadoop-${HADOOP_VERSION}-core.jar -d WordCount_classes WordCount.java

Create a jar file from your compiled class:

jar -cvf $HOME/code/hadoop/WordCount.jar -C WordCount_classes/ .

Create a directory for your input and copy all your input files into it, then run your job as follows:

bin/hadoop jar $HOME/code/WordCount.jar WordCount ${INPUTDIR} ${OUTPUTDIR}

The output of your job will be put in the ${OUTPUTDIR} directory. This directory is created by the Hadoop job, so make sure it doesn’t exist before you run the job.

See here for a full example.