Question

In my mapper code I am using a 3rd party library of JTS.jar. I need to put it on the distributed cache of hadoop so that all nodes can access it. I found at this link that -libjars can be used to do this.

I now execute my code using

hadoop jar -libjars JTS.jar my_jar.jar classname inputFiles outputFiles.

but this doesn’t work. Any suggestion on how to fix this?

Answer 1

In a different effort, I tried to follow this link.

1) I copied the Jar library to hadoop using:

hadoop fs -copyFromLocal JTS.jar /someHadoopFolder/JTS.jar

2) I then modified my Configuration as below:

        Configuration conf = new Configuration();

    Job job = new Job(conf);
    job.setJobName("TEST JOB");

    List<String> other_args = parseArguments(args, job);

    DistributedCache.addFileToClassPath(new Path("/someHadoopFolder/JTS.jar"), conf);

    job.setMapOutputKeyClass(LongWritable.class);
    job.setMapOutputValueClass(Text.class);

    job.setOutputKeyClass(LongWritable.class);
    job.setOutputValueClass(Text.class);

    job.setMapperClass(myMapper.class);
    //job.setCombinerClass(myReducer.class);
    //job.setReducerClass(myReducer.class);

    job.setInputFormatClass(TextInputFormat.class);   
    job.setOutputFormatClass(TextOutputFormat.class);


    String inPath = other_args.get(0);
    String outPath = other_args.get(1);     
    TextInputFormat.setInputPaths(job, inPath);
    TextOutputFormat.setOutputPath(job, new Path(outPath));

    TextInputFormat.setMinInputSplitSize(job, 32 * MEGABYTES);
    TextInputFormat.setMaxInputSplitSize(job, 32 * MEGABYTES);

    job.setJarByClass(myFile.class);

    job.waitForCompletion(true);

3) The tutorial then says “Use the cached file in mapper” so my mapper looks like this:

    public static class myMapper extends Mapper<LongWritable, Text, LongWritable, Text>{
       private Path[] localArchives;
       private Path[] localFiles;

       public void configure(Configuration conf) throws IOException {
         localArchives = DistributedCache.getLocalCacheArchives(conf);
         localFiles = DistributedCache.getLocalCacheFiles(conf);
       }



    public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{
            //ENVELOPE IS FROM THE JTS.JAR library
        Envelope e1 = new Envelope(-180, 85, 180, -85);
            context.write(key, value);

    }

}

Despite doing all these, The code still fails by throwing “Class bout found”. Any help?