Question

In hadoop 0.20.2 version one can add input/output compression to the jobconf in the following way:

jobConf.setBoolean("mapred.output.compress", true);

jobConf.setClass("mapred.output.compression.codec", BZip2Codec.class, CompressionCodec.class);

jobConf is deprecated and job should be used instead. How can I add compression/decompression there? In particular, how can I change the wordcount example to input bzip2 files:

public class WordCount {
public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
    Job job = new Job(conf, "Example Hadoop 0.20.1 WordCount");
    job.setJarByClass(WordCount.class);
    job.setMapperClass(TokenCounterMapper.class);
    job.setReducerClass(TokenCounterReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);

    FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
    FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));

    System.exit(job.waitForCompletion(true) ? 0 : 1);
}

Answer 1

Use the Configuration class as below while submitting the Job

Configuration conf = new Configuration();
conf.set("mapred.output.compression.codec",
    "org.apache.hadoop.io.compress.BZip2Codec");
Job job = new Job(conf);