Question

I’m writing an archiving program in Java. The files that will be archived already reside in HDFS. I need to be able to move the files from one location in HDFS to another location, with the final files being compressed with Gzip. The files to be moved can be quite large, and thus using the HDFS API to move them and compress them can be quite inefficient. So I was thinking that I could write a mapreduce job into my code to do that for me.

However, I have been unable to find any examples that show me how I could copy those files using the MapReduce API and have them output in gzip format. In fact, I’m even struggling to find a programmatic example of how to copy files inside of HDFS through mapreduce at all.

Can anybody shed some light on how I can accomplish this with the MapReduce API?

Edit: Here’s the job configuration code I have so far, which was adapted from the help that Amar has given me:

        conf.setBoolean("mapred.output.compress", true); 
        conf.set("mapred.output.compression.codec","org.apache.hadoop.io.compress.GzipCodec");
        Job job = new Job(conf);
        job.setJarByClass(LogArchiver.class);
        job.setJobName("ArchiveMover_"+dbname);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(Text.class);
        //job.setMapperClass(IdentityMapper.class);
        //job.setReducerClass(IdentityReducer.class);
        job.setInputFormatClass(NonSplittableTextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);
        job.setNumReduceTasks(0);
        FileInputFormat.setInputPaths(job, new Path(archiveStaging+"/"+dbname+"/*/*"));
        FileOutputFormat.setOutputPath(job, new Path(archiveRoot+"/"+dbname));
        job.submit();

Here is the class declaration for NonSplittableTextInputFormat which is inside of the LogArchiver class

public class NonSplittableTextInputFormat extends TextInputFormat {
    public NonSplittableTextInputFormat () {
    }

    @Override
    protected boolean isSplitable(JobContext context, Path file) {
        return false;
    }
}

Answer 1

You may write a custom jar implementation with IdentityMapper and IdentityReducer.
Instead of plain text files, you can generate gzip files as your output. Set the following configurations in run() :

conf.setBoolean("mapred.output.compress", true); 
conf.set("mapred.output.compression.codec","org.apache.hadoop.io.compress.GzipCodec");

In order to ensure that number of files in input and output are same, just that the output files must be gzipped, you have to do 2 things:

1. Implement a NonSplittableTextInputFormat
2. Set reduce tasks to zero.

In order to ensure that one file per mapper is read, you may extend the TextInputFormat as follows:

import org.apache.hadoop.fs.*;
import org.apache.hadoop.mapred.TextInputFormat;
public class NonSplittableTextInputFormat extends TextInputFormat {
    @Override
    protected boolean isSplitable(FileSystem fs, Path file) {
        return false;
    }
}

and use the above implementation as :

job.setInputFormatClass(NonSplittableTextInputFormat.class);

To set reduce tasks to zero, do the following:

job.setNumReduceTasks(0);

This would get the job done for you but for one last thing that file names won’t be the same! But I am sure for this too there must be a work-around here.