Question

I want to build a hadoop application which can read words from one file and search in another file.

If the word exists – it has to write to one output file
If the word doesn’t exist – it has to write to another output file

I tried a few examples in hadoop. I have two questions

Two files are approximately 200MB each. Checking every word in another file might cause out of memory. Is there an alternative way of doing this?

How to write data to different files because output of the reduce phase of hadoop writes to only one file. Is it possible to have a filter for reduce phase to write data to different output files?

Thank you.

Answer 1

How I would do it:

1. split value in ‘map’ by words, emit (<word>, <source>) (*1)
2. you’ll get in ‘reduce’: (<word>, <list of sources>)
3. check source-list (might be long for both/all sources)
4. if NOT all sources are in the list, emit every time (<missingsource>, <word>)
5. job2: job.setNumReduceTasks(<numberofsources>)
6. job2: emit in ‘map’ (<missingsource>, <word>)
7. job2: emit for each <missingsource> in ‘reduce’ all (null, <word>)

You’ll end up with as much reduce-outputs as different <missingsources>, each containing the missing words for the document. You could write out the <missingsource> ONCE at the beginning of ‘reduce’ to mark the files.

(*1) Howto find out the source in map (0.20):

private String localname;
private Text outkey = new Text();   
private Text outvalue = new Text();
...
public void setup(Context context) throws InterruptedException, IOException {
    super.setup(context);

    localname = ((FileSplit)context.getInputSplit()).getPath().toString();
}

public void map(Object key, Text value, Context context)
    throws IOException, InterruptedException {
...
    outkey.set(...);
    outvalue.set(localname);
    context.write(outkey, outvalue);
}