Question

I have sets of these files-

objectA-record-data.log file – Contains multiple lines of data with timestamps.

objectA-record-metadata.log file – Contains just one line. Think of it as a metadata for all the records of objectA.

Together they will form one record for processing.

I have been able to process the data file. The mapper sets objectA as the key for all the lines in the data file and reducer processes and prints the text file. Now i want to add metadata also to each of these records. Any ideas how can i do that?

Answer 1

Use DistributedCache:

In the driver/configuration class:

DistributedCache.addCacheFile(new URI("/user/chris/theMetaDataFile.txt"), conf);

In the mapper:

public void setup(Context context) {
    Configuration conf = context.getConfiguration();
    Path[] cachedFiles = DistributedCache.getLocalCacheFiles(conf);
    File metadataFile = new File(chachedFiles[0].toString());

    // metadataFile can now be read and the results stored locally for use in the map method
}