Question

I have an application which need to read a file which is a serialized result of ArrayList.(ArrayList<String>, 50000 records in this list, size: 20MB)
I don’t know exactly how to read the data in to hadoop platform. I only have some sense I need to override InputFormat and OutpurFormat.
I’m a beginner in hadoop platform. Could you give me some advise?

Thanks,

Zheng.

Answer 1

To start with you’ll need to extend the FileInputFormat, notable implementing the abstract FileInputFormat.createRecordReader method.

You can look through the source of something like the LineRecordReader (which is what TextInputFormat uses to process text files).

From there you’re pretty much on your own (i.e. it depends on how your ArrayList has been serialized). Look through the source for the LineRecordReader and try and relate that to how your ArrayList has been serialized.

Some other points of note, is your file format splittable? I.e. can you seek to an offset in the file and recover the stream from there (Text files can as they just scan forward to the end of the current line and then start from there). If your file format uses compression, you also need to take this into account (you cannot for example randomly seek to a position in a gzip file). By default FileInputFormat.isSplittable will return true, which you may want to initially override to be false. If you do stick with ‘unsplittable’ then note that your file will be processed by a single mapper (not matter it’s size).