Question

I’m looking for a solution here to my simple silly problem here.
Say, I have a huge 10GB text file with records delimited by ‘\n’ and I provide the file as a input to the Hadoop framework.
The output should be a file such that it maintains the same order of the records as the input file but with a serial number in front of every record.

For e.g.,

If I have a input text file say,

this is line one
this is line two
this is line three
this is line four
-----------------
-----------------

output file should be,

1 this is line one
2 this is line two
3 this is line three
4 this is line four
------------------
-----------------

Edit: Say instead of a 10GB file I have a 10 TB file, so what could be done now?
Instead of handling the file using Hadoop way what could be the other best approach to do it rather faster?

Moreover, I also want to use multiple reducer not a single reducer.

Answer 1

Not sure you would benefit anything from Hadoop for such a trivial operation, compared to just opening the file, reading line by line and storing each line in a new file (with a sequence/serial number prepended).