Question

hadoop writes in a SequenceFile in in key-value pair(record) format. Consider we have a large unbounded log file. Hadoop will split the file based on block size and save them on multiple data nodes. Is it guaranteed that each key-value pair will reside on a single block? or we may have a case so that key is in one block on node 1 and value(or parts of it) on second block on node 2? If we may have unmeaning-full splits, then what is the solution? sync markers?

Another question is: Does hadoop automatically write sync markers or we should write it manually?

Answer 1

I asked this question in hadoop mailing list. They answered:

Sync markers are written into sequence files already, they are part of
the format. This is nothing to worry about – and is simple enough to
test and be confident about. The mechanism is same as reading a text
file with newlines – the reader will ensure reading off the boundary
data in order to complete a record if it has to.

then I asked:

So if we have a map job analysing only the second block of the log
file, it should not transfer any other parts of that from other nodes
because that part is stand alone and meaning full split? Am I right?

They answered:

Yes. Simply put, your records shall never break. We do not read just
at the split boundaries, we may extend beyond boundaries until a sync
marker is encountered in order to complete a record or series of
records. The subsequent mappers will always skip until their first
sync marker, and then begin reading – to avoid duplication. This is
exactly how text file reading works as well — only here, it is
newlines.