Home/ Questions/Q 852387
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-15T07:35:08+00:00

I am working on development of an application to process (and merge) several large

  • 0

I am working on development of an application to process (and merge) several large java serialized objects (size of order GBs) using Hadoop framework. Hadoop stores distributes blocks of a file on different hosts. But as deserialization will require the all the blocks to be present on single host, its gonna hit the performance drastically. How can I deal this situation where different blocks have to cant be individually processed, unlike text files ?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    There’s two issues: one is that each file must (in the initial stage) be processed in whole: the mapper that sees the first byte must handle all the rest of that file. The other problem is locality: for best efficiency, you’d like all the blocks for each such file to reside on the same host.

    Processing files in whole:

    One simple trick is to have the first-stage mapper process a list of filenames, not their contents. If you want 50 map jobs to run, make 50 files each with that fraction of the filenames. This is easy and works with java or streaming hadoop.

    Alternatively, use a non-splittable input format such as NonSplitableTextInputFormat.

    For more details, see “How do I process files, one per map?” and “How do I get each of my maps to work on one complete input-file?” on the hadoop wiki.

    Locality:

    This leaves a problem, however, that the blocks you are reading from are disributed all across the HDFS: normally a performance gain, here a real problem. I don’t believe there’s any way to chain certain blocks to travel together in the HDFS.

    Is it possible to place the files in each node’s local storage? This is actually the most performant and easiest way to solve this: have each machine start jobs to process all the files in e.g. /data/1/**/*.data (being as clever as you care to be about efficiently using local partitions and number of CPU cores).

    If the files originate from a SAN or from say s3 anyway, try just pulling from there directly: it’s built to handle the swarm.

    A note on using the first trick: If some of the files are much larger than others, put them alone in the earliest-named listing, to avoid issues with speculative execution. You might turn off speculative execution for such jobs anyway if the tasks are dependable and you don’t want some batches processed multiple times.

    • 0