Home/ Questions/Q 7194171
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-28T20:19:02+00:00

Imagine a scenario you have a text file with say 10000 rows in it

  • 0

Imagine a scenario you have a text file with say 10000 rows in it and so it will be very small when you save it in the HDFS. Now your goal is to run a map/reduce job on this small file expecting every line of the text file being passed to the mapper. However the processing time for each map k,v is long and you want to launch as many as mappers on the cluster to get the maximum possible parallelism to finish the mapping job the soonest.

Because the file is small, it could be save in one or two hdfs blocks only and I assume the number of provisioned maps by hadoop for the job is going to be equal to the number of hdfs blocks which happened to be one or two. But this is undesired as you want to launch as many as possible mappers. Here are my questions?

  1. What are my options on controlling the proper number of mappers for my scenrio of processing on an small file?
  2. What are the pros and cons of each available options?
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Best approach is to use the NLineInputFormat. According to the documentation NLineInputFormat which splits N lines of input as one split. The con of this approach is that data locality is lost, since the data is small it shouldn’t matter.

    • 0