Home/ Questions/Q 6601833
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-25T18:48:15+00:00

I couldn’t find an answer to my issue while sifting through some Hadoop guides:

  • 0

I couldn’t find an answer to my issue while sifting through some Hadoop guides: I am committing various Hadoop jobs (up to 200) in one go via a shell script on a client computer. Each job is started by means of a JAR (which is quite large; approx. 150 MB). Right after submitting the jobs, the client machine has a very high CPU load (each core on 100%) and the RAM is getting full quite fast. That way, the client is no longer usable. I thought that the computation of each job is entirely done within the Hadoop framework, and only some status information is exchanged between the cluster and the client while the job is running.

So, why is the client fully stretched? Am I committing Hadoop jobs the wrong way? Is each JAR too big?

Thanks in advance.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    It is not about the jar. The client side is calculating the InputSplits.
    So it can be possible that when having large number of input files for each job the client machine gets a lot of load.
    But I guess when submitting 200 jobs the RPC Handler on the jobtracker has some problems. How many RPC handlers are active on the jobtracker?

    Anyways, I would batch the submission up to 10 or 20 jobs at a time and wait for their completion. I guess you’re having the default FIFO scheduler? So you won’t benefit from submitting all 200 jobs at a time either.

    • 0