If I increase the number of mappers and decrease the number of reducers, then is there any difference in the performance (increase/decrease) of any job while execution?
Also I want to ask that How to set the number of mappers and reducers? I have never played with this setting thats why I don’t know about this. I know hadoop but I have code with it as I use Hive a lot.
Also If I want to increase the number of mappers and reducers then how to set it and upto what value do I set it. Is it depend upon the number of instances (Lets say 10)?
Please reply me I want to try this and check the performance.
Thanks.
Changing number of mappers – is pure optimization which should not affect results. You should set number to fully utilize your cluster (if it is dedicated). Try number of mappers per node equal to number of cores. Look on CPU utilization, and increase the number until you get almost full CPU utilization or, you system start swapping. It might happens that you need less mappers then cores, if you have not enough memory.
Number of reducers impacts results so , if you need specific number of reducer (like 1) – set it
If you can handle results of any number of reducers – do the same optimization as with Mappers.
Theoretically you can became IO bound during this tuning process – pay attention to this also when tuning number of tasks. You can recognieze it by low CPU utilization despite increase of mappers / reducers count.