Question

I remember reading somewhere that Hadoop’s performance deteriorates significantly if the machines it runs on are very different from one another, but I can’t seem to find that comment anymore. I am considering running a Hadoop cluster on an array of VMs that is not directly managed by my group, and I need to know if this is a requirement that I should put in my request.

So, should I insist on all of my machines having identical hardware, or is it okay to run on different machines in different hardware configurations?

Thanks.

Answer 1

Following papers describes how heterogeneous cluster affect the performance of hadoop map-reduce:

In a heterogeneous cluster, the computing capacities of nodes may vary
significantly. A high-speed node can finish processing data stored in a
local disk of the node faster than low-speed counterparts. After a
fast node complete the processing of its local input data, the node
must support load sharing by handling unprocessed data located in one
or more remote slow nodes. When the amount of transferred data due to
load sharing is very large, the overhead of moving unprocessed data
from slow nodes to fast nodes becomes a critical issue affecting
Hadoop’s performance.

Following references has more details:

1. http://computerresearch.org/stpr/index.php/gjcst/article/view/749/658
2. http://www.usenix.org/event/osdi08/tech/full_papers/zaharia/zaharia.pdf

It also provides ways in which you could improve the performance on heterogeneous cluster or avoid this performance penalty.

It is wisely suggested that you have homogenous machines on your cluster but if these machines do not have wildly different specifications and performance difference, you should carry on with building your cluster.

For production systems, you should suggest for homogenous machines. For development, performance is not critical.

How ever, you should be able to benchmark your Hadoop cluster after you have built it.