Question

Hi I want to understand the map reduce performance better.

What dominates the performance of MapReduce algorithms implemented in Hadoop?

Is it the computation time, if there are a lot of data that has to be processed at a node, or is it the disk write and read times?

I have observed that disc write time takes a long time as compared to disc read time, when I ran some map reduce programs.

I want to know if the overhead of disc write is much greater than computation time(CPU time), needed to process a large collection of data at a node. Is CPU time trivial in comparison to I/O access?

The algorithm below is what happens at each reduce node:
I want to know if the CPU time for executing this algorithm is trivial compared to reading the input from HDFS and then after processing writing the output to the HDFS.

  Input : R is a multiset of records sorted by the increasing order of their sizes; each    record has been canonicalized by a global ordering O; a Jaccard similarity threshold t
  Output : All pairs of records hx, yi, such that sim(x, y) > t

  1 S <- null;
  2 Ii <- null (1 <= i <= |U|);
  3 for each x belongs to R do
  4 p <- |x| - t * |x| + 1;
  5 for i = 1 to p do
  6 w <- x[i];
  7 for each (y, j) belongs to Iw such
  that |y|>= t*|x| do /* size filtering on |y| */
  8 Calculate similarity s = (x intersection y) /* Similarity calculation*/ 
  9 if similarity>t
     S <- S U (x,y);
  10 Iw <- Iw Union {(x, i)}; /* index the current prefix */;

  11 return S

Answer 1

In general – it depends on the kind of processing you are doing. But it can be pointed out what takes time and consume resource aside of your code.
We will go over MR job process and point out noticeable resource consumptions.
1. Read your split from HDFS. Unless local read optimization takes place – data passed via socket (CPU) and or network + Disk IO. MD5 is calculated also during the read.
1. Input Format. Input data should be chopped into Key Values for the Mapper. Taking into account that it is java it is always dynamic memory allocations and de-allocations. Parsing input usually takes CPU time.
2. From Record Reader to mapper – no serious overhead.
3. Mapper output is sorted and serialized (a lot of CPU) + local disk.
4. Data is pulled by reducers from the mapper machines. A lot of networking.
5. Data merged on reducer side. CPU + Disk.
6. Output from reducer written to HDFS. x3 of data size disk IO + x2 network traffic because of replication.

In a nutshell 3,4,5 are usually most time and resource consuming.