My map task completes successfully and I can see the application logs, but reducer stays in pending state
Kind % Complete Num Tasks Pending Running Complete Killed Failed/Killed Task Attempts
map 100.00% 200 0 0 200 0 0 / 40
reduce 0.00% 1 1 0 0 0 0 / 0
When I look at reduce task, I see
All Task Attempts
No Task Attempts found
When I see the hadoop-hduser-jobtracker-master.log, I see the following :
2011-10-31 00:00:00,238 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker2:localhost/127.0.0.1:35984 has 67730714624 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:01,634 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 68041449472 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:03,863 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397398528 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:04,532 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker2:localhost/127.0.0.1:35984 has 67729956864 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:04,695 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67998732288 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:06,865 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397390336 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:08,057 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67948941312 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:09,566 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker2:localhost/127.0.0.1:35984 has 67728035840 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:11,118 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67907883008 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:12,258 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397365760 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:14,119 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67867385856 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:16,061 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397390336 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:17,120 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67813683200 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:19,386 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker2:localhost/127.0.0.1:35984 has 67726835712 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:20,122 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67896156160 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:21,949 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397390336 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:23,124 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 68150616064 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:25,078 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397365760 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:25,762 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker2:localhost/127.0.0.1:35984 has 67725717504 bytes free; but we expect reduce input to take 184825113602
How do I fix this issue?
Please help
Your disk on the host where the reducer is run is too small. Your reducer input takes approx. 172 gbs, but you have only 63gb free.
Maybe combiner could be useful for you. Have a look here: http://tumblr.mobocracy.net/post/6589143971/using-a-combiner-with-hadoop-jobs