Question

I’m trying to run the Fair Scheduler, but it’s not assigning Map tasks to some nodes with only one job running. My understanding is that the Fair Scheduler will use the conf slot limits unless multiple jobs exist, at which point the fairness calculations kick in. I’ve also tried setting all queues to FIFO in fair-scheduler.xml, but I get the same results.

I’ve set the scheduler in all mapred-site.xml files with the mapreduce.jobtracker.taskscheduler parameter (although I believe only the JobTracker needs it) and some nodes have no problem receiving and running Map tasks. However, other nodes either never get any Map tasks, or get one round of Map tasks (ie, all slots filled once) and then never get any again.

I tried this as a prerequisite to developing my own LoadManager, so I went ahead and put a debug LoadManager together. From log messages, I can see that the problem nodes keep requesting Map tasks, and that their slots are empty. However, they’re never assigned any.

All nodes work perfectly with the default scheduler. I just started having this issue when I enabled the Fair Scheduler.

Any ideas? Does someone have this working, and has taken a step that I’ve missed?

EDIT: It’s worth noting that the Fair Scheduler web UI page indicates the correct Fair Share count, but that the Running column is always less. I’m using the default per-user pools and only have 1 user and 1 job at a time.

Answer 1

The reason was the undocumented mapred.fairscheduler.locality.delay parameter. The problematic nodes were located on a different rack with HDFS disabled, making all tasks on these nodes non-rack local. Because of this, they were incurring large delays due to the Fair Scheduler’s Delay Scheduling algorithm, described here.