Question

Hadoop has configuration parameter hadoop.tmp.dir which, as per documentation, is `“A base for other temporary directories.” I presume, this path refers to local file system.

I set this value to /mnt/hadoop-tmp/hadoop-${user.name}. After formatting the namenode and starting all services, I see exactly same path created on HDFS.

Does this mean, hadoop.tmp.dir refers to temporary location on HDFS?

Answer 1

It’s confusing, but hadoop.tmp.dir is used as the base for temporary directories locally, and also in HDFS. The document isn’t great, but mapred.system.dir is set by default to "${hadoop.tmp.dir}/mapred/system", and this defines the Path on the HDFS where where the Map/Reduce framework stores system files.

If you want these to not be tied together, you can edit your mapred-site.xml such that the definition of mapred.system.dir is something that’s not tied to ${hadoop.tmp.dir}