Question

I’m trying to setup a hadoop cluster on 5 machines on same lan with NFS. The problem im facing is that the copy of hadoop on one machine is replicated on all the machines, so i cant provide exclusive properties for each slaves. Due to this, i get “Cannot create lock” kind of errors. The FAQ suggests that NFS should not be used, but i have no other option.
Is there a way where i can specify properties like, Master should pick its conf files from location1, slave1 should pick its conf files from location2 …..

Answer 1

Just to be clear, there’s a difference between configurations for compute nodes and HDFS storage. Your issue appears to be solely the storage for configurations. This can and should be done locally, or at least let each machine map to a symlink based on some locally identified configuration (e.g. Mach01 -> /etc/config/mach01, …).

(Revision 1) Regarding the comment/question below about symlinks: First, I’m going to admit that this is not something I can immediately solve. There are 2 approaches I see:

1. Have a script (e.g. at startup or as a wrapper for starting Hadoop) on the machine determine the hostname (e.g. hostname -a') which then identifies a local symlink (e.g./usr/local/hadoopConfig’) to the correct directory on the NFS directory structure.
2. Set an environment variable, a la HADOOP_HOME, based on the local machine’s hostname, and let various scripts work with this.

Although #1 should work, it is a method relayed to me, not one that I set up, and I’d be a little concerned about symlinks in the event that the hostname is misconfigured (this can happen). Method #2 is one that seems more robust.