As part of my Java mapper I have a command executes some standalone code on a local slave node. When I run a code it executes fine, unless it is trying to access some local files in which case I get the error that it cannot locate those files.
Digging a little deeper it seems to be executing from the following directory:
/data/hadoop/mapred/local/taskTracker/{user}/jobcache/job_201109261253_0023/attempt_201109261253_0023_m_000001_0/work
But I am intending to execute from a local directory where the relevant files are located:
/home/users/{user}/input/jobname
Is there a way in java/hadoop to force the execution from the local directory, instead of the jobcache directory automatically created in hadoop?
Is there perhaps a better way to go about this?
Any help on this would be greatly appreciated!
A workaround method I’m using right now that works consists of copying all the relevant files over to the jobcache working directory. Then you can copy the results back to user directory if necessary.
Unfortunately this doesn’t fully answer the question, but hopefully provides a useful workaround for others.
Cheers,
Joris