Can I write my own custom static method in JAVA inside map-reduce that I may want to call for every key-value pair that enters my mapper?
Share
Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
Can I write my own custom static method in JAVA inside map-reduce that I may want to call for every key-value pair that enters my mapper?
You can do such a thing and I do it all the time. The methods you call can be either static or not.
Basically:
If you call the outside code from code that you’ve written yourself thats included in your project, you need to bundle that up with your code when you export the project to a .jar file.
If you are referencing methods and such from a .jar file that you’ve included in the build path for your project, you need to place that .jar file on every node of your cluster where the job would run, and edit your HADOOP_CLASSPATH variable on your servers in the file $HADOOP_HOME/conf/hadoop-env.sh file and restart your cluster. Otherwise you will get a class not found error.
An example is: