Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
Is there a way to set the replication factor for the output of a specific MapReduce job to be different than the rest of the cluster (say 1)? I’d like my main data set to be 3x replicas (as it is currently), but the output of some of my jobs move out of the cluster quickly and get tossed out eventually, so no replication is needed and I could use the space.
I could use setrep but I think I can only do that after the fact.
When you upload a file, you can override the DFS default replication factor by passing
This should work as well when passed when you invoke a job.