Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
Is there a way to have a whole file sent to a mapper without being split?
I have read this but I am wondering if there is another way of doing the same thing without having to generate an intermediate file. Ideally, I would like an existing option on the command line to Hadoop.
I am using the streaming facility with Python scripts on Amazon EMR.
Just set the configuration property
mapred.min.split.sizeto something huge (10G):Or compress the input file using a codec that isn’t splittable (Gzip). With the .gz extension, TextInputFormat will return false to the
isSplittable(FileSystem, Path)method