Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
The map portion of my Mapreduce job depends on Numpy. So, that means I need to have numpy installed as part of the bootstrap actions.
What I’m thinking of doing is building a custom Numpy package stored on S3 that is fetched and installed during the boostrap actions.
Is there a better way?
Numpy now comes installed on Amazon Elastic MapReduce instances, but if you want to use other modules, you can zip them up, distribute them to your workers with the DistributedCache (using “-cacheFile”), and then import them with Python’s built-in “zipimport” module.
See: http://www.cloudera.com/blog/2008/11/sending-files-to-remote-task-nodes-with-hadoop-mapreduce/