Home/ Questions/Q 972647
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-16T03:09:24+00:00

Loving MRToolkit — great to get away from Java while writing Hadoop jobs. It

  • 0

Loving MRToolkit — great to get away from Java while writing Hadoop jobs. It has become apparent that the library was written to interface with an EC2 cluster, and not with Amazon’s elastic map/reduce system. Does anybody have insights into running jobs defined using the toolkit on elastic map/reduce servers? It isn’t readily apparent from the web interface, and I’d love to avoid the headache of setting up a cluster by hand on EC2.

I’ve looked into updloading files under the ‘streaming’ option (as that’s what MRToolkit uses), but Amazon is expecting separate files for the mapper and reducer — typical MRToolkit style defines them in the a single file as subclasses of predefined Base(Map|Reduce) classes.

Thanks much for any thoughts.

Isaac

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    It’s doable, but not through the web GUI.

    • Download and install the Ruby Client
    • Create your cluster: elastic-mapreduce –create –alive [params to size cluster]
    • Confirm your Elastic Map Reduce Master security group has port 22 open
    • SSH into your master node
    • Use git / scp to copy over your application code
    • Run your app
    • 0