Is it possible to write map/reduce jobs for Amazon Elastic MapReduce (http://aws.amazon.com/elasticmapreduce/) using .NET languages? In particular I would like to use C#.
Preliminary research suggests not. The above URL’s marketing text suggests you have a “choice of Java, Ruby, Perl, Python, PHP, R, or C++”, without mentioning .NET languages. This Amazon thread (http://developer.amazonwebservices.com/connect/thread.jspa?messageID=136051 — “Support for C# / F# map/reducers”) explicitly says that “currently Amazon Elastic MapReduce does not support Mono platform or languages such as C# or F#.”
The above suggests that it can’t be done. I’m wondering if there are any workarounds, though. For example, can I modify the Elastic MapReduce machine image for my account, and install Mono on there?
An alternative, suggested by Amazon FAQs “Using Other Software Required by Your Jar” (http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/index.html?CHAP_AdvancedTopics.html) and “How to Use Additional Files and Libraries With the Mapper or Reducer” (http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/index.html?addl_files.html), is to make the first step of the Map/Reduce job be to install Mono on the local instance. That sounds kind of inefficient, but maybe it could work?
Maybe a saner alternative would be to try to forgo the convenience of Elastic MapReduce, and manually set up my own Hadoop cluster on EC2. Then I assume I could install Mono without difficulty.
Yes, it is possible using the Bootstrap action as previous answerers have suggested.
The blog posting – http://atbrox.com/2011/02/07/an-example-of-using-f-and-c-netmono-with-amazons-elastic-mapreduce-hadoop/ – gives a description of having a C# mapper and a F# reducer with mono