I have a simple pure C program, that takes an integer as input, runs for a while (let’s say an hour) and then returns me a text file. I want to run this program 1000 times with input integers from 1 to 1000.
Currently I’m running this program in parallel (4 processors), what takes 250 hours. The program is such that it fits in a AWS micro instance (I’ve tested it). Would it be possible to use 1000 micro instances in AWS to do the whole job in one hour? (at a cost of ~20$ – $0.02/instance)?
If it is possible, does anybody have some guidlines on how to do that?
If it is not, does anybody have an also low-budget alternative to that?
Thanks a lot!
In order to achieve this, you will need to:
Create a S3 bucket to store your bootstrapping scripts, application, input data and output data
Custom Lightweight AMI: you might want to create a custom lightweight AMI which knows about downloading the bootstrapping script
Bootstrapping Script: will download your software from your S3 bucket, parse a custom instance tag which will contain the integer [1..1000] and download any additional data.
Your application: does the processing stuff.
End of processing script is which uploads the result to a another result S3 bucket and terminates the instance, you might also want to send a SNS notification to communicate the end of processing status.
If you need result consolidation you might want to create another instance and use it as a coordinator, waiting for all “end of processing” notifications in order to finish the processing. In this case you might consider using in the future the Amazon’s Hadoop map reduce engine, since it will do almost all this heavy lifting for free.