I would like to start about 15 independent Amazon EC2 ‘processes’. These ‘processes’ should load the input_files (that are stored in S3) and independently process them.
I’d like all the 15 independent Amazon EC2 processes to write to the same output file.
Upon completion, this output file will be saved in S3.
What would be better:
(1) Start 15 small Amazon EC2 instances, each running 1 process
OR
(2) Start 1 big (CPU-intense) Amazon EC2 instance, and have all the 15 processes run as part of it?
Which of the above will be more economical / easier to manage?
Not sure about economics, without doing the math, but it should be fairly straightforward to figure out which one is cheaper.
Option 2 is without question easier to manage. 1 box is almost always simpler than 15. There’s just fewer moving parts. Thread management is handled almost entirely by the CPU. File locking and resource management will be much easier. Writing everything to the same output file will require some form of locking, and having the file on the same machine is undoubtedly simpler than writing over a network.
The benefit of small machines is largely for ease of resource manipulation. With 15 servers, it’s easy to spin up another 10, or take 10 down as the load changes.