I’ve got a web app which has pretty intensive processor requirements, specifically in relation to the amount of data that gets stored. Because of the massive amounts of data coming in and out (and being sorted, formatted, etc), I’m routinely bringing the server down. The problem is usually CPU related, not memory related, and it is happening because of the massive number of inserts, updates, and selects to the database.
The really processor intensive stuff is done in the background, all run via scheduled cron jobs. The front end, where the users are interacting, is relatively simple.
In order to keep the server up and running quickly, I have to slow down the background processes significantly. I hate to do this, because it limits the whole apps usefulness.
It seems to me that this would be an ideal candidate for something like EC2 – leave the front end files and interaction on the server, and outsource all the processing and data storage to the cloud. However, after doing some reading on EC2, this doesnt sound nearly as simple as I thought it might be – questions of persistent storage and the constant suggestion that I “change the way I think” about cloud computing are making me nervous – mostly because I hear that, but don’t get an explanation of what the change should be.
Can someone help me understand if what I’ve got is a good candidate for EC2, or some other type of system?
Disclosure: I am a former Amazon employee on the EC2 team, and I am now at a company that uses EC2 for a great deal of our processing needs.
I will say this, to start: yes, I think EC2 is a good idea for backing a CPU-intensive process, and I have no idea what “change the way you think” is referring to there.
The reason it is a good idea is because its the least-effort path to bringing up a server cluster. By “it” I am now referring to any kind of service that allows you to spin up / tear down machines at will, not specifically EC2. The downside is that it is more expensive than maintaining your own machines, though the cost covers the fact that it is less work and more flexible (for example, if you only need all that compute power for a day or a week).
And generally speaking, if you have a web application that has to do CPU-intensive data processing and also needs to have a responsive interface, you certainly don’t want to share those responsibilities on a single machine.
Regarding persistent storage, depending on what you want to store, take a look at SimpleDB, S3 and EBS. SimpleDB is a key-value store, S3 holds file objects, and EBS is like the EC2 equivalent of hard drives – you get virtual storage that you can mount to / detach from EC2 instances. One of those will likely suit your needs, and all interface very nicely with EC2.