Question

How do people (and start up companies) actually go about prototyping/deploying things on amazon and keep costs reasonable? Last month we were experimenting with some specific applications and running own hadoop cluster and managed to spend almost 1.5k just for tests ? Sure – they have micro instances, but what if you application is so intensive it actually requires a larger instance to even test? So I’d like some input as to how people go about doing this?

Answer 1

Several key issues:

1. Consider a local testbed for some purposes & consider if a given test really needs EC2. If it’s really so hard to wrangle 2-4 machines to use as a testbed for Hadoop, there’s a different problem. Get your head around whatever you’re going to run, how Hadoop will play a role, and kick the tires on that. In time, you will also want to change your grid, upgrade software, tinker with other ideas, etc. When you go to EC2, you’ll have smoothed some rough edges already.
2. Don’t use a larger capacity machine than you need while getting the hang of things. If you’re not pushing lots of data or compute cycles through at this stage, don’t bother with cluster compute nodes, massive RAM instances, etc. Just focus on getting things set up correctly.
3. When you are ready to retarget to more powerful machines, try a few different machine setups. Maybe the cluster compute instances will pay off, maybe you don’t need that kind of throughput: until you know your bottlenecks, don’t overspend.
4. Be sure to use spot instances frequently during the testing phase. You will typically pay about 50% of the on-demand price.
5. If you get to a point where you want to pay for on-demand instances, have a separate instance start and stop Hadoop instances as needed – unless you need a big cluster all on cluster compute instances.
6. Prepare your AMIs to get launched as quickly as possible (under 1 minute) and never leave anything running overnight or over a weekend if it isn’t necessary.

Until you get the system set up and running, you’re basically paying tuition to learn how to get everything tailored to your needs. Just pay the “tuition” to learn each lesson (configurations, bottlenecks, scaling up, etc.), rather than try to take on everything at once. When you approach it as a series of lessons to be learned, it is less painful to spend the money, but as long as you know what you’re about to test and learn, you will also spend money more judiciously.

Finally, compare the $1500 to the labor costs of this learning experience – it probably isn’t a big deal in the long run. Once you know that something is going to be a reasonable block of computational effort, it’s well engineered, and will finish quickly (albeit on many machines), it isn’t so painful to spend money on it. Right now, it’s hard to appreciate what you’re learning because it doesn’t yet benefit your org’s goals.