Question

Is there a way to reduce the I/O’s associated with either mysql or a python script? I am thinking of using EC2 and the costs seem okay except I can’t really predict my I/O usage and I am worried it might blindside me with costs.

I basically develop a python script to parse data and upload it into mysql. Once its in mysql, I do some fairly heavy analytic on it(creating new columns, tables..basically alot of math and financial based analysis on a large dataset). So is there any design best practices to avoid heavy I/O’s? I think memcached stores a everything in memory and accesses it from there, is there a way to get mysql or other scripts to do the same?

I am running the scripts fine right now on another host with 2 gigs of ram, but the ec2 instance I was looking at had about 8 gigs so I was wondering if I could use the extra memory to save me some money.

Answer 1

By IO I assume you mean disk IO… and assuming you can fit everything into memory comfortably. You could:

• Disable swap on your box†
• Use mysql MEMORY tables while you are processing, (or perhaps consider using an Sqlite3 in memory store if you are only using the database for the convenience of SQL queries)

Also: unless you are using EBS I didn’t think Amazon charged for IO on your instance. EBS is much slower than your instance storage so only use it when you need the persistance, ie. not while you are crunching data.

†probably bad idea