Question

In our application we use a EBS partition with a Mysql database. Eventually, we ran out of space and had to a bigger partition for the DB.
We used the AWS panel functionality to create a new volume using a snapshot of the previous one.
Mysql was stopped, and now we are using the new, bigger EBS partition.
However, we had noticeable degradation of the database performance. We are not sure how this could happen, since theoretically we are using the same Mysql configuration and the same database.
Is it possible that maybe we have to rebuild indexes or re-optimize tables? I am not sure if that would be worth anything, so we haven’t tried it yet because we are afraid it could slow down the database even more, and our application cannot be stopped easily, as it runs 24/7.
Can anyone help please?

Answer 1

When you start using a new EBS volume (whether created from a snapshot or created empty) there is always a first-use performance penalty for each block. This will manifest itself as slow performance from your MySQL database using the new volume.

You can “dd” the EBS volume to /dev/null to make sure all blocks have been hit. Here’s an article I wrote on how to do this: http://alestic.com/2010/03/ebs-volume-initialization-from-snapshot

There may also be a performance hit while the database is brought into memory through queries. This is a standard IO issue that would happen when restarting the database on any platform and is unrelated to EC2 or EBS.

If the performance stays slow after everything has warmed up and should be humming, then you may try things like:

• Create a new EBS volume and test it just in case the slow one was using defective hardware at EC2.

• Move your EC2 instance to new hardware just in case your neighbors on the current hardware are network heavy and interfering with your EBS IO. This can be done through a simple stop/start (I wrote about it here: http://alestic.com/2011/02/ec2-move-hardware )

• Move your database to 4-8 EBS volumes configured in RAID-0. This is a common approach to try to smooth out EBS IO volatility.

• Consider trying out Amazon RDS. Some people have found that they get better performance with Amazon taking care of this part of the infrastructure.

Note also that you may experience IO slowness while an EBS snapshot is being created from an EBS volume that is being written to heavily. One approach to alleviate this is to replicate the master MySQL database to a different server and create the snapshots on the second server’s EBS volume(s).