I have a MySQL table with about 2 million rows, and a script that updates approximately 100 rows per second. I’d like to reduce the amount of disk write I/O that’s going on. For this particular table, ACID isn’t important as if I were to lose some rows in a crash the script would just resume at the proper place. Even if I lost the past hour’s work, it wouldn’t be that big of deal.
The table was using InnoDB, but I switched over to MyISAM because I figured if it wasn’t logging every write that could cut the I/O by half.
But even with MyISAM there is a lot of write I/O going on. The table + index takes up about 1300 MB on disk, but MySQL is writing about 1600 MB to disk every hour. I’ve calculated that if each row could be written to disk perfectly efficiently, that would be about 160 MB written per hour. So its writing about 10x as much data as it needs to. I realize there are some inefficiencies, but I’m guessing that most of the writes are because its writing an entire page out to disk.
Is there any way to make it write less often, so it waits until more rows on each page have been updated so that it can be more efficient with the writes (even though there would be more data lost in event of a crash)?
If you are using innodb you could set innodb_flush_log_at_trx_commit to 2 for example. This greatly improved the I/O during updates on our system.
Here some clarification on the setting:
http://www.mysqlperformanceblog.com/?s=innodb_flush_log_at_trx_commit