Question

I have a program which writes data to mysql database and also huge amounts of logs to a file.. i have noticed that if i give huge amounts of data as input to the program, i.e data that creates logs as big as 70GB and mysql database table count(*), of the table that i use, to >1,000,000 entries, the whole programme slows down after some time..

But when initially the reports were collected at the rate of around 1000/min but the same becomes < 400/min wen the data is as i said before. Is this the database writes or the file writes that makes the program slower?

The logs are just cout from my program that are redirected to a file. No buffering is done there.

Answer 1

There’s an easy way to test for this.

If you create a blackhole table, MySQL will pretend to do everything but never really write any data to file.

1. create table(s) just like your
   normal table(s),
2. Make a copy of the logs.
3. Now write to the blackhole database just like you would in the real database.

If it’s much faster it’s MySQL giving you grief.

See: http://dev.mysql.com/doc/refman/5.5/en/blackhole-storage-engine.html