Question

I have a web crawler that saves information to a database as it crawls the web. While it does this, it also saves a log file of its actions, and any errors it encounters to a log field in a mysql database (field becomes anywhere from 64kb to 100kb. It accomplishes this by concatenating (using the mysql CONCAT function).

This seems to work fine, but I am concerned about the cpu useage / impact it has on the mysql database. I’ve noticed that the web crawling is performing slower than before I implemented saving the log to the database.

I view this log file from a management webpage, and the current implementation seems to work fine other than the slow loading. Any recommendations for speeding this up, or implementation recommendations?

Answer 1

Reading 100kb strings into memory numerous time then write them to disk via a db. Of course your going to experience slowdown! Every part of what you are doing is going to task memory, disk, and cpu (especially if memory usage hits the system max and you start swapping to disk). Let me count some of the ways your going to possibly decrease overall site performance:

1. Sql connections max out and back up as the time to store 100kb records increases time a single process holds a connection
2. Webserver processes eat up free process pool and max out and take longer to free up because they have to wait on db connections to free.
3. Web server processes begin to bloat and take more memory each, possibly more than the system can handle without swapping. This is compounded by using the max. Umber of processes due to #2
4. … A book could be written on your situation.