Question

I have a script residing on my webserver’s cron that should run every night. It has stopped running recently due to exceeding the time limits set by the webserver on cron jobs. It used to run fine. Anytime I manually ran it, it was very quick (well under 5 minutes). All of the sudden, it takes over half an hour.

The script basically updates a MySQL database. The DB is around 60mb according to them. I can’t seem to find this information, but it seems reasonable (though the file that I transfer to the server every night is only around 2mb).

I have undertaken their steps suggested to optimize my DB, but nothing really came of that. It still takes ages for the script to run. All the script does is delete everything out of the DB and fill it in again with our updated inventory.

So now I am running “show full processlist” on one Putty window, while running the script in another. “show full processlist” shows only a couple of items, both of which show 0 for the time.

mysql> show full processlist;
+-----------+--------------+--------------------+-------------------------+---------+------+-------+-----------------------+
| Id        | User         | Host               | db                      | Command | Time | State | Info                  |
+-----------+--------------+--------------------+-------------------------+---------+------+-------+-----------------------+
| 142841868 | purposely omitted | purposely omitted | purposely omitted_net_-_main | Sleep   |    0 |       | NULL                  |
| 142857238 | purposely omitted | purposely omitted | NULL                    | Query   |    0 | NULL  | show full processlist |
+-----------+--------------+--------------------+-------------------------+---------+------+-------+-----------------------+
2 rows in set (0.05 sec)

If I keep using the show full processlist command really quickly, occasionally I can catch other things being listed in this table but then they disappear the next time I run it. This indicates to me that they are being processed very quickly!

So does anyone have any ideas what is going wrong? I am fairly new to this 🙁

Thanks!!

PS here is my code

#!/usr/bin/perl
use strict;
use DBI;
     my $host = 'PURPOSLEY OMITTED';
     my $db = 'PURPOSLEY OMITTED';
     my $db_user = 'PURPOSLEY OMITTED';
     my $db_password = "PURPOSLEY OMITTED";

my $dbh = DBI->connect("dbi:mysql:$db:$host", "$db_user", "$db_password");
$dbh->do("DELETE FROM main");
$dbh->do("DELETE FROM keywords");

open FH, "PURPOSLEY OMITTED" or die;

while (my $line = <FH>) {

my @rec = split(/\|/, $line);

print $rec[1].' : '.$rec[2].' : '.$rec[3].' : '.$rec[4].' : '.$rec[5].' : '.$rec[6].' : '.$rec[7];

$rec[16] =~ s/"//g;
$rec[17] =~ s/"//g;
$rec[13] =~ chomp($rec[13]);

my $myquery = "INSERT INTO main (medium, title, artist, label, genre, price, qty, catno,barcode,created,received,tickler,blurb,stockid) values (\"$rec[0]\",\"$rec[1]\",\"$rec[2]\",\"$rec[3]\",\"$rec[4]\",\"$rec[5]\",\"$rec[6]\",\"$rec[7]\",\"$rec[8]\",\"$rec[9]\",\"$rec[10]\",\"$rec[11]\",\"$rec[12]\",\"$rec[13]\")"; 

$dbh->do($myquery);

$dbh->do("INSERT IGNORE INTO keywords VALUES (0, '$rec[2]','$rec[13]')");
$dbh->do("INSERT LOW_PRIORITY IGNORE INTO keywords VALUES (0, \"$rec[1]\", \"$rec[13]\")");

print "\n";
}
close FH;

$dbh->disconnect();

Answer 1

I have two suggestions:

1. (less impact) use TRUNCATE instead of DELETE, it is significantly faster, and is particularly easy to use when yo don’t need to worry about an auto-incrementing value.
2. Restructure slightly to work in batches for the inserts. Usually I do this by keeping a stack variable of a given size (start with something like 20 rows), and for the first 20 rows, it just fills the stack; but on the 20th row it also actually performs the insert and resets the stack. It might boggle your mind how much this can improve performance 🙂

Pseudo-code:

const buffer_size = 20
while(row) {
  stack.addvalues(row.values)
  if(stack.size >= buffer_size) {
    // INSERT INTO mytable (fields) VALUES stack.all_values()
    stack.empty()
  }

then play with the “buffer” size. I have seen scripts where tweaking the buffer to upwards of 100-200 rows at a time sped up massive imports by almost as many times (i.e. a drastically disproportionate amount of work was involved in the “overhead” of executing the individual INSERTs (network, etc)