Question

Please bear with me on this question.

I’m looking to create a relatively large MySQL database that I want to use to do some performance testing. I’m using Ubuntu 11.04 by the way.

I want to create about 6 tables, each with about 50 million records. Each table will have about 10 columns. The data would just be random data.

However, I’m not sure how I can go about doing this. Do I use PHP and loop INSERT queries (bound to timeout)? Or if that is inefficient, is there a way I can do this via some command line utility or shell script?

I’d really appreciate some guidance.

Thanks in advance.

Answer 1

Probably it is fastest to run multiple inserts in one query as:

INSERT INTO `test` VALUES
    (1,2,3,4,5,6,7,8,9,0),
    (1,2,3,4,5,6,7,8,9,0),
     .....
    (1,2,3,4,5,6,7,8,9,0)

I created a PHP script to do this. First I tried to construct a query that will hold 1 million inserts but it failed. Then I tried with 100 thousend and it failed again. 50 thousends don’t do it also. My nest try was with 10 000 and it works fine. I guess I am hitting the transfer limit from PHP to MySQL. Here is the code:

<?php
set_time_limit(0);
ini_set('memory_limit', -1);
define('NUM_INSERTS_IN_QUERY', 10000);
define('NUM_QUERIES', 100);

// build query
$time = microtime(true);
$queries =  array();
for($i = 0; $i < NUM_QUERIES; $i++){
    $queries[$i] = 'INSERT INTO `test` VALUES ';
    for($j = 0; $j < NUM_INSERTS_IN_QUERY; $j++){
            $queries[$i] .= '(1,2,3,4,5,6,7,8,9,0),';
    }
    $queries[$i] = rtrim($queries[$i], ',');
}

echo "Building query took " . (microtime(true) - $time) . " seconds\n";

mysql_connect('localhost', 'root', '') or die(mysql_error());
mysql_select_db('store') or die(mysql_error());
mysql_query('DELETE FROM `test`') or die(mysql_error());

// execute the query
$time = microtime(true);
for($i = 0; $i < NUM_QUERIES; $i++){
    mysql_query($queries[$i]) or die(mysql_error());
    // verify all rows inserted
    if(mysql_affected_rows() != NUM_INSERTS_IN_QUERY){
            echo "ERROR: on run $i not all rows inserted (" . mysql_affected_rows() . ")\n";
            exit;
    }
}

echo "Executing query took " . (microtime(true) - $time) . " seconds\n";
$result = mysql_query('SELECT count(*) FROM `test`') or die(mysql_error());
$row = mysql_fetch_row($result);
echo "Total number of rows in table: {$row[0]}\n";
echo "Total memory used in bytes: " . memory_get_usage() . "\n";
?>

The result on my Win 7 dev machine are:

Building query took 0.30241012573242 seconds
Executing query took 5.6592788696289 seconds
Total number of rows in table: 1000000
Total memory used in bytes: 22396560

So for 1 mil inserts it took 5 and a half seconds. Then I ran it with this settings:

define('NUM_INSERTS_IN_QUERY', 1);
define('NUM_QUERIES', 1000000);

which is basically doing one insert per query. The results are:

Building query took 1.6551470756531 seconds
Executing query took 77.895285844803 seconds
Total number of rows in table: 1000000
Total memory used in bytes: 140579784

Then I tried to create a file with one insert per query in it, as suggested by @jancha. My code is slightly modified:

$fid = fopen("query.sql", "w");
fputs($fid, "use store;");
for($i = 0; $i < 1000000; $i++){
    fputs($fid, "insert into `test` values (1,2,3,4,5,6,7,8,9,0);\n");
}
fclose($fid);
$time = microtime(true);
exec("mysql -uroot < query.sql");
echo "Executing query took " . (microtime(true) - $time) . " seconds\n";

The result is:

Executing query took 79.207592964172 seconds

Same as executing the queries through PHP. So, probably the fastest way is to do multiple inserts in one query and shouldn’t be a problem to use PHP to do the work.