Question

I have a database table with 10,000,000+ rows which I process to create a new table. What is the best type of storage engine for reading?

Currently I use:

$sql = "UPDATE `packages_unsorted` SET `acv_return_viadetails` = '0';";

Then to select the records:

$sql = "SELECT * FROM `packages_unsorted` WHERE `acv_return_viadetails` = '0' LIMIT 1000;";

I collect the primary key id into an array called $ids and I process the records and write the data to a CSV file for import at the end of each 1000 rows.
I then update the acv_return_viadetails field with:

$sql = "UPDATE `packages_unsorted` SET `acv_return_viadetails` = '1' WHERE `id` IN ('".implode("','",$ids)."') LIMIT 1000;";

As the above is the only write to the table and I select all fields every time. Should I be using a memory table or some other type of storage engine to make things faster? Or are there any other optimizations I could introduce that would speed things up?

Also what would be a reasonable my.cnf configuration for such an operation.

Thanks in advance 🙂

Answer 1

A few suggestions:

1) Replace

$sql = "SELECT * FROM `packages_unsorted` WHERE `acv_return_viadetails` = '0' LIMIT 1000;";

with

$sql = "SELECT `id` FROM `packages_unsorted` WHERE `acv_return_viadetails` = '0' LIMIT 1000;";

2) Check if it is possible to load all the ids at once without the 1000 limit. With 10M records, it should take only a few hundred MBs to load all the ids.

$sql = "SELECT `id` FROM `packages_unsorted`";

If not, consider increasing the limit.

3) You can try something like the following instead of using acv_return_viadetails:

$sql = "SELECT `id` FROM `packages_unsorted` LIMIT 0, 1000;";
$sql = "SELECT `id` FROM `packages_unsorted` LIMIT 1000, 1000;";
$sql = "SELECT `id` FROM `packages_unsorted` LIMIT 2000, 1000;";