I have a simple script that pulls about 100 million results, 1 million at

Question

0

Asked: June 1, 20262026-06-01T07:09:02+00:00 2026-06-01T07:09:02+00:00

I have a simple script that pulls about 100 million results, 1 million at

0

I have a simple script that pulls about 100 million results, 1 million at a time from a remote MySQL server and updates my local database. The remote server isn’t under my control so doing some kind of master/slave setup for automatic updates isn’t possible. It takes a few hours to execute normally, but I realized that the portion of code that pulls the results from the remote server and the portion of code that updates the local entries take roughly the same amount of time to execute. So, theoretically I could cut execution time nearly in half if I fetch the next batch of rows from the remote server while I am updating the local entries. I’ve seen some example code for parallel database queries in PHP (for example, question 322275), but I can’t think of a way to get it to work with the foreach update loop in my script. Is this possible in PHP or would I be better off doing this is another language like Python or Ruby (Although I don’t have experience with either)?

Edit: Also the remote server is half way around the world so the network latency is fairly high.

Sample code:

for ($i = 0; $i < 100; $i++)
{
    $min = $i * LIMIT;
    $max = ($i + 1) * LIMIT;

    $updates = $db_remote->fetchAll("SELECT * FROM table_name WHERE content_id >= $min AND content_id < $max"); // Pull 1 million results from the remote SQL server

    foreach ($updates as $u)
    {
        $db_local->update("UPDATE table_name SET ... WHERE content_id = $u['content_id'] LIMIT 1"); // Update each of the local entries
    }
}