Question

I am running a webapp (php) that needs to process some “heavy-work” tasks (from 30 sec to 3 min). I know it is not extremely heavy, but I cannot have my user waiting for them, so I’ve set up an internal API like: http://localhost/process-picture/745884/ and I store this operation in a MySQL table.

Now I want to create a “process” that fetches that MySQL table and performs the oldest queued operation, and once it is done, it get the next, and so on.

First of all I thought about making a PHP Script that calls by cURL the system like that:

fetchOperation.php connects to DB and gets the URL to the operation to call it by cURL.
Every operation: performs itself, and after that deletes itself from the queue and calls (cURL) fetchOperation.php again.

I feel that this system is kinda tricky, so I was wondering if there is (and in which language to write it) any way to set up a background process that checks the DB every 15 sec, and does the following:

• Delete all rows marked as DONE.
• Check if there is any row marked as PROCESSING, if so, just exit and wait for the next 15 sec.
• If there’s no PROCESSING row, fire the oldest PENDING one (FIFO queue).

This way I can manage what is being processed at any time, and even control the server load (for instance, at night allow to have up to three PROCESSING items).

Sorry for such a long explanation, and thanks in advance!

Answer 1

Finally I solved it that way:

1st. Design the operations in a way that they are url callable. For instance: http://localhost/render-image/14523665

2nd. Store pending operations in a table like this: operations(opertation-id, operation-url, operation-status).

Once this was ready I designed a short PHP script that does the following:

<?php
if( WORKER_IS_ONLINE !== true ) exit();
OperationsQueue::CleanQueue();
while( OperationsQueue::RunningOperations() < WORKER_MAX_OPERATIONS ) {
    launch_operation( OperationsQueue::DequeOperation() );
}
sleep( WORKER_SLEEP_TIME );
launch_operation( '/operations_queue/deque' );

This auxiliar functions (launch_operation and the OperationsQueue class) do as follows (notice it’s not yet implemented):

<?php 
define('WORKER_SLEEP_TIME', 15);     // Time to sleep between sweeps.
define('WORKER_HOST', 'localhost');  // Hostname where the operations will be performed.
define('WORKER_PORT', 80);       // Port where the operations will be performed.
define('WORKER_SOCKET_TIMEOUT', 80); // Port where the operations will be performed.
define('WORKER_MAX_OPERATIONS', 2);  // Max simultaneous operations.
define('WORKER_MAX_LAUNCHES', 2);    // Max operations launched within a sweep.
define('WORKER_IS_ONLINE', false);   // Bool value that marks whether the worker must be working or not.

function launch_operation($operation) {
    $fp = fsockopen(WORKER_HOST, WORKER_PORT, $errno, $errstr, WORKER_SOCKET_TIMEOUT);
    if ($fp) {
        $out  = 'GET ' . $operation . " HTTP/1.1\r\n";
        $out .= 'Host: ' . WORKER_HOST . "\r\n\r\n";
        fwrite($fp, $out);
        fclose($fp);
    }
}

class OperationsQueue {

    public static function RunningOperations(){
        // connect to DB an perform: SELECT COUNT(*) FROM operations WHERE status = 'PROCESSING';
        return 1;
    }
    public static function CleanQueue(){
        // connect to DB an perform: DELETE FROM operations WHERE status IN ('DONE', 'CANCELED');
    }
    public static function DequeOperation(){
        // connect to DB an perform: SELECT * FROM operations WHERE status = 'PENDING' ORDER BY id ASC LIMIT 0, 1;
        // UPDATE operations SET status = 'PROCESSING', tries = tries+1 WHERE id = $id;
        return 'operation_url';
    }

}

I thought it could be useful to some other people. As you see it chains the operations.