Question

I came up with a very simple job queueing system using PHP, MySQL and cron.

1. Cron will call a website, which has a function that calls function A() every 2 seconds. A() searches and retrieves a row from table A
2. Upon retrieving a row, A() will update that row with value 1 in column working
3. A() then does something to the data in the retrieved row
4. A() then insert a row in table B with the value obtained during processing step 3.

Problem: I notice that there are sometimes duplicate values in the table B due to function A() retrieving the same row from table A multiple times.

Which part of the design above is allowing the duplicate processing, and how should it be fixed?

Please don’t suggest something like rabbitMQ without at least showing how it can be implemented in more details. I read some of their docs and did not understand how to implement it. Thanks!

Update: I have a cron job that calls a page (which calls function c()) every minute. This function c() that does a loop 30 times which calls function A(), using sleep() to delay.

Answer 1

One problem could be the processing at first:

Cron will call a function A() that searches and retrieves a row from table A every 2 seconds.

The processing of this part of the script could take longer than two seconds on a table without indexes as such you could pick multiple rows.

You could remedy this with an exclusive file lock.

I have a feeling there is more than just the workflow, if you can show some basic code attached maybe there might be a problem in the code as well.

edit

I think it is timing judging by your last update:

Update: I have a cron job that calls a page (which calls function c())
every minute. This function c() that does a loop 30 times which calls
function A(), using sleep() to delay.

Thats a lot of jumping through hoops and I think you might have a threading problem where crons are overlapping.