Question

I have a dilemma, maybe you can help me.

I have a table which functions as a work queue. Records are inserted and need to be processed. After a record is processed, it is deleted from the queue.
There are a few restrictions :

• only one entity can process a record at any given time (and by “entity”, I mean : a thread, or a separate computer which connects to the same database)
• entities are somewhat dynamic. They might change (either number of entities, or characteristics)
• an entity processes a record in 1 transaction
• processing must happen in parallel (if entity1 picks batch1, entity2 must be able to process batch2 in parallel, without waiting for entity1 to finish processing)
• once an entity has picked a record to process, the whole “batch” of records this one belongs to, must not be picked by other entity. When I say “batch”, I mean that the table is (logically) organized as follows :
  • row1 (batch1)
  • row2 (batch1)
  • row3 (batch2)
  • row4 (batch2)
  • row5 (batch2)
  • …. and so on.

So lets say entity1 and entity2 both want to pick a processing slice from the table.
If entity1 picks row1, then entity2 can pick anything else except batch1 (anything else except row1 and row2).

Lets abstract out the processing part, because it doesn’t matter what the actual processing is. I’m interested to know how I can stop the entities from clashing with each other, using only a mysql database, but also keeping the parallel nature of the processing.

From my point of view, I see two very general directions :

1. Use some sort of status field, which indicates that a particular entity has picked a batch, and this one has to be excluded from future picks. This idea has the disadvantage that if the entity that picked the batch, crashes, then it’s a bit difficult to resume the processing by other entities.
2. Using mysql locks, which has the disadvantage that it’s difficult to ensure parallel processing, and not sequential. For example I could do a select… for update, for entity1. But entity2 cannot do the same select… for update, because this would wait for the first entity to finish processing before acquiring the batch it needs.

I’m interested to know :

• which direction would result in the smallest coding effort
• are there any other directions I’m missing here (keeping in mind that the entities cannot communicate with each other except through database)
• if there is a standard pattern for this kind of problem
• if you can point me to an article debating this kind of problem.
• what is the most efficient way to solve this problem.

So what I have here is that the database must split a table between different entities, for processing, and would like to know the best way to do it. I hardly think I’m the first one dealing with this problem, and would like to know what you think.
Also, please note that the records can be split in batches through a fairly simple criteria (say, a batchId)

Kind regards,
Andrei.

Answer 1

Hmm, from what you describe, I don’t see how db record locking will do what you want. Db locks say “if this record isn’t available, wait”, not “if this record isn’t available, pick another”.

Question: When a batch is processed, is it then “done”? I mean, suppose node A starts processing batch 1. Then node B comes along, and however you do it, figures out that batch 1 is in-work and so it starts on batch 2. Then node A finishes. Then node C comes along. Batch 1 is not currently being processed. Should node C get batch 1? Or is batch 1 done, batch 2 in work, and so we must go on to batch 3? If a batch is done, then any db locking is clearly not going to work, as it doesn’t remember that a record was accessed in the past, only what’s happenning now.

Maybe someone else has a more clever solution, but I think you’re going to have to do this with status flags. I’d say:

Create a separate Batch table, with one record per batch. Put a foreign key in the Work-queue table that points to the Batch table. That way we keep the data normalized.

In the batch table, add a status flag with values for in-work and not-in-work; or unprocessed, in-work, and processed. (Depending on whether batches get “done” once and for all.) Also create a “processing entity” field that identifies what entity is processing this batch if it is in-work, null otherwise. (If the only statuses are in-work and not-in-work, than this field can double as the in-work flag: null=not in-work, not null=in-work.)

When an entity starts processing a batch, it sets the processing-entity field to the entity id. When it is finished, it resets the field to null. When an entity is looking for a batch to process, it selects “where processing_entity is null” (along with whatever other conditions, of course).

Then to make sure an entity hasn’t crashed leaving the flag set, every time an entity starts up, it checks if there is a Batch record indicating that it is processing it. If so, it must have crashed, so it resets the flag and does whatever clean-up.

That only works if the set of entities is fixed, e.g. if an entity is a server, or if it is a thread that comes from a thread pool. It wouldn’t work if they’re threads that can come and go arbitrarily, because you couldn’t count on the thread restarting to tell that it left unfinished work.

Another method I’ve used is to put a timestamp in the Batch record that says when we started processing. Then I’ve had another process that runs every now and then, checks the timestamps, and compares to some reasonable maximum time. If, say, we know that the process should take a few seconds and we see one with a timestamp that’s an hour old, then the process either died or hung, and we should do clean-up appropriately. This does have the catch that it requires you to be able to say what a maximum time is.