Question

I’ve got a database that maintains Jobs to be processed by various processing machines. Its basic schema is thus:

+-------------+--------------+------+-----+---------+----------------+
| Field       | Type         | Null | Key | Default | Extra          |
+-------------+--------------+------+-----+---------+----------------+
| ID          | int(11)      | NO   | PRI | NULL    | auto_increment |
| EndTime     | datetime     | YES  |     | NULL    |                |
| GroupID     | varchar(255) | NO   | MUL | NULL    |                |
| HostAddress | varchar(15)  | YES  |     | NULL    |                |
| StartTime   | datetime     | YES  |     | NULL    |                |
+-------------+--------------+------+-----+---------+----------------+

ID is auto-increment, HostAddress represents the processing machine which has claimed this Job, StartTime represents the beginning of the most recent attempt at processing it, EndTime is the time at which it successfully completed processing, and GroupID is an arbitrary string for referencing other tables with.

All of the processing machines synchronize around this table for grabbing work. New records are only inserted manually, although all of the processing machines can update existing records. The idea was to have a processing machine do the following whenever it’s out of work:

• See if any jobs belong to it (HostAddress = its IP) and have yet to be started.
• If there are none, see if any jobs have yet to be claimed (HostAddress IS NULL).
• If there are unclaimed jobs, claim some (update HostAddress to its IP).
• Process all jobs that belong to it (same check as #1 except that we may have added some via #3).

I had thought that this sequence of operations would cause the database to synchronize different machines’ attempts at the same job for me; even if both machines tried to claim the same job at the same time, only one of their IPs would end up in the HostAddress column, so when they asked again for all jobs at their HostAddress only one of them would get that job back.

But that does not appear to be the case. When starting 35 processing machines nearly simultaneously last night, I observed multiple cases of multiple machines processing the same job, even though only one of them ended up with it claimed in the database. That implies to me that the last check isn’t working properly. Here’s the more concrete version of what I’m doing. The database calls use em.createNamedQuery which I’m just going to summarize below them for brevity. JPA is being provided by Hibernate 3.6.8 and the database is MySQL 5.1.61.

protected void poll(EntityManager em) {
    List<JobRecord> candidates = null;
    //Synchronized only for this machine. Others are running concurrently.
    synchronized (em) {
        //Check if anything is already claimed by us.
        candidates = JobRecord.selectReady(em);
        //SELECT record FROM JobRecord record WHERE HostAddress=[IP]
        //    AND StartTime IS NULL AND EndTime IS NULL;
            if (candidates.isEmpty()) {
            //None claimed. Check if any jobs aren't claimed by anyone.
            candidates = JobRecord.selectAvailable(em);
            //SELECT record FROM JobRecord record WHERE HostAddress IS NULL
            //    AND StartTime IS NULL AND EndTime IS NULL;
            if (candidates.isEmpty()) {
                //All jobs have been processed.
                return;
            }
            //Claim these jobs we found for ourselves.
            em.getTransaction().begin();
            for (JobRecord job : candidates) {
                job.setStartTime(null);
                job.setEndTime(null);
                job.setHostAddress([IP]);
                em.merge(job);
            }
            em.getTransaction().commit;
            //Only process what is actually claimed by us; could be nothing.
            candidates = JobRecord.selectReady(em);
            //(The first query again.)
        }
    //Do processing with candidates list.
}

The only explanation that comes to mind is that when I do the em.getTransaction().commit the results are cached somehow, and that when I do the selectReady NamedQuery just after it, it’s returning the cached result without bothering to consult the database. But that might not even be the case, and I’m not sure I could prove that. There might even be something fundamentally flawed with my scheme that I’m overlooking.

So, to actually pose my question, why did this database synchronization routine fail and what can I do to correct it?

Answer 1

Multiple machines can invoke selectAvailable() before any of them perform the UPDATE transaction. Consequently, they may each think that the same jobs are available.

You need to begin the transaction before the selectAvailable() call, which should use SELECT ... FOR UPDATE in order to lock the available job records so that no other database connection can read from them until the transaction is committed.