Question

We have a system that has a database based queue for processing items in threads instead of real time. It’s currently implemented in Mybatis calling a this stored procedure in mysql:

DROP PROCEDURE IF EXISTS pop_invoice_queue;
DELIMITER ;;
CREATE PROCEDURE pop_invoice_queue(IN compId int(11), IN limitRet int(11)) BEGIN

   SELECT LAST_INSERT_ID(id) as value, InvoiceQueue.* FROM InvoiceQueue 
      WHERE companyid = compId 
      AND (lastPopDate is null OR lastPopDate < DATE_SUB(NOW(), INTERVAL 3 MINUTE)) LIMIT limitRet FOR UPDATE;
   UPDATE InvoiceQueue SET lastPopDate=NOW() WHERE id=LAST_INSERT_ID(); 

END;;

DELIMITER ;

The problem is that this pops N items from the queue but only updates the lastPopDate value for the last item popped off the queue. So if we call this stored procedure with limitRet = 5, it will pop five items off the queue and start working on them but only the fifth item will have a lastPopDate set so when the next thread comes and pops off the queue it will get items 1-4 and item 6.

How can we get this to update all N records ‘popped’ off the database?

Answer 1

If you are willing to add a BIGINT field to the table via:

ALTER TABLE InvoiceQueue
ADD uuid BIGINT NULL DEFAULT NULL,
INDEX ix_uuid (uuid);

then you can do the update first, and select the records updated, via:

CREATE PROCEDURE pop_invoice_queue(IN compId int(11), IN limitRet int(11))
BEGIN
   SET @uuid = UUID_SHORT();

   UPDATE InvoiceQueue
   SET    uuid = @uuid,
          lastPopDate = NOW()
   WHERE  companyid = compId
   AND    uuid IS NULL 
   AND    (lastPopDate IS NULL OR lastPopDate < NOW() - INTERVAL 3 MINUTE)
   ORDER BY
          id
   LIMIT  limitRet;

   SELECT * 
   FROM   InvoiceQueue 
   WHERE  uuid = @uuid
   FOR    UPDATE;
END;;

For the UUID_SHORT() function to return unique values, it should be called no more than 16 million times a second per machine. Visit here for more details.

For performance, you may want to alter the lastPopDate field to be NOT NULL as the OR clause will cause your query to not use an index, even if one is available:

ALTER TABLE InvoiceQueue
MODIFY lastPopDate DATETIME NOT NULL DEFAULT '0000-00-00';

Then, if you do not already have one, you could add an index on the companyid/lastPopDate/uuid fields, as follows:

ALTER TABLE InvoiceQueue
ADD INDEX ix_company_lastpop (companyid, lastPopDate, uuid);

Then you can remove the OR clause from your UPDATE query:

   UPDATE InvoiceQueue
   SET    uuid = @uuid,
          lastPopDate = NOW()
   WHERE  companyid = compId 
   AND    lastPopDate < NOW() - INTERVAL 3 MINUTE
   ORDER BY
          id
   LIMIT  limitRet;

which will use the index you just created.