Question

I have an email account set up that triggers a python script whenever it receives an email. The script goes through several functions which can take about 30 seconds and writes an entry into a MYSQL database.

Everything runs smoothly until a second email is sent in less than 30 seconds after the first. The second email is processed correctly, but the first email creates a corrupted entry into the database.

I’m looking to hold the email data,

msg=email.message_from_file(sys.stdin)

in a queue if the script has not finished processing the prior email.

I’m using python 2.5.
Can anyone recommend a package/script that would accomplish this?

Answer 1

I find this a simple way to avoid running a cronjob while the previous cronjob is still running.

fcntl.lockf(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) 

This will raise an IOError that I then handle by having the process kill itself.

See http://docs.python.org/library/fcntl.html#fcntl.lockf for more info.

Anyways you can easily use the same idea to only allow a single job to run at a time, which really isn’t the same as a queue (since any process waiting could potentially acquire the lock), but it achieves what you want.

import fcntl
import time
fd = open('lock_file', 'w')
fcntl.lockf(fd, fcntl.LOCK_EX)
# optionally write pid to another file so you have an indicator
# of the currently running process
print 'Hello'
time.sleep(1)

You could also just use http://docs.python.org/dev/library/multiprocessing.html#exchanging-objects-between-processes, which does exactly what you want.