Question

Requirements:

• Define “job” that has start time t, batch size s, invocation interval i and list that should be processed
• Starting with time t grab next s items from list every i seconds and process it
• Job can be paused and resumed (user should be able to tell the job to stop grabbing new list items for processing)

Flask will be used for web app. Obviously, I need some background process/thread that will periodically execute processing code.

Since the state will be in persisted to database, simplest approach that I can think of is to define a cronjob that will periodically execute python script that checks for active jobs and performs the processing.

Any suggestion on how to design this using only python?

• Start another python process that will periodically check and execute active jobs?
• Spawn a worker thread from Flask?
• … ?

Answer 1

I would strongly suggest that you employ a queuing mechanism like Redis or RabbitMQ. Flask would act as a producer and and your “worker” would consume and process.

Setting either of these tools up is far less daunting that you might expect.

sudo apt-get install redis-server
sudo apt-get install python-pip
sudo pip install redis

Your flask app acts as a producer

>>> from redis import Redis
>>> r = Redis()
>>> r.lpush('task_queue', 'task1')
1L

And your “worker” consumes and processes asynchronously:

>>> r.rpop('task_queue')
'task1'