Question

I have a Flask web app, that shows information from a rss feed. I want to process the rss feed regularly, e.g every 30 minutes. Extract some of the information and store it in a sqlite db.

But I can’t figure out how to schedule a function to certain intervals.

I have used the APScheduler, and my code is the following:

def main():
    # Start the scheduler
    filename = os.path.abspath(os.path.join(os.path.dirname( __file__ ), '..', 'tmp')) + '\\' + 'spider.log'
    logging.basicConfig(filename=filename, level=logging.DEBUG,format='%(levelname)s[%(asctime)s]: %(message)s')
    sched = Scheduler()
    sched.start()
    sched.add_interval_job(run_job, minutes=30)
    time.sleep(9999)

I have a run.py function

from app import app, spider

spider.main()
app.run(debug=True)

The app.run(debug=True) starts the Flask web app. The problem is that the code never reaches app.run.

So is it possible to spawn another process to handle the spider.main() call, and run the process in the background? Or should I use another approach?

NB: I know I could use Flask-Celery, but for this small app, that seems too heavyweight…

Answer 1

You don’t need time.sleep – when you run spider.main it starts your scheduler and then puts the process to sleep for 9999 seconds – after which it will run the next line. So app.run will start ~2.78 hours after spider.main started.

So spider should look like this:

def main():
    # Start the scheduler
    filename = os.path.abspath(os.path.join(os.path.dirname( __file__ ), '..', 'tmp')) + '\\' + 'spider.log'
    logging.basicConfig(filename=filename, level=logging.DEBUG,format='%(levelname)s[%(asctime)s]: %(message)s')
    sched = Scheduler()
    sched.start()
    sched.add_interval_job(run_job, minutes=30)