Question

I have written a scraper that does html scraping and then use API to get some data, since its a very lengthy code I haven’t put it here. I have implemented random sleep method and using it within my code to monitor throttle. But I want to make sure I don’t over run this code, so my idea is to run for an 3-4 hours then taker breather and then run again. I haven’t done anything like this in python I was trying to search but not really sure where to start from, it would be great if I get some guidance on this. If python has a specific module link to that would be a great help.

Also is this relevant? I don’t I need this level of complication?

Suggestions for a Cron like scheduler in Python?

I have functions for every single scraping task, and I have main method calling all those functions.

Answer 1

You can use a threading.Timer object to schedule an interrupt signal to the main thread after the time is exceeded:

import thread, threading

def longjob():
    try:
        # do your job
        while True:
            print '*', 
    except KeyboardInterrupt:
        # do your cleanup
        print 'ok, giving up'

def terminate():
    print 'sorry, pal'
    thread.interrupt_main()

time_limit = 5  # terminate in 5 seconds
threading.Timer(time_limit, terminate).start()
longjob()

Put this in your crontab and run every time_limit + 2 minutes.