Question

I’m working on a Python script that will constantly scrape data, but it will take quite a long time. Is there a safe way to stop a long running python script? The loop will run for more than 10 minutes and I need a way to stop it if I want, after it’s already running.

If I execute it from a cron job, then I’m assuming it’ll just run until it’s finished, so how do I stop it?

Also, if I run it from a browser and just call the file. I’m assuming stopping the page from loading would halt it, correct?

---

Here’s the scenario:
I have one python script that is gather info from pages and put it into a queue. Then I want to have another python script that is in an infinite loop that just checks for new items in the queue. Lets say I want the infinite loop to begin at 8am and end at 8pm. How do I accomplish this?

Answer 1

Let me present you an alternative. It looks like you want real-time updates for some kind of information. You could use a pub/sub interface (publish/subscribe). Since you are using python, there are plenty of possibilities.

One of them is using Redis pub/sub functionality: http://redis.io/topics/pubsub/ – and here is the corresponding python module: redis-py

–Update–

Example

Here is an example from dirkk0 (question / answer):

import sys
import threading

import cmd


def monitor():
    r = redis.Redis(YOURHOST, YOURPORT, YOURPASSWORD, db=0)

    channel = sys.argv[1]
    p = r.pubsub()

    p.subscribe(channel)

    print 'monitoring channel', channel
    for m in p.listen():
        print m['data']


class my_cmd(cmd.Cmd):
    """Simple command processor example."""

    def do_start(self, line):
        my_thread.start()

    def do_EOF(self, line):
        return True

if __name__ == '__main__':
    if len(sys.argv) == 1:
        print "missing argument! please provide the channel name."
    else:
        my_thread = threading.Thread(target=monitor)
        my_thread.setDaemon(True)

        my_cmd().cmdloop()

–Update 2–

In addition, look at this tutorial:

http://blog.abourget.net/2011/3/31/new-and-hot-part-6-redis-publish-and-subscribe/