Question

Just for instance, this code is working

import urllib

image = urllib.URLopener()
file_ = 1
name = 1
for i in range(1,1000):
    try:
        image.retrieve("http://mangawriter.com/pics/pic"+str(file_)+".jpeg","pic"+str(name)+".jpeg")
        print "save file %s" %file_ 
        name += 1
        file_ += 1
    except IOError:
        file_ += 1

How could I make it stop after some time was spent, even if the code is still being ran? Please, help me to figure it out.

Answer 1

I would use the multiprocessing module, which generates new processes (not threads) for parallelizing tasks.

How to do it? First, the actual download code should be put in a function:

import urllib
import multiprocessing
import time

def download_images():
    image = urllib.URLopener()
    file_ = 1
    name = 1
    for i in range(1,1000):
        try:
            image.retrieve("http://mangawriter.com/pics/pic"+str(file_)+".jpeg","pic"+str(name)+".jpeg")
            print "save file %s" %file_ 
            name += 1
            file_ += 1
        except IOError:
            file_ += 1

Now, we create a new multiprocessing.Process object, passing the function above as its target. This object will start a process just to execute this function:

downloader = multiprocessing.Process(target=download_images)

Once we have created the process object, just call its start() method. This will start a process that will run in parallel:

downloader.start()

Since it is running in parallel, the main program keeps executing. Now, we define a timeout and sleep for the time of this timeout. In the example below, the timeout is 15 seconds:

timeout = 15
time.sleep(timeout)

Once the timeout ends, just terminate the downloader process:

downloader.terminate()

The full program can be found here.