Question

I have a function that is calling wget via subprocess.Popen. The purpose of this function is to spawn wget and spider a website for a list of links.

Is it possible to tell when the wget process has completed and then continue executing the rest of the python function e.g.

def get_urls(url, uname, pword, output):
    subprocess.Popen (['wget', '-nd', '-r', '--user=', uname, '--password=', pword,  
    '--no-parent','--spider',url, '--output-file= ',output], stdout=subprocess.PIPE)

    #some method telling wget has finished writing to the output file, so continue

    foo = bar() #rest of function etc.

Also is there a better method of spidering a site (and passing in login credentials) via python rather than making system calls?

Thanks

Answer 1

Why you want use subprocess, may be better use urllib

import urllib

url = 'http:......'
filename = 'your_filename'
urllib.urlretrieve(url, filename)