Question

The only reliable method that I a have found for using a script to download text from wikipedia is with cURL. So far the only way I have for doing that is to call os.system(). Even though the output appears properly in the python shell I can’t seem to the function it to return anything other than the exit code(0). Alternately somebody could show be how to properly use urllib.

Answer 1

From Dive into Python:

import urllib sock = urllib.urlopen('http://en.wikipedia.org/wiki/Python_(programming_language)') htmlsource = sock.read() sock.close() print htmlsource 

That will print out the source code for the Python Wikipedia article. I suggest you take a look at Dive into Python for more details.

Example using urllib2 from the Python Library Reference:

import urllib2 f = urllib2.urlopen('http://www.python.org/') print f.read(100) 

Edit: Also you might want to take a look at wget.
Edit2: Added urllib2 example based on S.Lott’s advice