Question

I need to make HTTP and HTTPS requests using POST, GET and other methods and specifying headers and timeouts.

On the Internet there are many examples, and they are all different:

import urllib.parse
import urllib.request

url = 'http://www.someserver.com/cgi-bin/register.cgi'
user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'
values = {'name' : 'Michael Foord',
          'location' : 'Northampton',
          'language' : 'Python' }
headers = { 'User-Agent' : user_agent }

data = urllib.parse.urlencode(values)
req = urllib.request.Request(url, data, headers)
response = urllib.request.urlopen(req)
the_page = response.read()

or

    fetcher = urllib2.build_opener()
    fetcher.addheaders.append(('Cookie', 'aaaa=%s' % aaaa))
    res = fetcher.open(settings.ABC_URL)

or

req = urllib2.Request(url=url)
req.add_header('X-Real-IP', request.META['REMOTE_ADDR'])
req.add_header('Cookie', request.META['HTTP_COOKIE'])
req.add_header('User-Agent', request.META['HTTP_USER_AGENT'])
resp = urllib2.urlopen(req).read()

or

handler = urllib.urlopen('http://...')
response = handler.read()

I guess under the hood some of these methods use the same mechanism.

There also other examples using httplib, httplib2. And i read urllib2 is the preferred lib to use.

Which one is preferred method? Which advantages and disadvantages each of them has?

Answer 1

Try out the requests-module which fixes the URL/download library madness in Python.