Question

HTTP Post may have multiple params such as:

http://example.com/controller/ (while params=({‘country’:’US’},{‘city’:’NYC’})

I am developing a web spider with Python, I face a problem how to track difference with same url with different params. Now I can load the content, but I have no idea how to store the post params in a field of SQLite3 table. It is easy to store the params in database like MySQL for system developer, but since the params of different sites are various. I prefer to store the post params in single field, rather than one-on-one relationship mapping in a table.

HTTP GET

>>> import urllib
>>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
>>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query?%s" % params)
>>> print f.read()

HTTP POST

>>> import urllib
>>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
>>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params)
>>> print f.read()

My project configuration:

• Python + SQLite3
• Store http url and post params and tracking the changes.
• The post params contains multiple key-value pairs
• The stored params should be decoded back to params.
• The encode issues should be covered.

I saw multiple solutions like JSON, XML and YAML. I guess this format actually stored as string (CHAR) type in SQLite, in UTF-8. But I have no idea if there is any handy way to convert them back to Python tuple type? Or, can I encode the post params into get params with + and & symbal, and decode it back to post params?

Sorry, I am just a newbie for Python.

Answer 1

You can convert to and from json easily like this:

>>> import json
>>> json.dumps({'spam': 1, 'eggs': 2, 'bacon': 0})
'{"eggs": 2, "bacon": 0, "spam": 1}'
>>> json.loads('{"eggs": 2, "bacon": 0, "spam": 1}')
{u'eggs': 2, u'bacon': 0, u'spam': 1}

>>> json.dumps((1,2,3,4))
'[1, 2, 3, 4]'
>>> json.loads('[1, 2, 3, 4]')
[1, 2, 3, 4]
>>> 

Better use it because it is more versatile than home made & separated encoding, it supports any nesting complexity.