Question

I am trying to write a python web app that will take some sql and a bunch of other things and return a Json file, the latter part is not the issue and I have not even put it in the script yet, the issue is that the url being passed is being utf-8 encoded and then url encoded

turning our example

query :SELECT + ;
test: 2

into

test=2&query=SELECT+%2B+%3B

This seems to be ok

but the receiving get seems to think that it can expand the codes back into chars

and it receives

test=2&query=SELECT+++;

then this is url decoded and it chops off the semicolon, and i want to keep the semicolon!
it also turns the +’s which are rightly spaces into spaces but the previous bug made the real plus code into a literal plus which turns it into a space!

{'test': '2', 'query': 'SELECT   '}

code is as follows:

#!/usr/bin/python

import web
import psycopg2
import re
import urllib
import urlparse

urls = (
    '/query', 'query',
    '/data/(.*)', 'data'
)

app = web.application(urls, globals())
render = web.template.render('templates/')

class query:
    def GET(self):
        return render.query()

    def POST(self):
        i = web.input()
        data = {}
        data['query'] = i.sql.encode('utf-8')
        data['test'] = '2'

        murl = urllib.urlencode(data)
        return "<a href=\"http://localhost:8080/data/"+murl+"\">go!</a>"

class data:
    def GET(self, urlEncodedDict):

        print "raw type:", type(urlEncodedDict)
        print "raw:", urlEncodedDict

        urlEncodedDict = urlEncodedDict.encode('ascii', 'ignore')
        print "ascii type:", type(urlEncodedDict)
        print "ascii:", urlEncodedDict

        data = dict(urlparse.parse_qsl(urlEncodedDict, 1))    #bad bit
        print "dict:", data
        print "element:", data['query']

        if ( re.match('SELECT [^;]+ ;', data['query'])):
            return 'good::'+data['query']
        else:
            return 'Bad::'+data['query']


if __name__ == "__main__":
    app.run()

Url generated from my test form is:

http://localhost:8080/data/test=2&query=SELECT+%2B+%3B

Output is as follows:

raw type: <type 'unicode'>
raw: test=2&query=SELECT+++;
ascii type: <type 'str'>
ascii: test=2&query=SELECT+++;
dict: {'test': '2', 'query': 'SELECT   '}
element: SELECT   
127.0.0.1:53272 - - [16/Nov/2012 11:05:44] "HTTP/1.1 GET /data/test=2&query=SELECT+++;" - 200 OK
127.0.0.1:53272 - - [16/Nov/2012 11:05:44] "HTTP/1.1 GET /favicon.ico" - 404 Not Found

I wish to get the same dict out of the get that i encode in the first place.

Answer 1

If you want to pass data into a GET request, you need to use the query string syntax using the question mark character [?] as a delimiter.

The URL should be:

http://localhost:8080/data/?test=2&query=SELECT+%2B+%3B

After that, you just have to use web.input() to get a dictionary with all arguments already decoded.

urls = (
    '/query', 'query',
    '/data/', 'data'
)

[...] 

class data:
    def GET(self):
        data = web.input()
        print "dict:", data
        print "element:", data['query']

        if ( re.match('SELECT [^;]+ ;', data['query'])):
            return 'good::'+data['query']
        else:
            return 'Bad::'+data['query']

Result:

dict: <Storage {'test': u'2', 'query': u'SELECT + ;'}>
element: SELECT + ;
127.0.0.1:44761 - - [16/Nov/2012 15:06:06] "HTTP/1.1 GET /data/" - 200 OK