Question

I’m working with Pyramid framework, trying to post UTF8 data to it.
Here the exception I got. I’m not sure how to fix this issue. Any ideas why it happents?

mod_wsgi (pid=1389): Exception occurred processing WSGI script '/home/user/transcriptions/pyramid.wsgi'.
Traceback (most recent call last):
File "/home/user/transcriptions/lib/python2.7/site-packages/pyramid-1.3.2-py2.7.egg/pyramid/router.py", line 187, in __call__
response = self.handle_request(request)
File "/home/user/transcriptions/lib/python2.7/site-packages/pyramid-1.3.2-py2.7.egg/pyramid/tweens.py", line 20, in excview_tween
response = handler(request)
File "/home/user/transcriptions/lib/python2.7/site-packages/pyramid_tm-0.5-py2.7.egg/pyramid_tm/__init__.py", line 100, in tm_tween
response = handler(request)
File "/home/user/transcriptions/lib/python2.7/site-packages/pyramid-1.3.2-py2.7.egg/pyramid/router.py", line 164, in handle_request
response = view_callable(context, request)
File "/home/user/transcriptions/lib/python2.7/site-packages/pyramid-1.3.2-py2.7.egg/pyramid/config/views.py", line 333, in rendered_view
result = view(context, request)
File "/home/user/transcriptions/lib/python2.7/site-packages/pyramid-1.3.2-py2.7.egg/pyramid/config/views.py", line 471, in _requestonly_view
response = view(request)
File "/home/user/transcriptions/transcriptions/frontend_views.py", line 179, in rpc_adduser
data = request.params
File "/home/user/transcriptions/lib/python2.7/site-packages/WebOb-1.2.1-py2.7.egg/webob/request.py", line 832, in params
params = NestedMultiDict(self.GET, self.POST)
File "/home/user/transcriptions/lib/python2.7/site-packages/WebOb-1.2.1-py2.7.egg/webob/request.py", line 783, in POST
vars = MultiDict.from_fieldstorage(fs)
File "/home/user/transcriptions/lib/python2.7/site-packages/WebOb-1.2.1-py2.7.egg/webob/multidict.py", line 74, in from_fieldstorage
obj.add(field.name, decode(field.value))
File "/home/user/transcriptions/lib/python2.7/site-packages/WebOb-1.2.1-py2.7.egg/webob/multidict.py", line 67, in <lambda>
decode = lambda b: b.decode('utf8')
File "/usr/lib/python2.6/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 6-8: invalid data

The string that it’s choking on is “Venus\xe4gen”.

I’m submitting the data from an HTML form defined with these attributes:

<form name="form_reg" id="formtrans" method="post" action="signup"
        content="text/html; charset=utf-8">

Answer 1

Ok guys, seems I have found the problem. The thing is that the form is based on the server with PHP. It uses cURL to post the data from one server to another. The second server has Pyramid app, which accepts the request. I have added iconv(‘ISO-8859-1′,’UTF-8’,$fields_string) and this get it fixed

            $fields = $_REQUEST;

            foreach($fields as $key=>$value) { $fields_string .= $key.'='.$value.'&'; }
            rtrim($fields_string,'&');                
            //open connection
            $ch = curl_init();
            curl_setopt($ch, CURLOPT_HTTPHEADER, "Content-Type: text/html; charset=utf-8");

            curl_setopt($ch, CURLOPT_URL, $url);
            curl_setopt($ch, CURLOPT_POST, count($fields));
            $fields_string = iconv('ISO-8859-1', 'UTF-8', $fields_string);
            curl_setopt($ch, CURLOPT_POSTFIELDS, $fields_string);

            $result = curl_exec($ch);
            curl_close($ch);

I’m not sure why PHP is acting like this, the $_REQUEST should be in UTF-8 as the page’s charset (meta as well) is utf-8. I’m not a big fan of PHP though. Thanks everyone for your help!