Question

I’m learning Django. There is a search bar. That is intended to be used with russian words.
The word is caught with a simple form.

How can I encode the word to utf-8 right at the beginning ? It creates the UnicodeEncodeError at some point because of this.
Or what are the ways to change the original request?

The database is set to utf-8.

Thank you.

Here is the Traceback

Traceback:
File "/usr/local/lib/python2.6/dist-packages/django/core/handlers/base.py" in get_response
  111.                         response = callback(request, *callback_args, **callback_kwargs)

File "/home/teodor/moldova/search/views.py" in results
  27.   return render_to_response(template_name, locals(), context_instance = RequestContext(request))

File "/usr/local/lib/python2.6/dist-packages/django/shortcuts/__init__.py" in render_to_response
  20.     return HttpResponse(loader.render_to_string(*args, **kwargs), **httpresponse_kwargs)

File "/usr/local/lib/python2.6/dist-packages/django/template/loader.py" in render_to_string
  176.         return t.render(context_instance)

File "/usr/local/lib/python2.6/dist-packages/django/template/base.py" in render
  140.             return self._render(context)

File "/usr/local/lib/python2.6/dist-packages/django/template/base.py" in _render
  134.         return self.nodelist.render(context)

File "/usr/local/lib/python2.6/dist-packages/django/template/base.py" in render
  823.                 bit = self.render_node(node, context)

File "/usr/local/lib/python2.6/dist-packages/django/template/debug.py" in render_node
  74.             return node.render(context)

File "/usr/local/lib/python2.6/dist-packages/django/template/loader_tags.py" in render
  123.         return compiled_parent._render(context)

File "/usr/local/lib/python2.6/dist-packages/django/template/base.py" in _render
  134.         return self.nodelist.render(context)

File "/usr/local/lib/python2.6/dist-packages/django/template/base.py" in render
  823.                 bit = self.render_node(node, context)

File "/usr/local/lib/python2.6/dist-packages/django/template/debug.py" in render_node
  74.             return node.render(context)

File "/usr/local/lib/python2.6/dist-packages/django/template/loader_tags.py" in render
  123.         return compiled_parent._render(context)

File "/usr/local/lib/python2.6/dist-packages/django/template/base.py" in _render
  134.         return self.nodelist.render(context)

File "/usr/local/lib/python2.6/dist-packages/django/template/base.py" in render
  823.                 bit = self.render_node(node, context)

File "/usr/local/lib/python2.6/dist-packages/django/template/debug.py" in render_node
  74.             return node.render(context)

File "/usr/local/lib/python2.6/dist-packages/django/template/loader_tags.py" in render
  62.             result = block.nodelist.render(context)

File "/usr/local/lib/python2.6/dist-packages/django/template/base.py" in render
  823.                 bit = self.render_node(node, context)

File "/usr/local/lib/python2.6/dist-packages/django/template/debug.py" in render_node
  74.             return node.render(context)

File "/usr/local/lib/python2.6/dist-packages/django/template/loader_tags.py" in render
  62.             result = block.nodelist.render(context)

File "/usr/local/lib/python2.6/dist-packages/django/template/base.py" in render
  823.                 bit = self.render_node(node, context)

File "/usr/local/lib/python2.6/dist-packages/django/template/debug.py" in render_node
  74.             return node.render(context)

File "/usr/local/lib/python2.6/dist-packages/django/template/base.py" in render
  1178.                     _dict = func(*resolved_args, **resolved_kwargs)

File "/home/teodor/moldova/search/templatetags/search_tags.py" in pagination_links
  23.   params = urllib.urlencode(raw_params)

File "/usr/lib/python2.6/urllib.py" in urlencode
  1267.             v = quote_plus(str(v))

Exception Type: UnicodeEncodeError at /search/results/
Exception Value: 'ascii' codec can't encode characters in position 0-4: ordinal not in range(128)

I did a search for word: Найти

Answer 1

I don’t have much experience with this, but as per Jonathan “Wolf” Rentzsch’s guide to using UTF-8 on the web, you might want to try adding the accept-charset="UTF-8" attribute to your <form> tag.

Where are you seeing the UnicodeEncodeError? Can you show us some code?

Edit: ah, okay. Yup — I think Django provides all text as Unicode objects. If I’m understanding the traceback correctly, the issue occurs when you call str on this unicode object to use it with urllib.quote_plus.

I think you need to provide urllib.quote_plus with text encoded in ASCII (See e.g. Is there a unicode-ready substitute I can use for urllib.quote and urllib.unquote in Python 2.6.5?).

And as per this question, I think you can do that like this:

import unicodedata
v_as_ascii = unicodedata.normalize('NFKD', v.decode('UTF-8')).encode('ascii', 'ignore')
v = quote_plus( v_as_ascii )

(I haven’t tested any of this though, so I could be entirely wrong. Hopefully someone who has experience with this stuff will come by.)