Question

I’m trying to pass big strings of random html through regular expressions and my Python 2.6 script is choking on this:

UnicodeEncodeError: ‘ascii’ codec can’t encode character

I traced it back to a trademark superscript on the end of this word: Protection™ — and I expect to encounter others like it in the future.

Is there a module to process non-ascii characters? or, what is the best way to handle/escape non-ascii stuff in python?

Thanks!
Full error:

E
======================================================================
ERROR: test_untitled (__main__.Untitled)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Python26\Test2.py", line 26, in test_untitled
    ofile.write(Whois + '\n')
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2122' in position 1005: ordinal not in range(128)

Full Script:

from selenium import selenium
import unittest, time, re, csv, logging

class Untitled(unittest.TestCase):
    def setUp(self):
        self.verificationErrors = []
        self.selenium = selenium("localhost", 4444, "*firefox", "http://www.BaseDomain.com/")
        self.selenium.start()
        self.selenium.set_timeout("90000")

    def test_untitled(self):
        sel = self.selenium
        spamReader = csv.reader(open('SubDomainList.csv', 'rb'))
        for row in spamReader:
            sel.open(row[0])
            time.sleep(10)
            Test = sel.get_text("//html/body/div/table/tbody/tr/td/form/div/table/tbody/tr[7]/td")
            Test = Test.replace(",","")
            Test = Test.replace("\n", "")
            ofile = open('TestOut.csv', 'ab')
            ofile.write(Test + '\n')
            ofile.close()

    def tearDown(self):
        self.selenium.stop()
        self.assertEqual([], self.verificationErrors)

if __name__ == "__main__":
    unittest.main()

Answer 1

You’re trying to pass a bytestring to something, but it’s impossible (from the scarcity of info you provide) to tell what you’re trying to pass it to. You start with a Unicode string that cannot be encoded as ASCII (the default codec), so, you’ll have to encode by some different codec (or transliterate it, as @R.Pate suggests) — but it’s impossible for use to say what codec you should use, because we don’t know what you’re passing the bytestring and therefore don’t know what that unknown subsystem is going to be able to accept and process correctly in terms of codecs.

In such total darkness as you leave us in, utf-8 is a reasonable blind guess (since it’s a codec that can represent any Unicode string exactly as a bytestring, and it’s the standard codec for many purposes, such as XML) — but it can’t be any more than a blind guess, until and unless you’re going to tell us more about what you’re trying to pass that bytestring to, and for what purposes.

Passing thestring.encode('utf-8') rather than bare thestring will definitely avoid the particular error you’re seeing right now, but it may result in peculiar displays (or whatever it is you’re trying to do with that bytestring!) unless the recipient is ready, willing and able to accept utf-8 encoding (and how could WE know, having absolutely zero idea about what the recipient could possibly be?!-)