Question

When running a batch import script in Python (openblock), I’m getting the following invalid byte sequence for encoding “UTF8”: 0xca4e error for an accented character:

It shows up as:
GRAND-CH?NE, COUR DU

But is actually “GRAND-CHÊNE, COUR DU”

What is the best way to handle this? Ideally I’d like to keep the accented character. I suspect I need to encode it somehow?

Edit: the ? is actually supposed to be Ê. Also note that the variable is coming from an ESRI Shapefile. When I try davidcrow’s solution, I get “Unicode not supported”, because presumably the strings that don’t have accented characters are already Unicode strings.

Here’s the ESRIImporter code I’m using:

from django.contrib.gis.gdal import DataSource

class EsriImporter(object):
    def __init__(self, shapefile, city=None, layer_id=0):
        print >> sys.stderr, 'Opening %s' % shapefile
        ds = DataSource(shapefile)

        self.layer = ds[layer_id]
        self.city = "OTTAWA" #city and city or Metro.objects.get_current().name
        self.fcc_pat = re.compile('^(' + '|'.join(VALID_FCC_PREFIXES) + ')\d$')

    def save(self, verbose=False):
        alt_names_suff = ('',)
        num_created = 0
        for i, feature in enumerate(self.layer):
            #if not self.fcc_pat.search(feature.get('FCC')):
            #    continue
            parent_id = None
            fields = {}
            for esri_fieldname, block_fieldname in FIELD_MAP.items():
                value = feature.get(esri_fieldname)
                #print >> sys.stderr, 'Looking at %s' % esri_fieldname

                if isinstance(value, basestring):
                    value = value.upper()
                elif isinstance(value, int) and value == 0:
                    value = None
                fields[block_fieldname] = value
            if not ((fields['left_from_num'] and fields['left_to_num']) or
                    (fields['right_from_num'] and fields['right_to_num'])):
                continue
            # Sometimes the "from" number is greater than the "to"
            # number in the source data, so we swap them into proper
            # ordering
            for side in ('left', 'right'):
                from_key, to_key = '%s_from_num' % side, '%s_to_num' % side
                if fields[from_key] > fields[to_key]:
                    fields[from_key], fields[to_key] = fields[to_key], fields[from_key]
            if feature.geom.geom_name != 'LINESTRING':
                continue
            for suffix in alt_names_suff:
                name_fields = {}
                for esri_fieldname, block_fieldname in NAME_FIELD_MAP.items():
                    key = esri_fieldname + suffix
                    name_fields[block_fieldname] = feature.get(key).upper()
                    #if block_fieldname == 'postdir':
                        #print >> sys.stderr, 'Postdir block %s' % name_fields[block_fieldname]


                if not name_fields['street']:
                    continue
                # Skip blocks with bare number street names and no suffix / type
                if not name_fields['suffix'] and re.search('^\d+$', name_fields['street']):
                    continue
                fields.update(name_fields)
                block = Block(**fields)
                block.geom = feature.geom.geos
                print repr(fields['street'])
                print >> sys.stderr, 'Looking at block %s' % unicode(fields['street'], errors='replace' )

                street_name, block_name = make_pretty_name(
                    fields['left_from_num'],
                    fields['left_to_num'],
                    fields['right_from_num'],
                    fields['right_to_num'],
                    '',
                    fields['street'],
                    fields['suffix'],
                    fields['postdir']
                )
                block.pretty_name = unicode(block_name)
                #print >> sys.stderr, 'Looking at block pretty name %s' % fields['street']

                block.street_pretty_name = street_name
                block.street_slug = slugify(' '.join((unicode(fields['street'], errors='replace' ), fields['suffix'])))
                block.save()
                if parent_id is None:
                    parent_id = block.id
                else:
                    block.parent_id = parent_id
                    block.save()
                num_created += 1
                if verbose:
                    print >> sys.stderr, 'Created block %s' % block
        return num_created

Output:

'GRAND-CH\xcaNE, COUR DU'
Looking at block GRAND-CH�NE, COUR DU
Traceback (most recent call last):

  File "../blocks_ottawa.py", line 144, in <module>
    sys.exit(main())
  File "../blocks_ottawa.py", line 139, in main
    num_created = esri.save(options.verbose)
  File "../blocks_ottawa.py", line 114, in save
    block.save()
  File "/home/chris/openblock/src/django/django/db/models/base.py", line 434, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)
  File "/home/chris/openblock/src/django/django/db/models/base.py", line 527, in save_base
    result = manager._insert(values, return_id=update_pk, using=using)
  File "/home/chris/openblock/src/django/django/db/models/manager.py", line 195, in _insert
    return insert_query(self.model, values, **kwargs)
  File "/home/chris/openblock/src/django/django/db/models/query.py", line 1479, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/home/chris/openblock/src/django/django/db/models/sql/compiler.py", line 783, in execute_sql
    cursor = super(SQLInsertCompiler, self).execute_sql(None)
  File "/home/chris/openblock/src/django/django/db/models/sql/compiler.py", line 727, in execute_sql
    cursor.execute(sql, params)
  File "/home/chris/openblock/src/django/django/db/backends/util.py", line 15, in execute
    return self.cursor.execute(sql, params)
  File "/home/chris/openblock/src/django/django/db/backends/postgresql_psycopg2/base.py", line 44, in execute
    return self.cursor.execute(query, args)

django.db.utils.DatabaseError: invalid byte sequence for encoding "UTF8": 0xca4e
HINT:  This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".

Answer 1

More information please. What platform – Windows / Linux / ???

What version of Python?

If you are running Windows, your encoding is much more likely to be cp1252 or similar than ISO-8859-1. It’s definitely not UTF-8.

You will need to: (1) Find out what your input data is encoded with. Try cp1252; it’s the usual suspect. (2) decode your data into unicode (3) encode it into UTF-8.

How are you getting the data out of your ESRI shapefile? Show your code. Show the full traceback and error message. To avoid visual problems (it’s E-grave! no, it’s E-acute!) print repr(the_suspect_data) and copy/paste the result into an edit of your question. Go easy on the bold type.