Question

The JSON data I’m getting back from an API call has a character in it \x96

It’s causing the following error when I make the API call:

'ascii' codec can't encode character u'\x96' in position 56: ordinal not in range(128)

This happens occasionally with the API calls — maybe 5% of the time when there is I guess this unicode data it can’t decode. Any idea how to decode these characters?

Here’s my code:

    start = str(start)
    limit = str(limit) 
    if sort_by:
        url = self.base_url + specific_url + "?q=" + str(query) + "&filters=[[%22category_id%22,[" + category_id + "]]]" + "&start=" + start  + "&limit=" + limit + "&sortby=" + sort_by + "&apikey=" + self.api_key 
    else:
        url = self.base_url + specific_url + "?q=" + str(query) + "&filters=[[%22category_id%22,[" + category_id + "]]]" + "&start=" + start  + "&limit=" + limit + "&apikey=" + self.api_key

    response_json = self.web_fetch(url)
    return simplejson.loads(response_json)

Answer 1

simplejson.loads must take a unicode object for this to work.

This means that you need to amend the web_fetch() function to return a Unicode object, or you can just decode response_json as UTF8 if you know that that is its encoding.

response_json = response_json.decode('utf-8')