Question

Greetings fellow enthusiasts:

I am debugging a program integrated with the twitter-api where the purpose is to search for specific tweets in a given radius and returns them in a csv file. It works….yet sometimes it encounters a “UnicodeEncodeError: ‘ascii’ codec can’t encode character u’\u201c’ in position 0: ordinal not in range(128)” error, yet I’ve adjusted in my code for it to ignore Unicode data from twitter outside its range.

So why is that error still occurring sometimes?

Below is the code run it with the test input given in the comments:

import urllib2, json, pprint, codecs, unicodedata, csv
## test coordinate input: 29.762778,-95.383056
## test radius 10
## test query: tebow
##Initial user input
city = raw_input("Please enter to 6 decimal places the city\ncoordinates to be searched ex. lat,long: ")
radius = raw_input("Please enter the numeric value of the\nradius in miles you'd like to search ex. 10: ")
term= raw_input("Please enter the search term you wish to query ex. tebow: ")

u = urllib2.urlopen('http://search.twitter.com/search.json?q='+term+'&geocode='+city+','+radius+'mi&page=1&rpp=20')
datares = json.load(u)
##pprint.pprint(datares)

with codecs.open('Geotweets.csv',mode='w', encoding='utf-8',errors='ignore') as cache:
    writer = csv.writer(cache)
    for tweet in datares['results']:
        writer.writerow([tweet['text'], tweet['location'], tweet['created_at'], tweet['from_user']])

Answer 1

There is a hidden conversion happening in writerow. Encode the data yourself with e.g. tweet['text'].encode('ascii', 'ignore').