Question

I’m sending an email address to my python webapp by sms using Twilio. However, when I attempt to convert the sms body containing the email address I get an exception.

<type 'exceptions.UnicodeEncodeError'>, UnicodeEncodeError('ascii', u'myemail\xa1gmail.com', 13, 14, 'ordinal not in range(128)'), <traceback object at 0x10cca5f8>)
Traceback (most recent call last):
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 570, in dispatch
    return method(*args, **kwargs)
  File "/base/data/home/apps/s~tester-sandbox/1.360113171877023968/website/sms.py", line 22, in post
    from_body = str(cgi.escape(self.request.get('Body', None))).strip()
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa1' in position 13: ordinal not in range(128)

I get that this is an encoding error that’s stumbling on the @ character in the email address. And after googling this issue I found a recommendation of

sys.setdefaultencoding( "latin-1" )

But that simply replaced the @ with a � symbol…. What is the simplest way to overcome this?

Answer 1

I contacted Twilio to get to the bottom of this and it turns out it’s not a python problem. When a user sends in their email address (by sms) it sends in an ¡ character (instead of @) which is unfortunately non-ascii – hence the exception.

From Twilio:

Some carriers have not made the fix to allow the proper formatting to
be preserved in SMS messages sent over their networks, so you will see
corruption of @ signs in SMS handled by some carriers.

One potential fix is to enable unicode character encoding on your
Twilio account. A potential downside to this is that you would be
limited to sending SMS messages that are 70 characters or less in
length