Question

I have some code to parse an email and find the attachments then store them into the Datastore as db.BlobProperties (might change that to Blobstore later). The problem is that when I send a UTF8 encoded text file it generates an error.

The code basically saves the file and returns a key that is converted to a string then stored in the parent email entity. As you can see I decode the file then store it as a blob. I have sent lots of attachments and this code works on everything but Text encoded with Unicode. Is there a better way to do this? What can I do to handle Unicode text attachments?

code snipit

    my_file = []
    my_list = []
    if hasattr(mail_message, 'attachments'):
        file_name = ""
        file_blob = ""
        for filename, filecontents in mail_message.attachments:
            file_name = filename
            file_blob = filecontents.decode()
            my_file.append(file_name)
            my_list.append(str(store_file(self, file_name, file_blob)))

store_file

def store_file(self, file_name, file_blob):
    new_file = myblob(file_name = file_name, 
                      file_blob = file_blob)
    return new_file.put()

I have tried using file_blob = str(file_blob) in the above to no avail. That just breaks the code and the file never gets stored.

log 1 of Unicode Text File

Property file_blob must be convertible to a Blob instance (Blob() argument should be str instance, not unicode)
Traceback (most recent call last):
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1530, in __call__
    rv = self.router.dispatch(request, response)
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1278, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1102, in __call__
    return handler.dispatch()
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 572, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 570, in dispatch
    return method(*args, **kwargs)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/webapp/mail_handlers.py", line 65, in post
    self.receive(mail.InboundEmailMessage(self.request.body))
  File "/base/data/home/apps/s~ae-baseapp/1.359073377819595139/controllers/InboundHandler.py", line 51, in receive
    file_list.append(str(store_file(self, file_name, file_blob)))
  File "/base/data/home/apps/s~ae-baseapp/1.359073377819595139/models/MyModel.py", line 63, in store_file
    file_blob = file_blob)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 974, in __init__
    prop.__set__(self, value)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 614, in __set__
    value = self.validate(value)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 2780, in validate
    (self.name, self.data_type.__name__, err))
BadValueError: Property file_blob must be convertible to a Blob instance (Blob() argument should be str instance, not unicode)

Log 2 of removing the filecontents.decode() and replacing it with just filecontents.

Property file_blob must be convertible to a Blob instance (Blob() argument should be str instance, not EncodedPayload)
Traceback (most recent call last):
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1530, in __call__
    rv = self.router.dispatch(request, response)
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1278, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 1102, in __call__
    return handler.dispatch()
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 572, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.1/webapp2.py", line 570, in dispatch
    return method(*args, **kwargs)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/webapp/mail_handlers.py", line 65, in post
    self.receive(mail.InboundEmailMessage(self.request.body))
  File "/base/data/home/apps/s~ae-baseapp/1.359097282640216691/controllers/InboundHandler.py", line 57, in receive
    file_list.append(str(store_file(self, file_name, file_blob)))
  File "/base/data/home/apps/s~ae-baseapp/1.359097282640216691/models/MyModel.py", line 64, in store_file
    file_blob = file_blob)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 974, in __init__
    prop.__set__(self, value)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 614, in __set__
    value = self.validate(value)
  File "/base/python27_runtime/python27_lib/versions/1/google/appengine/ext/db/__init__.py", line 2780, in validate
    (self.name, self.data_type.__name__, err))
BadValueError: Property file_blob must be convertible to a Blob instance (Blob() argument should be str instance, not EncodedPayload)

Answer 1

Attachment payloads are instances of the EncodedPayload class. Attachments have an encoding and an optional character set. The former refers to transfer encodings such as base64; the latter to character encodings such as UTF-8 (character set’s a bit of an outdated and misleading term here). The EncodedPayload.decode() method decodes both transfer encoding and text encoding, which as you’ve noticed is not very helpful if you just want to get the original bytes the user attached to their message.

There’s a number of approaches you can take here, but what I’d recommend is duplicating EncodedPayload’s logic for decoding transfer encoding, which looks something like this:

if filecontents.encoding and filecontents.encoding.lower() != '7bit':
  try:
    payload = filecontents.payload.decode(filecontents.encoding)
  except LookupError:
    raise UnknownEncodingError('Unknown decoding %s.' % filecontents.encoding)
  except (Exception, Error), e:
    raise PayloadEncodingError('Could not decode payload: %s' % e)
else:
  payload = filecontents.payload

Note that if the attachment was text, you need to include the character encoding when you store it, or there’ll be no way to interpret it when you send it back to the user – the original text could have been encoded using any character encoding.

Likewise, you should also save the mimetype of the attachment if you can, but this doesn’t appear to be exposed anywhere in the API. You might want to consider avoiding using the IncomingMessage class at all, and instead decoding the body of the POST request using Python’s mime message module.