Question

How can I distinguish sender-generated carriage returns from word wrap auto-generated carriage returns in an email body? I’m using Python imaplib to access Gmail and download message bodies like so:

user='[e-mail]'
pwd='password'
m = imaplib.IMAP4_SSL("imap.gmail.com")
m.login(user,pwd)
m.select("INBOX")
resp, items = m.search(None, "ALL")
items = items[0].split()
messages = []
for emailid in items:
    resp, data = m.fetch(emailid, "(RFC822)")
    email_body = data[0][1]
    mail = email.message_from_string(email_body)
    for part in mail.walk():
        if part.get_content_type() == 'text/plain':
            body = part.get_payload(decode=1)
            messages.append(body)

I’m focusing on the case of messages received from another Gmail user. The message body text has a number of carriage returns (‘\r\n’) in it. These fall into two classes: 1) those inserted by the sender of the email, the “true” returns, 2) those created by Gmail word wrapping at ~78 characters, the “false” returns. I want to remove the second class of carriage returns only. I’m sure I could come up with a programmatic approximation that searches for the ‘\r\n’ at a window around every 78th character but that wouldn’t be bulletproof and isn’t what I want. Interestingly, I notice that when the message displays in Gmail in the web browser, there are not returns for the second class of carriage returns. Gmail somehow knows to remove/not display these specifically. How? Is there some special encoding I’m missing?

Answer 1

Gmail sends messages in both the MIME multipart format, in both a text/plain version (what you are grabbing) and a text/html version. The latter version is what contains fancy formatting like bold, italic, links, etc., and is what Gmail displays. While the text/html version is also line-broken at 78 characters (a part of the e-mail standard — the underlying text must never have a line exceeding 78 characters), the “real” line breaks that you are looking for are embedded therein as HTML <br> tags. You can see this yourself if you send yourself a message and then, using the little down-arrow next to the Reply button, click “Show original”.

You cannot distinguish between “fake” and “real” line-breaks in the text/plain version of the message, at least not reliably (as you obviously know). You can, however, pull the text/html version instead, knowing then that the “real” line-breaks are the <br> tags, however you then have to deal with the additional HTML (as well as first correctly processing the “Content-Transfer-Encoding” used therein).