Question

Working on a project that requires incoming email to be parsed, and certain information be extracted and stored in the database. We’re using postmarkapp to extract the body content of the email so we only have the text only guts of it, but I’m currently a bit stuck on how to parse the email the most efficient way.

Over time we’ll be adding more ‘accepted’ formats of incoming mail, but to start off with we’ll have probably 4 common emails coming in, that is, they’ll follow the same format and the information that we want to extract (contact details, id’s, links, bio) will be in the same place, (per supported format).

I’m thinking that we’ll have an interface that will handle the common tasks, and each supported format will implement that, however just how to get that information is where I’m stuck.

Open to any thoughts and ideas on different methods / technologies to do this, ideally PHP, but if we need to use something else, that’s fine.

Answer 1

There is a similar feature on a site that I developed. Our users get emails from their suppliers with pricing. They copy and paste the body of the email into a textarea on our site and click a button. Then we parse the text to find products and prices and stick the info into a database.

To do the parsing, we first have to determine the supplier, like you’ll need to do to determine which template was used. We look for certain strings in the text – the supplier’s name usually, or a line that’s unique to their emails. We do that in a method called something like getParserForText(). That method returns a Parser object which implements a simple interface with a parseText() method.

There’s a Parser implementation class for each format. The parseText() method in each class is responsible for getting the data out of the text. We looked for ways of making these elegant and generic and have simply not found a really good way to do that. We’re using a combination of regular expressions, splitting the string into smaller sections, and walking through the string.

Pseudocode:

$text = $_POST['emailBody'];
$parser = getParserForText($text);
$result = $parser->parseText($text);
if(count($result["errors"]) > 0)
{
    // handle errors
}
else
{
    saveToDatabase($result["prices"]);
}

We have no control over the formats the suppliers use, so we have to resort to things like:

1. split the text into an array of strings around each line with a date (prey_split())

2. for each element in that array, the first line contains the date, the next three to six lines contain products and prices

3. pull the date out and then split the string on new lines

4. for each line, use a regex to find the price ($000.0000) and pull it out

5. trim the rest of the line to use as the product name

We use a lot of prey_split(), preg_match_all() and explode(). While it doesn’t seem to me to be particularly elegant or generic, the system has been very robust. By leaving a little wiggle room in the regular expressions, we’ve made it through a number of small format changes without needing to change the code. By “wiggle room” I mean things like: Don’t search for a space, search for any whitespace. Don’t search for a dollar sign and two numbers, search for a dollar sign and any number of numbers. Little things like that.

EDIT:

Here’s a question I asked about it a few years ago:
Algorithms or Patterns for reading text