Question

I’ve got a web application with approx. 30k users and growing fast. We want to enable some functionality over email, which means assigning each user a unique email address (for security reasons). Emails sent to these unique addresses should be forwarded to a PHP script. The PHP script needs access to all parts of the email – the headers, body and any attachments. I’m not sure what the right way is to set this up. We’re currently using sendmail and would prefer to stick with that (this is linux, if it matters). I’ve been able to get an individual email forwarding to a script, but it is not scalable (in my mind) to create 30k aliases and then a new one each time a new user signs up. I’d be much happier with some type of catch-all / RegEx-based solution that just tells the PHP script the unique email address that was the recipient, which would let us look-up the user.

Answer 1

30k accounts does not give us any information about the traffic you expect to handle. But let’s assume it is another question.

You could create catch all type of account, which would simply accept any email in domain you create your users’ addresses in (basically *@domain). Then you could launch your processing script on each incoming mail (you usually would be able to fetch whole mail body from stdin stream (in PHP accessible via php://stdin), and process it as you like. But this is bad if you want to use PHP (or almost any scripting language), as each mail would need to spawn new PHP process, init all the extensions and so on. It all this takes time and resources, and one bigger spam flood and your box is almost toasted.

Alternatively, you could collect your mail in mailbox as any other mail and then have your script getting it from there (either directly, digging into maildir or via mail protocols like IMAP or POP). This looks better from administration standpoint and as long as your mail server can accept incoming mails, your whole system still works. You can even try to “partition” your system by setting some filter rules (on MTA level) and deliver incoming mails not to single mailbox but more. Then you could process these mails in parallel with less hassle (no need to bother about processing same mail with more than one instance etc). “Partition” schema could be quite anything, i.e. depending on your full address naming scheme, if letter based it’d be separate mailbox for a*@, b*@ or [a-d]*@] (depends on traffic), does not really matter.

Alternatively, you could try to put bodies of incoming mails into database and then process as described above, without any need of POP/IMAP handling.

Whatever you choose I recommend NOT putting your processing script of any form directly in mail receiving process chain. It usually is not a problem to have 1 minute delay (so you can have any sort of daemon running and fetching mails for processing, cronjob etc), users can wait that much w/o complaining – it looks like instant processing is most likely not mandatory in your case as well. But if it is, then PHP might not be the best tool choice I am afraid.

EDIT: (to answer comment question and not hit comment size limit)

I am not familiar with Sendmail enough to give you precise configuration directives here (I am on customized qmail), but some general hints would apply I hope. First, I believe Sendmail supports virtual users (which means mail boxes for users not present in /etc/passwd). If so, then MTA usually should let you either query DB (to find out if the target email address is valid and where is that user’s home dir to put mail into) with some built-in providers. Or (which is even better) invoke external program to do that. The latter is a winner here because there are no users really, so you could write small script or (again, better for performance) C app that would do the trick. Since all you need is to “separate” incoming mails into several physical mailboxes based on some simple predefined rules, couple of if()s + substr()s and you are done. Couple of lines, no DB needed, pretty quick. Or, even better, you could even try the dumbest roundrobin approach here as well – it is in fact irrelevant if mail hits accountA or accountB as long as 50% of mails ends in A and other 50% in B (or 25% if you go for 4 target accounts etc) and that’s the only condition you should care I think. You can try to use procmail for that, but I think it is pointless and is not best for performance.

As for putting mails into database. It depends on MTA – by default nobody does so as it slower than putting mails into maildirs (as DB is usually other host), but there still are couple of solutions. Either look for 3rd party patches for Sendmail that do that or add another script/app that would “copy” mails from physical mailboxes into DB. You may say – but it would be slower. Sure, but again – I do not know if non-realtime processing really matters in your case (my blind guess – it does not), so why really bother. If you save couple of milliseconds here but loose minutes on further maintenance of the whole solution, then choice is quite obvious.

BTW: just in case – whenever I wrote mailbox I definitely meant container for user’s mails, NOT mailbox file format. As for file format maildir (or something derrivative) is much, much better choice.