Hoping someone has some suggestions or work-arounds.
I recently wrote an email parser to work for mail forwarded through a pipe in CPanel and have the details entered in a mySQL table.
The client now wants historical mail importing to this database and has approx 50 Gmail accounts with some having over 20,000 messages in.
I have been able to get up to around 7000 emails out of 25000 on the ‘test’ account I’m using. To try and work out why I’ve stripped it down so there’s a very simple php imap script running just to dump the a couple of parts of the email in to a cell in a table and was going to worry about parsing it after, but the same thing happens. It hits around the 6000-7000 email mark and stops – no errors or anything, it just stops.
I’ve just found out that Gmail has a bandwidth limit for IMAP of 750MB per hour and doing the math it looks quite likely I’m hitting that and that’s why it’s stopping.
First question – anyone know how I can check if that’s the reason?
Secondly – assuming it is that can anyone think of a best solution?
I considered doing it in blocks of a few thousand emails but this would mean a lot of manual intervention because of the number of accounts and it would take a LONG time to download them all which I don’t have.
The second (and so far best) idea I’ve had is to use one of Gmail’s recommended methods to migrate all the mail to an email address created on the client’s Cpanel and use the inbuilt pipe feature to forward the mail through my script in the same way I’m doing with their new mail. (I’m not 100% sure the migration will trigger CPanel to apply the pipe rule, I’d have to test).
Appreciate any input.
thanks in advance.
I don’t have an answer to your first question.
But as far as your second question goes you don’t necessarily have to re-run your program manually if you split the task up into a couple thousand emails. You could either use sleep to stop the program for an hour after x number of emails have been pulled over, or you could setup a cron job that runs once per hour and you would just have to save a pointer to the last email you processed in a database or temp file so you know where to pick up and start again next cron job run.