Question

I have a very long running script that is doing a pretty significant amount of work with about 30 million database records. Every time I start it with the CLI and let it run, it runs fairly quickly given the amount of work it’s doing (about 5k records/minute). However, after about 90 minutes of this, it slows down dramatically, taking 2 hours to complete 5k records. If I restart it, it runs fine again for about 90 minutes.

The Apache log doesn’t show anything for the time it slows down, and I’m just not sure where else to look.

Running on the command line, PHP’s max execution time shouldn’t matter, and I’ve commented out the spot in CodeIngiter where it sets it to 300; I’ve even added a set_time_limit(0) to the top of the script.

My database is PostgreSQL.

Any suggestions on where to look?

Edit: Ok, Definitely Seems to be a Memory Problem. But I’m using several arrays to cache results to do batch inserts and updates, but I am clearing them out after and they are used a ton of times before I hit the 90 minute mark.

Is there a way to see what’s currently in memory?

Edit: I won’t know if this is my solution for another 90 minutes, but If you’re a CI user with Memory issues, check out this http://codeigniter.com/forums/viewthread/140012/#689396

Answer 1

You can view your memory usage with this functionmemory_get_usage().

The only things that you can see in memory are the arrays or variables you have delcared. Anything that has been unset(), reused, or nulled out is added for deletion by the garbage collector.

---

I did PHP import scripts back in the day and what you are probably running into is memory issues. Be sure to empty out arrays once you are done with them.

Depending on how much fun you want to have with your task, this might peak your interest:
http://gearman.org/

Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work.