Question

I have a PHP script that imports CSV files and goes through tens of thousands of iterations. As the script runs over a course of hours, the memory use goes up and up, and if the file is big enough, the script uses up so much memory that the whole machine grinds to a halt.

Right now the only technique I’m using is to unset() everything I can when I’m done with it. I’ve tried to isolate the part that’s using the most memory, but it seems like every function in my script is just one more straw on the camel’s back, and using “as little memory as possible.”

So what can I do?

I’ve tried looking into benchmarking/profiling tools but I haven’t found anything good. I’m on a Windows machine, SSHing into a Linux box.

Answer 1

Ok, since you’re looking for techniques, let me list some…

1. Don’t read files, stream them

Rather than calling $data = file_get_contents($file), open it with fopen and only read the data you need at that point in time (fgets or fgetcsv, etc). It’ll be a touch slower, but it’ll use FAR less memory.

2. Upgrade to 5.3.4

If you’re still on PHP 5.2.x, memory will be greatly conserved by upgrading to 5.3.x (latest 5.3.4). It includes a garbage collector that will clean up freed memory after a while.

3. Don’t use anything in the global scope

Don’t store any information in the global scope. it’s never cleaned until the end of execution, so it could be a memory leak in and of itself.

4. Don’t pass around references

PHP uses copy-on-right. Passing around references only increases the chances that unset won’t get all of them (because you forgot to unset one of the references). Instead, just pass around the actual variables.

5. Profile the code

Profile your code. Add debug hooks to the start and end of each function call, and then log them watching the memory usage at the entrance and exit of every function. Take the diffs of these and you’ll know how much memory is being used by each function. Take the biggest offenders (those that are called a lot, or use a lot of memory) and clean them up… (lowest hanging fruit).

6. Use a different language

While you can do this with PHP (I have and do quite often), realize it may not be the best tool for the job. Other languages were designed for this exact problem, so why not use one of them (Python or Perl for example)…

7. Use Scratch Files

If you need to keep track of a lot of data, don’t store it all in memory the entire time. Create scratch files (temporary files) to store the data when you’re not explicitly using it. Load the file only when you’re going to use that specific data, and then re-save it and get rid of the variables.

8. Extreme cases only: don’t use large arrays!

If you need to keep track of a large number of integers (or other simple data types), don’t store them in an array! The zval (internal data structure) has a fair bit of overhead. Instead, if you REALLY need to store a LARGE number of integers (hundreds of thousands or millions), use a string. For a 1 byte int, ord($numbers[$n]) will get the value of the $n index, and $numbers[$n] = chr($value); will set it. For multy-byte ints, you’d need to do $n * $b to get the start of the sequence where $b is the number of bytes. I stress that this should only be used in the extreme case where you need to store a TON of data. In reality, this would be better served by a scratch file or an actual database (Temporary Table likely), so it may not be a great idea…

Good Luck…