I have an array that I’m appending to in a loop:
array_push($obj->{$type.'listings'}, $listing);
The loop makes a bunch of remote calls and pulls in data that I want to store in a file and use as a sort of cache.
While running this “cache routine”, if I serialize this array it reports to be just over 5MB. To create the 5MB array, the script exhausts 100MB of memory. I’m being very careful to unset variables to free memory as I can. I’ve went as far as commenting everything out and function by function uncommented to see where the memory build up is. It certainly happens as I push to the array. If I don’t use the array_push, then using memory_get_usage() and gc_collect_cycles() I can see a rather steady amount of memory used with some spikes, but it frees itself as it moves along.
As soon as I push to the array, it gets all crazy and the memory usage just piles up.
Are there any ideas on how to handle a situation like this?
Can I not build large arrays like this with PHP?
Is there a way to flush the array to a file while I’m building it? And lets assume that I could get it built and stored in a file. When I want to use it, will i use the same amount of resources once pulled from the file….or is the memory exhaustion just happening because it’s dynamically being added to as it’s being built?
Is this something that I should just consider using SQL for? and storing each run of the loop in a separate row?
Just for kicks, I added: ini_set('memory_limit', '-1'); just to see if it would run.
It did, and memory usage peaked at just over 100 MB, with a serialized object size of just over 8MB. Now, this isn’t really helpful as I am just processing a sample of data and this could grow to be much bigger.
Anyway, any thoughts you might have on optimizing a situation like this would be great.
For 5MB of array PHP doens’t use 100MB for sure.
Use http://it.php.net/memory_get_usage in your script to see where it grows
Consider I build in PHP arrays of millions (2 millions) of entry without problems. (memory usage less than 200MB)