I have a new challenge today. I will develop a script that will take some large data_log (fopen) and insert it into a database. Yesterday I did my first tests, and I estimate that this script will need a week running to process the log.
I made some env configs, like set_time_limit and set_memory_limit.
But, in this first test after an while (15min), the app crashes. The machine was running in 2gb memory usage when it crashes.
The msg error:
PHP Fatal error: Allowed memory size of X bytes exhausted (tried to allocate Y) in whatever.php
Please keep in mind that the script/code is ok.
The script code isn’t ok. You aren’t timing out, you’re simply running out of memory to store whatever data you are loading from the log file.
Without seeing the code or knowing the context, you should look at the following options.
Clean up after yourself so you don’t use more and more memory. If you’re reading/writing data from or to a file, make sure you clear what data is in the memory every once in a while (depends on the loop)
Modify the script to work with chunks of data. Make the script run every minute or so and make it shut down after you’ve worked with a portion of the data. This will keep the memory requirement low and you also don’t have a continuously running script (easy to continue from where you stopped)