Home/ Questions/Q 4030186
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-20T11:26:23+00:00

I have a massive amount of data that needs to be read from mysql,

  • 0

I have a massive amount of data that needs to be read from mysql, analyzed and based on the results split up and stored in new records.
five record takes about 20 seconds, but the records vary in length so I can’t really estimate how long the program will take, however have calculated that the process should not take longer much longer than 5 hours, so I’d like to run it over night and feel quite sure that when I come back to the office the next morning the program is done.

Assuming the code is fail safe (I know right 😉 how should set up Apache / PHP /Mysql settings so that when I execute the script so that I can be sure that the program will not time out and/or not run out of ram?

(it is basically running in a loop fetching sets of 100 rows until it can’t anymore a loop so, I am hoping the fact that the variables are being reset at the beginning of each iteration will keep the memory usage constant.)

The actual size of the database when dumped is 14mb, so the volume of the data is not so high

(on a side note, it might also be that I haven’t assigned the maximum resources to the server settings, so maybe that’s why it takes 20 seconds to run 5 records)

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Make sure you have removed any max_execution_time limits by setting this to 0 (unlimited) in your PHP.ini or by calling set_time_limit(0). This will ensure that PHP doesn’t stop the script mid-execution.
    If it all possible, you should run the script from the CLI so that you don’t have to worry about Apache timing your request out (it shouldn’t, but it might).
    Since you are working with only 15 MB of data I wouldn’t worry about memory usage (128 MB is the default in PHP). If you are really worried you can remove memory limits in PHP by modifying the memory_limit to be either a higher number of -1 (infinite memory).

    Keep in mind modifying the PHP.ini will affect all scripts that are interpreted by that installation. I prefer to use the appropriate ini setting functions at the top of my scripts to prevent dangerous global changes.

    On a side note: This doesn’t really sound like a job for PHP. I’m not trying to discourage your use of PHP here, but there are other languages that are better suited for command line usage.

    • 0