Question

My task is to parse a large .txt file (circa. 15,000 lines) into a MySQL database. The problem is I’m working with a 30 second maximum execution time. I’ve tried using this:

        $handle = @fopen('http://www.someothersiteyouknow.com/bigfile.txt', "r"); 
    if ($handle) { 
       while (!feof($handle)) { 
           $lines[] = fgets($handle, 4096); 
       } 
       fclose($handle); 
    }

I can then access the $lines array and parse the data whichever way I need to but it takes too long for the script to finish running. My feeling is that I should read the file in chunks, maybe 1000 lines at a time. But I only understand how to read from the beginning of the .txt file. Please may you impart some ideas for methods of doing this correctly? Just to clarify, I don’t require specific code examples, just ideas for how to parse large .txt files using PHP.

Answer 1

This doesn’t seem like the best idea, to be honest. What if multiple users access the same page at, or around, the same time? You’ll have (number of users*large text file) being processed concurrently.

Suggest you bring the file local (save it locally if the file doesn’t already exist), and work with the local file. This should help reduce your transaction time

This should help bring you into the 30s limit … if the file doesn’t take longer than 30s to download!