I’m writing a PHP script to scrape data from the web. End-result, I want to have all the scraped data tucked and formatted nicely in a mysql database.
But as there are multiple loops and arrays to iterate through within the PHP script, my gut feeling is that for speed and efficiency, it would be best NOT to keep loop-cycling access to the mysql database (inserting data on each go-around of the loops) — instead, storing the data in temporary arrays within PHP, and then only at the end of the script, dumping the arrays into mysql in one go.
What say ye?
yes, generally being minimal when it comes to the number of querys is performant. But, your script is limited by network io(download speed), not the database. Your script will be asleep most of the time while the os downloads the data. Any queries you issue would be relatively infrequent because of that.
my advice is to just get it working.