Question

This is more like a strategy question:

I need to check regularly a long list of websites if they have a code installed. I can’t check all of them once. How to do it? I have PHP on my server.

Should I use cURL and regex? or file_get_contents()?

Answer 1

You could use cron jobs ( http://www.webmasters-central.com/article-blog/tutorials/cron-tutorial-managing-cron-tab-or-cron-job-is-easy/ )

1. First create a table in a database that will keep track when the latest “check” has occurred for each website ( use unix time ).
2. Figure out how many website “checks” you want each time you run the cron job ( let’s say 10 ).
3. Create a php script that gathers the 10 oldest “checked” websites and use curl to check those websites.
4. update the table with the current “checked” time for each website.
5. figure out how often this php script has to run to update the whole list.
   Let’s say you have 100 websites and you want ALL these sites to be updated every hour.Each time the cron job runs it will check 10 websites. Then this cron-job has to run at least 10 x an hour to accomplish the job….

This will create a loop on all the websites but will limit each request.
You have to do some trial test to see how many websites you can check every time and then implement this with a cron-job…

Good luck!