Question

We’re using a back-end proprietary database to feed data to over 100 dynamic pages on our website. The URL’s are in the form http://www.example.com/blah?id=1, http://www.example.com/blah?id=2, etc. These pages can take up to 5 seconds to load since the retrieval of data is so slow (because of the database we’re connecting to).

My goal is to write a cron job (in PHP), that will “loop” through and make a request to all of our dynamic pages so they are cached (we have a 24 hour cache for these pages).

Is my best approach a PHP script with embedded “wget” commands where the output is ignored? I have access to cURL if that is preferred. Is there a performance issue with writing a script that will hit 100 URL’s just to cache the pages? We’ll of course run it off peak hours at night.

Thanks for any guidance on this one!

Answer 1

I suggest using multi-cURL to get this done. That way, your requests will run in parallel to each other.

You can do this with the built-in PHP functions, but I suggest using one of the many classes that wrap them:

https://github.com/jmathai/php-multi-curl

http://curl.haxx.se/libcurl/php/examples/multi.html