Question

I’m currently running a Linux based VPS, with 768MB of Ram.

I have an application which collects details of domains and then connect to a service via cURL to retrieve details of the pagerank of these domains.

When I run a check on about 50 domains, it takes the remote page about 3 mins to load with all the results, before the script can parse the details and return it to my script. This causes a problem as nothing else seems to function until the script has finished executing, so users on the site will just get a timer / ‘ball of death’ while waiting for pages to load.

**(The remote page retrieves the domain details and updates the page by AJAX, but the curl request doesnt (rightfully) return the page until loading is complete.

Can anyone tell me if I’m doing anything obviously wrong, or if there is a better way of doing it. (There can be anything between 10 and 10,000 domains queued, so I need a process that can run in the background without affecting the rest of the site)

Thanks

Answer 1

A more sensible approach would be to "batch process" the domain data via the use of a cron triggered PHP cli script.

As such, once you’d inserted the relevant domains into a database table with a "processed" flag set as false, the background script would then:

1. Scan the database for domains that aren’t marked as processed.

2. Carry out the CURL lookup, etc.

3. Update the database record accordingly and mark it as processed.

   …

To ensure no overlap with an existing executing batch processing script, you should only invoke the php script every five minutes from cron and (within the PHP script itself) check how long the script has been running at the start of the "scan" stage and exit if its been running for four minutes or longer. (You might want to adjust these figures, but hopefully you can see where I’m going with this.)

By using this approach, you’ll be able to leave the background script running indefinitely (as it’s invoked via cron, it’ll automatically start after reboots, etc.) and simply add domains to the database/review the results of processing, etc. via a separate web front end.