Question

I am writing a script that will probably need half a day because it gets data from about 14000 webpages from a website.

To find out whether it makes progress or not, is there any way to observe its execution, i.e. the outgoing connections to each of the scraped pages, with the mac os shell?

I am using curl to get the page contents, if that is of any help.

Thanks a lot!
Charles

EDIT
The script is written in php and executed from localhost.

Answer 1

When writing custom scripts it is very helpful to output some sort of status to stdout.

this can be done in a uniform way using printf http://www.php.net/manual/en/function.sprintf.php

What you log to stdout depends on what information you need to see. Perhaps for a curl request I would log Url, Response code, maybe start time and end time. Its really up to you, just make sure you can verfiy it’s status/progress.

printf('%40s | %5s', 'URL', 'Status Code');
printf('%40s | %5s', $the_url, $status_code);