Question

I have a working Python script that checks the 6,300 or so sites we have to ensure they are up by sending an HTTP request to each and measuring the response. Currently the script takes about 40 min to run completely, I was interested in possibly some other ways to speed up the script, two thoughts were either threading or multiple running instances.

This is the order of execution now:

1. MySQL query to get all of the active domains to scan (6,300 give or take)
2. Iterate through each domain and using urllib send an HTTP request to each
3. If the site doesn’t return ‘200’ then log the results
4. repeat until complete

This seems like it could possibly be sped up significantly with threading but I am not quite sure how that process flow would look since I am not familiar with threading.

If someone could offer a sample high-level process flow and any other pointers for working with threading or offer any other insights on how to improve the script in general it would be appreciated.

Answer 1

The flow would look something like this:

1. Create a domain Queue
2. Create a result Queue
3. MySQL query to get all of the active domains to scan
4. Put the domains in the domain Queue
5. Spawn a pool of worker threads
6. Run the threads
7. Each worker will get a domain from the domain Queue, send a request and put the result in the result Queue
8. Wait for the threads to finish
9. Get everything from the result Queue and log it

You’ll probably want to tune the number of threads, thus the pool, and not just 6300 threads for every domain.