Question

I need to scan a given set of URLs and get the HTTP status codes like "200 OK" in Python. I’m currently using urllib to do this. Is there a faster way to do this?

Python code

def get_status(url):
try:
    return urllib.urlopen(url).getcode()

except StandardError :
    return None

Answer 1

Yes there is.

1. Use multiple threads all checking different URLs at the same time.
2. Use raw sockets implementing a simple HTTP request. As soon as you get the 200 response (or any other code) you close the connection, avoiding unnecessary transfer of data.