Question

I’d like to write a script to pre-fetch a list of domain names for my caching dns server. I’m using the top 1,000,000 accessed websites from Alexa, available here:

http://s3.amazonaws.com/alexa-static/top-1m.csv.zip

How can I write a Python script to read this CSV file and perform an “nslookup” (or more efficient way) on each domain name listed, perhaps with a slight delay between each query? Or is there a better way to do this?

I’m guessing it would be most efficient to process the CSV line by line rather than read it all at once to minimize memory usage.

Specifically, I’m looking for a strategy for approaching this problem (libraries, tools, etc …). Sample code is appreciated, but not necessary.

Answer 1

You can totally stick with the python standard modules, as they offer everything you need.

Since open returns a iterable file-object (without loading the whole file into memory), you could use code like this:

from socket import gethostbyaddr 

with open('top-1m.csv') as input:
    for line in input:
        index, host = line.strip().split(',')
        try: 
            output = gethostbyaddr(host)
            print "%s %s is %s/%s" % (index, host, output[0], output[2])
        except: 
            print host, "not found"

Result:

1 facebook.com is www-slb-10-08-prn1.facebook.com/[‘69.171.234.21’]
2 google.com is fra07s07-in-f100.1e100.net/[‘209.85.148.100’]
3 youtube.com is fra07s07-in-f93.1e100.net/[‘209.85.148.93’]
4 yahoo.com is ir2.fp.vip.bf1.yahoo.com/[‘98.139.183.24’]
baidu.com not found

I would not recommend the csv here, since there are always just two values in each line. Use it if you need to handle things like quotecharacters or if you need to write a csv file and their like.

While twisted is also a great module for networking, it would be a little overkill for such a simple task. Just use the socket module.