Question

I am a new to python and having some issues with threading and queues.

Basically i have a file of URL’s, and would like to create a queue of 10 workers which sequentially work down through the file and perform a urllib2 request.

I just cant get my head around it! Could anyone help please with some simple links or code?

Thanks, Matt

Answer 1

The built in “Queue” module would make a good underlying queue for your app. It handles all multi-threaded locking semantics for you.

http://docs.python.org/library/queue.html

You can load your file, populate your Queue, spin up your worker threads, and have them pull from the Queue and do their work.