Question

I’m downloading relatively large files (10mb each) with urllib2, and then loading it as a json file and inserting data into a mysql database, and then repeating the process in an infinite loop. The downloading takes a minute or so, and then it loads everything into mysql. Is there a way to create a thread that does the downloading while the main thread inserts into mysql using python?

my pseudocode:

while 1:
 download file with urllib2
 decode as json file
 extract data I want
 do some computations on data
 insert data into mysql

Thank you so much!

Answer 1

What you could do is use threads and queues. A file IO thread would read and process the file and then insert the result into a queue, where a database IO thread would then discover the result and do the work. Rather than code up an example, I’ll direct you here: http://www.ibm.com/developerworks/aix/library/au-threadingpython/

Alternately you could use the python select module to manage multiple file read operations and handle them one by one as they complete: http://docs.python.org/library/select.html