Question

The name kind of says it all. I’m writing this program in python 2.7, and I’m trying to take advantage of threaded queues to make a whole bunch of web requests. Here’s the problem: I would like to have two different queues, one to handle the threaded requests, and a separate one to handle the responses. If I have a queue in my program that isn’t named “queue”, for example if I want the initial queue to be named “input_q”, then the program crashes and just refuses to work. This makes absolutely no sense to me. In the code below, all of the imported custom modules work just fine (at least, they did independently, passed all unit tests, and don’t see any reason they could be the source of the problem).

Also, via diagnostic statements, I have determined that it crashes just before it spawns the thread pool.

Thanks in advance.

EDIT: Crash may be the wrong term here. It actually just stops. Even after waiting half an hour to complete, when the original program ran in under thirty seconds, the program wouldn’t run. When I told it to print out toCheck, it would only make it part way through the list, stop in the middle of an entry, and do nothing.

EDIT2: Sorry for wasting everyones time, I forgot about this post. Someone had changed one of my custom modules (threadcheck). It looks like it was initializing the module, then running along its merry way with the rest of the program. Threadcheck was crashing after initialization, when the program was in the middle of computations, and that crash was taking the whole thing down with it.

code:

from binMod import binExtract
from grabZip import grabZip
import random
import Queue
import time
import threading
import urllib2
from threadCheck import threadUrl
import datetime

queue = Queue.Queue()
#output_q = Queue.Queue()
#input_q = Queue.Queue()
#output = queue

p=90
qb = 22130167533
url = grabZip(qb)
logFile = "log.txt"
metaC = url.grabMetacell()
toCheck = []
print metaC[0]['images']
print "beginning random selection"
for i in range(4):
    if (len(metaC[i]['images'])>0):
        print metaC[i]['images'][0]
        for j in range(len(metaC[i]['images'])):
            chance = random.randint(0, 100)
            if chance <= p:
                toCheck.append(metaC[i]['images'][j]['resolution 7 url'])

print "Spawning threads..."
for i in range(20):
    t = threadUrl(queue)
    t.setDaemon(True)
    t.start()
print "initializing queue..."
for i in range(len(toCheck)):
    queue.put(toCheck[i])

queue.join()    
#input_q.join()
output = open(logFile, 'a')
done = datetime.datetime.now()
results = "\n %s \t %s \t %s \t %s"%(done, qb, good, bad)
output.write(results)   

Answer 1

What the names are is irrelevant to Python — Python doesn’t care, and the objects themselves (for the most part) don’t even know the names they have been assigned to. So the problem has to be somewhere else.

As has been suggested in the comments, carefully check your renames of queue.

Also, try it without daemon mode.