Question

I’m getting an weird error using queues/multiprocessing. I basically have a function that takes a list then parses it.

I have to do this many times so I thought it would be a good idea to spread it over the cores I have and I tried to setup a queue but its not working. Here’s my code:

# Establish communication queues
tasks = multiprocessing.Queue()

# Start consumers
num_consumers = multiprocessing.cpu_count() * 2
print 'Creating %d consumers' % num_consumers
worker = [ rules(tasks)
          for i in xrange(num_consumers) ]
for w in consumers:
    w.start()


def loadGraph(dayCurrent, day2Previous):



for dayCurrentCount  in graph[dayCurrent]:
    dayCurrentValue = graph[dayCurrent][dayCurrentCount]
    for day1Count  in graph[day2Previous]:
        day1Value = graph[day2Previous][day1Count]
        rulesDataToPass = [day1Count, day1Value, dayCurrentCount, dayCurrentValue, dayCurrent, day2Previous]

        tasks.put(rulesDataToPass)



tasks.close()
tasks.join_thread()

I’m not sure what I’m doing wrong. Without using a queue it works fine. The problem is my function requires multiple variables to run(before I was sending it individually but decided to bunch it all together as a list so I can queue it up).

please help!

Thanks in advance!

UPDATE: as requested, here is the traceback:

Traceback (most recent call last):
  File "/Users/lostsoul/Dropbox/code/learning/python/game.py", line 213, in <module>
    for i in xrange(num_consumers) ]
  File "Users/lostsoul/Dropbox/code/learning/python/game.py", line 191, in rules
    day1Count = dict[0]
TypeError: 'Queue' object does not support indexing

basically I have sending the data to a function I have called ‘rules’, the dict is then parsed like day1Count = dict[0],..day1value dict[1]…etc.. I am basically trying to compare two dictionaries against each other. so the queue is from a nested for loop that pairs every entry together(adding it to the queue which I’m hoping will get processed on all my cpu’s..it works right now individually but takes 15 min using one cpu).

UPDATE2: Here is the rules function. Its simple. basically takes the values, and then passes them to other functions(its my version of a rules engine)

def rules(dict):
    day1Count = dict[0]
    day1Value = dict[1]
    dayCurrentCount = dict[2] 
    dayCurrentValue = dict[3] 
    dayCurrent = dict[4] 
    day2Previous = dict[5]
    exactSame(day1Count, day1Value, dayCurrentCount, dayCurrentValue, dayCurrent, day2Previous)
    withinFivePercentChange(day1Count, day1Value, dayCurrentCount, dayCurrentValue, dayCurrent, day2Previous)
    deleteNonEdgeNodes(dayCurrentCount, dayCurrentValue, dayCurrent)

Without queue this works fine.

Answer 1

This line TypeError: 'Queue' object does not support indexing is indicating that your dict variable is of type Queue (which you probably dont want it to be). Double check lines 191 and up to make sure you are properly assigning the dict variable properly.

---

Also, NEVER use a type name as a variable name. dict is a type, you shouldn’t use it as the name for a dictionary variable. Use my_dict or something similar. Its possible that by using dict as a variable name, your setting some other variable to a dict object, instead of assigning it the value of the dict variable.

See stdtypes for more info on the dict class.

---

Heres the problem, your sending tasks which is a Queue to rules(), which is expecting an iterable type (list):

tasks = multiprocessing.Queue()
worker = [rules(tasks) for i in xrange(num_consumers)]

def rules(dict):
    day1Count = dict[0]
    day1Value = dict[1]
    dayCurrentCount = dict[2] 
    dayCurrentValue = dict[3]