Question

After some lookup into google and posts of stackoverflow and other sites, i’m still confused on how i can apply a queue and threading on my code:

import psycopg2
import sys
import re
# for threading and queue
import multiprocessing
from multiprocessing import Queue
# for threading and queue
import time
from datetime import datetime

class Database_connection():
    def db_call(self,query,dbHost,dbName,dbUser,dbPass):
        try:
            con = None
            con = psycopg2.connect(host=dbHost,database=dbName,
                                   user=dbUser,password=dbPass)
            cur = con.cursor()
            cur.execute(query)
            data = cur.fetchall()
            resultList = []
            for data_out in data:
                resultList.append(data_out)
            return resultList
        except psycopg2.DatabaseError, e:
                print 'Error %s' % e
                sys.exit(1)
        finally:
            if con:
                con.close()

w = Database_connection()
sql = "select stars from galaxy"

startTime = datetime.now()
for result in  w.db_call(sql, "x", "x", "x", "x"):
    print result[0]
print "Runtime: " + str(datetime.now()-startTime)

lets supose the result will be 100+ values. How can i, put those 100+ results on queue and execute ( print, for example ) then 5 at time using queue and multiprocessing module?

Answer 1

I was able to solve the problem with the following:

def worker():
    w = Database_connection()
    sql = "select stars from galaxy"
    for result in  w.db_call(sql, "x", "x", "x", "x"):
        if result:
            jobs = []
            startTime = datetime.now()
            for i in range(1):
               p = multiprocessing.Process(target=worker)
               jobs.append(p)
               p.start()
            print "Runtime: " + str(datetime.now()-startTime)

I belive it is not the best way to do it, but for now solved my problem 🙂