Question

I have a multiprocessing code that search database records for a certain condition, when it reaches the end of the database the loop must stop, how can I do that? Here is the code:

import psycopg2
from multiprocessing import Pool

conn = psycopg2.connect(a database)
query=conn.cursor()
query.execute(some query)

def scanfile():
    try: row=query.fetchone()
    except: return False #here the loop must stop 
    #do something with row...

if __name__=='__main__':
    pool=Pool()
    while True:
        pool.apply_async(scanfile)
    pool.close()
    pool.join()

Answer 1

Move your query / cursor over as the iterable and let multiprocessing distribute the work for you.

import psycopg2
from multiprocessing import Pool


def scanfile(row):
    #do something with row...


if __name__ == '__main__':
    conn = psycopg2.connect(a database)
    query = conn.cursor()
    query.execute(some query)
    pool = Pool()
    pool.map_async(scanfile, query).wait()
    pool.close()
    pool.join()