Question

I can’t figure out what I’m doing wrong(or how to correct it). It might be easier to show some code(its a bit simplified from what I’m doing but it proves my point):

from multiprocessing import Pool
import MySQLdb
import sys

#sql connection
try:
    conn = MySQLdb.connect (host = "127.0.0.1",user = "user",passwd = "password", db = "mydb")
except MySQLdb.Error, e:
     print "Error %d: %s" % (e.args[0], e.args[1])
     sys.exit (1)

#with database
cursor = conn.cursor ()
cursor.execute ("DROP TABLE IF EXISTS data_table")
cursor.execute ("""
    CREATE TABLE data_table(
        value     CHAR(80)
    ) ENGINE=MyISAM 
""")

cursor.execute (""" INSERT INTO data_table (value) VALUES ('%s')""" % [0, 0]) #need to insert basecase
conn.commit()


def build_table(i,x):    # i is index, x is data[i]
    conn = MySQLdb.connect (host = "127.0.0.1",user = "user",passwd = "password", db = "mydb")
    cursor = conn.cursor ()
    #print i,x
    target_sum = 100
    for s in range(target_sum + 1): 
        for c in range(target_sum + 1): 
            #print c, i
            cursor.execute ("""
INSERT INTO data_table (value) 
   SELECT  '%s'
   FROM dual
   WHERE ( SELECT COUNT(*) FROM data_table WHERE value='%s' )
         = 1
     AND NOT EXISTS
         ( SELECT * FROM data_table WHERE value='%s' )
           """ % ([s, i+1], [s - c * x, i], [s, i+1]))
            conn.commit()

    conn.close()

data = [2,5,8]
pool = Pool(processes=4)
for i, x in enumerate(data): 
    build_table(i,x) #creates 250 records
    #pool.apply_async(build_table, (i, x))
pool.close()
pool.join()

print 'completed'

It basically creates a table in mysql. The code above creates 250 entries(which is correct), but if you comment out build_table(i,x) in the for loop and uncomment pool.apply_async(build_table, (i, x)) it creates only 52 records. Why is there a difference when multiprocessing the same function and is there anything I can do to fix it so the results are the same(I thought quickly committing updates would fix it but no luck)?

If I play around pool = Pool(processes=4) and change it to 1, it works but I guess thats expected because its not multiprocessing really at that point. Also, if it helps I’m using InnoDB.

UPDATE: when I change to MyISAM I get 240 results being updated(not quite the 250 I need but much better than 52).

UPDATE2: mysql command was combined into a single command, and results seem to vary. Sometimes I get 248 results in the database, sometimes 240 or less. Maybe multiprocessing is causing this diverge between expected and actual results ?

Answer 1

I would try to combine the 2 Selects and the Insert in one Insert statement:

#print c, i
cursor.execute(""" SELECT value FROM data_table WHERE value='%s' """ % ([s - c * x, i]))
if cursor.rowcount == 1:
    cursor.execute(""" SELECT value FROM data_table WHERE value='%s' """ % [s, i+1])
    if cursor.rowcount == 0:
        cursor.execute (""" INSERT INTO data_table (value) VALUES ('%s')""" % [s, i+1])

Into something like:

#print c, i
cursor.execute ("""
    INSERT INTO data_table (value) 
       SELECT  '%s'
       FROM dual
       WHERE ( SELECT COUNT(*) FROM data_table WHERE value='%s' )
             = 1
         AND NOT EXISTS
             ( SELECT * FROM data_table WHERE value='%s' )
               """ % ([s, i+1], [s - c * x, i], [s, i+1]))

Not sure about the syntax in the last line. You’ll need to pass 3 parameters.