Question

I have a number of large files with many thousands of lines in python dict format. I’m converting them with json.dumps to json strings.

import json
import ast

mydict = open('input', 'r')
output = open('output.json', "a")

for line in mydict:
        line = ast.literal_eval(line)
        line = json.dumps(line)
        output.write(line)
        output.write("\n")

This works flawlessly, however, it does so in a single threaded fashion. Is there an easy way to utilize the remaining cores in my system to speed things up?

Edit:

Based on the suggestions I’ve started here with the multiprocessing library:

import os
import json
import ast
from multiprocessing import Process, Pool

mydict = open('twosec.in', 'r')

def info(title):
        print title
        print 'module name:', __name__
        print 'parent process: ', os.getppid()
        print 'process id:', os.getpid()

def converter(name):
        info('converter function')
        output = open('twosec.out', "a")
        for line in mydict:
                line = ast.literal_eval(line)
                line = json.dumps(line)
                output.write(line)
                output.write("\n")

if __name__ == '__main__':
        info('main line')
        p = Process(target=converter, args=(mydict))
        p.start()
        p.join()

I don’t quite understand where Pool comes into play, can you explain more?

Answer 1

Wrap the code above in a function that takes as its single argument a filename and that writes the json to an output file.

Then create a Pool object from the multiprocessing module, and use Pool.map() to apply your function in parallel to the list of all files. This will automagically use all cores on your CPU, and because it uses multiple processes instead of threads, you won’t run into the global interpreter lock.

Edit: Change the main portion of your program like so;

  if __name__ == '__main__':
     files = ['first.in', 'second.in', 'third.in'] # et cetera
     info('main line')
     p = Pool()
     p.map(convertor, files)
     p.close()

Of course you should also change convertor() to derive the output name from the input name!

Below is a complete example of a program to convert DICOM files into PNG format, using the ImageMagick program

"Convert DICOM files to PNG format, remove blank areas."

import os
import sys # voor argv.
import subprocess
from multiprocessing import Pool, Lock

def checkfor(args):
    try:
        subprocess.check_output(args, stderr=subprocess.STDOUT)
    except CalledProcessError:
        print "Required program '{}' not found! exiting.".format(progname)
        sys.exit(1)

def processfile(fname):
    size = '1574x2048'
    args = ['convert', fname, '-units', 'PixelsPerInch', '-density', '300', 
            '-crop', size+'+232+0', '-page', size+'+0+0', fname+'.png']
    rv = subprocess.call(args)
    globallock.acquire()
    if rv != 0:
        print "Error '{}' when processing file '{}'.".format(rv, fname)
    else:
        print "File '{}' processed.".format(fname)
    globallock.release()

## This is the main program ##
if __name__ == '__main__':
    if len(sys.argv) == 1:
        path, binary = os.path.split(sys.argv[0])
        print "Usage: {} [file ...]".format(binary)
        sys.exit(0)
    checkfor('convert')
    globallock = Lock()
    p = Pool()
    p.map(processfile, sys.argv[1:])
    p.close()