Question

I have a subroutine of a Python application that takes a list of eight-character serial numbers and searches a set of directories for filenames containing those SNs with a range of possible extensions. The function that does it is as follows.

def CompleteSN(SNorig, directories, exts, directoryContents=None):

    # Split SNorig into its directory path and basename
    directory, basename = os.path.split(SNorig)

    #Don't try to complete if it has a directory already
    if len(directory) > 0:
        return SNorig

    # Split the basename into extension (ext) and serial number (SN)
    SN = basename[:8].upper()

    #If we have a cache, check it
    if directoryContents is not None:
        if SN in directoryContents:
            return directoryContents[SN]
        else:
            return None

    possSNs = ['.'.join((SN, ext)) for ext in exts]

    for pSN in possSNs:
        for d in directories:
            dpath = os.path.join(d, pSN)
            if os.path.exists(dpath):
                return dpath

    return None

SN is the serial number to turn into a complete path, directories is a list of directories to look for it in, exts is the list of possible extensions to try (with no leading dots), and directoryContents is either None or a large (hundreds of thousands of entries) dict mapping the serial numbers of files in the directories to search to their full names. My thought was that for large sets of serial numbers to complete, it would be faster to put all the serial numbers that could be found into a dictionary so they could be quickly looked up rather than making a system call for each one. For large numbers of SNs to complete, this would be worth the cost of making the dict.

And it is much faster once directoryContents is populated, as I predicted. However, this is only the case when not using multiprocessing. I allow the user to toggle multiprocessing mode, which is faster as expected when directoryContents is None. (i.e. when we’re finding the files by checking their existence with os.path.exists) However, when checking the dict for them, calling CompleteSN in a multiprocessing.Pool is bizarrely slow, on the order of fewer than 10 SNs/second. (Compared with thousands when not using multiprocessing) My best guess as to what is going on is that whatever mechanism Python uses to share data between pool workers (of which I think there are 8) is getting bogged down with such a large dict structure. Can anyone help me learn what is going on here?

Answer 1

In general, you shouldn’t share data structures between multiple processes. The overhead in synchronization often kills performance. Perhaps you can create individual copies of your look-up dict, one for each process?