Question

I have a super simple python script as defined here

import multiprocessing
from multiprocessing import Pool

print "CPUs: " + str(multiprocessing.cpu_count())
convertPool = Pool(multiprocessing.cpu_count())

On Linux this appears to behave as I would of expected, with the program just starting and then printing out the number of cores, then exiting. However, on Windows this program will continue to invoke new python commands and print out “CPUs: 4” (I have 4 cores) and never stop running and will eventually kill the box. Can someone please explain what is going on here?

Thanks

EDIT:
I now have the following program which still doesn’t operate as I would expect

import sys
import os
import subprocess
from subprocess import Popen, PIPE
from threading import Thread
import multiprocessing
from multiprocessing import Pool, freeze_support

try:
    from Queue import Queue, Empty
except ImportError:
    from queue import Queue, Empty  # python 3.x

ON_POSIX = "posix" in sys.builtin_module_names

def myPrint(line, logWriter):
    if logWriter is None:
        # This will print without a newline, cause the process
        # has its own newlines
        sys.stdout.write(line)
    else:
        logWriter.write(line.strip())
        logWriter.write("\n")
        logWriter.flush()

# This is gotten from http://stackoverflow.com/questions/375427/non-blocking-read-on-a-subprocess-pipe-in-python
def executeCommand(cmd, logWriter):
   myPrint(cmd + "\n", logWriter)
    p = Popen(cmd, stdout=PIPE, bufsize=4096, close_fds=ON_POSIX, shell=True)
    q = Queue()
    t = Thread(target=enqueue_output, args=(p.stdout, q))
    t.daemon = True # thread dies with the program
    t.start()

    # read line without blocking
    while t.isAlive() or not q.empty():
        try:
            line = q.get_nowait() # or q.get(timeout=.1)
        except Empty:
            pass # Do nothing
        else: # If there is a line, then print it 
            if logWriter is not None:
                myPrint(line, logWriter)
        # Sleep to prevent python from using too much cpu
        time.sleep(0.05)

if __name__ == "__main__":
    freeze_support()
    convertPool = Pool(multiprocessing.cpu_count())
    # Now let's smooth and threshold all the masks
    for foo in bar:
        threshSmoothCmd = "ConvertImage.exe -i " + foo + " -o " + foo
        myPrint(threshSmoothCmd + "\n", None)
        convertPool.apply_async(executeCommand,(threshSmoothCmd, None,))

    convertPool.close()
    convertPool.join()
print "Finished processing"

ConvertImage.exe is an executable I wrote myself foo and bar are just placeholders. On Linux this will fire up multiprocessing.cpu_count() number of ConvertImage.exe processes and then print Finished processing once all of the ConvertImage.exe processes have finished. On Windows this immediately fires up len(bar) ConvertImage.exe processes and then immediately prints Finished processing and exits. How do I get the Windows version to behave like the Linux version?

Answer 1

I figure this out, it was working correctly, my example program was posted wrong as I forgot some imports and that caused the described behavior. After I got all the imports correct this example worked fine on Windows and Linux.