Basic setup:
I am using a python script for automatic testing of a programming project that I am working on. In the test, I run my executable with lots of different options and compare the result with previous runs. The testing takes quite a lot of time since I have roughly 600k different tests to run.
At the moment, I have split my script into two parts, a test-module that grabs tests from a job-queue and places results in a result-queue, and a main-module that creates the job-queue and then checks the results. This allows me to play around with using several test-processes/threads which so far has not given any improvement in testing speed (I am running this on a dual-core computer, I would expect more test-processes to work better on a quad-core).
In the test module, I create a command string that I then execute using
subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
I then read the results from the pipe and place it in the result-queue.
Question:
Is this the most efficient way of running lots and lots of command strings on a multi-core system? Every Popen I do creates a new process, which seems like it might create quite a bit of overhead, but I can’t really think of a better way to do it.
(I am currently using python 2.7 in case this matters.)
EDIT:
OS is Linux
The subprocesses that I spawn are commandline C-executables with arguments.
In the end I created python C-bindings (with SWIG) directly to the code that I wanted to test. It turned out to be several hundred times faster than starting subprocesses.