Question

I’ve written a little Python (2.7.2+) module (called TWProcessing) that can be described as an improvised batch manager. The way it works is that I pass it a long list of commands that it will then run in parallel, but limiting the total number of simultaneous processes. That way, if I have 500 commands I would like to run, it will loop through all of them, but only running X of them at a time so as to not overwhelm the machine. The value of X can be easily set when declaring an instance of this batch manager (the class is called TWBatchManager) :

batch = TWProcessing.TWBatchManager(MaxJobs=X)

I then add a list of jobs to this object in a very straightforward manner :

batch.Queue.append(/CMD goes here/)

Where Queue is a list of commands that the batch manager will run. When the queue has been filled, I then call Run() which loops through all the commands, only running X at a time :

batch.Run()

So far, everything works fine. Now what I’d like to do is be able to change the value of X (i.e. the maximum number of processes running at once) dynamically i.e. while the processes are still running. My old way of doing this was rather straightforward. I had a file called MAXJOBS that the class would know to look at, and, if it existed, it would check it regularly to see if the desired value has changed. Now I’d like to try something a bit more elegant. I would like to be able to write something along the lines of export MAXJOBS=newX in the bash shell that launched the script containing the batch manager, and have the batch manager realize that this is now the value of X it should be using. Obviously os.environ['MAXJOBS'] is not what I’m looking for, because this is a dictionary that is loaded on startup. os.getenv('MAXJOBS') doesn’t cut it either, because the export will only affect child processes that the shell will spawn from then on. So what I need is a way to get back to the environment of the parent process that launched my python script. I know os.ppid will give me the parent pid, but I have no idea how to get from there to the parent environment. I’ve poked around the interwebz to see if there was a way in which the parent shell could modify the child process environment, and I’ve found that people tend to insist I not try anything like that, lest I be prepared to do some of the ugliest things one can possibly do with a computer.

Any ideas on how to pull this off? Granted my “read from a standard text file” idea is not so ugly, but I’m new to Python and am therefore trying to challenge myself to do things in an elegant and clean manner to learn as much as I can. Thanks in advance for your help.

Answer 1

For me it looks that you are asking for inter-process communication between a bash script and a python program.

I’m not completely sure about all your requirements, but it might be a candidate for a FIFO (named pipe):

1) make the fifo:

mkfifo batch_control

2) Start the python – server, which reads from the fifo. (Note: the following is only a minimalistic example; you must adapt things:

while True:
    fd = file("batch_control", "r")
    for cmd in fd:
        print("New command [%s]" % cmd[:-1])
    fd.close()

3) From the bash script you can than ‘send’ things to the python server by echo-ing strings into the fifo:

$ echo "newsize 800" >batch_control
$ echo "newjob /bin/ps" >batch_control

The output of the python server is:

New command [newsize 800]
New command [newjob /bin/ps]

Hope this helps.