Question

I have an action-queue in which I submit different jobs to different servers remotely using ssh. e.g.

ssh s1 job1.py
ssh s2 job2.py

The problem is job1.py and job2.py can take a long time to finish and I do not want my action-queue to block. I wonder how I can somehow reparent my jobs.

My current solution is: job1.py uses subprocess.Popen(['my_actual_job.py']). Using this, the ssh would not block, yet my_actual_job.py never completes. It somehow terminates long before it finishes its tasks. If I do ssh s1 "job1.py 2>1", my_actual_job.py finishes, but it blocks my action-queue.

Anyone knows how can I somehow reparent my child processes (my_actual_job.py), so that the ssh can terminate but my jobs can finish their tasks in background?

I saw PEP 3143: Standard daemon process library, but is there any better and cleaner way of doing it?

I cannot change my ssh command… I need to somehow do it in my job1.py. I did Double Forking, but still doesn’t work…

Answer 1

I ended up doing: subprocess.Popen([‘nohup’,’my_actual_job.py’]) with sending both stderr and stdout to /dev/null.