Question

I’m using resque, with a queue processor which, as part of its execution, will start a shell process. Currently, I am using PTY.spawn() to invoke the shell command and handle its output.

I’d like to augment this code so that a quantity (N) can be given (the command executed onboards VMs, I want to be able to start a variable number with one call), and have the shell process be called N times in separate processes, without the Nth call having to wait for call N-1 to finish, and so on. I also want to capture all STDOUT from each invocation, so that I can do work on the output once the call is done.

I have looked at Kernel::fork but the scope of code inside a forked block is not the same as its parent (for pretty obvious reasons).

What tool(s) can I use so that each process can be spawned independently, their output can be captured, and I can still have the parent process wait for them all to finish before moving on?

Answer 1

Here:

stdouts=[]
numberOfProcesses.times do 
  stdouts<<PTY.spawn(command_line)[0..-1]
end

That’s pretty basic if you just spawn them and get a bunch of STDOUT/STDIN pairs. If you want to be able to work on each process’s output as soon as it is done, try this:

threads=[]
numberOfProcesses.times do
  threads<<Thread.new(command_line) |cmd|
    stdout, stdin, pid = PTY.spawn(cmd)
    Process.waitpid(pid)
    process_output(stdout.read)
  end
end
threads.each {|t| t.join}

That spawns them in parallel, each thread waiting for when it’s instance is done. When it’s instance is done, it processes output and returns. The main thread sits waiting for all of the others to finish.