Question

So I’m trying to execute a shell script which produces a lot of output(in 100s of MBs) from a Java file.
This hangs the process and never completes.

However, within the shell script, if I redirect the output of the script to some log file or /dev/null Java file executes and completes in a jiffy.

Is it because of amount of data that the Java program never completes?
If so, is there any documentation as such? or is there any limit on the amount of data(documented)?

Here’s how you can simulate this scenario.

Java file will look like:

import java.io.InputStream;

public class LotOfOutput {

    public static void main(String[] args) {
        String cmd = "sh a-script-which-outputs-huuggee-data.sh"; 
        try {
            ProcessBuilder pb = new ProcessBuilder("bash", "-c", cmd);
            pb.redirectErrorStream(true);
            Process shell = pb.start();
            InputStream shellIn = shell.getInputStream();
            int shellExitStatus = shell.waitFor();
            System.out.println(shellExitStatus);
            shellIn.close();
        } catch (Exception ignoreMe) {
        }
    }
}

The script ‘a-script-which-outputs-huuggee-data.sh’ may look like:

#!/bin/sh
# Toggle the line below
exec 3>&1 > /dev/null 2>&1

count=1
while [ $count -le 1000 ]
do
        cat some-big-file
        ((count++))
done

echo
echo Yes I m done

Free beer for the right answer. 🙂

Answer 1

It’s because you’re not reading from the Process‘ output.

As per the class’ Javadocs, if you don’t do this then you may end up with a deadlock; the process fills its IO buffer and waits for the “shell” (or listening process) to read from it and empty it. Meanwhile your process, which should be doing this, is blocking waiting for the process to exit.

You’ll want to call getInputStream() and read from that reliably (perhaps from another thread) to stop the process blocking.

Also take a look at Five Java Process Pitfalls and When Runtime.exec() Won’t – both informative articles about common problems with Process.