Question

When creating sub-processes in java using Runtime.exec(), I am aware that I have to fill the input/drain the output streams to prevent blocking of the subprocess.

Interestingly, the javadoc of Process states a little bit more:

...failure to promptly write the input stream or read the output stream of
the subprocess may cause the subprocess to block, and even deadlock.

I am wondering that in this situation the subprocess can also deadlock!

Questions:
1. Under which conditions does it deadlock?
2. Why does it deadlock?
3. Can you provide a short example program which shows this deadlock?
4. Is this deadlock a bug in the OS?

Answer 1

Deadlock can occur due to limited buffer sizes when the parent tries to send too much data to the input stream of its child before reading any of the output.

Consider this code:

final int LINES = 10;
// "tr" is a Unix command that translates characters;
// Here, for every line it reads, it will output a line with
// every 'a' character converted to 'A'.  But any command that outputs
// something for every line it reads (like 'cat') will work here
Process p = Runtime.getRuntime().exec("tr a A");
Writer out = new OutputStreamWriter(p.getOutputStream());
for (int i = 0; i < LINES; i++) {
    out.write("abcdefghijklmnopqrstuvwxyz\n");
}
out.close();
// Read all the output from the process and write it to stdout
BufferedReader in = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line;
while ((line = in.readLine()) != null) {
    System.out.println(line);
}

For small values for lines, it will work fine; all of tr‘s output can fit in the OS buffer before we start reading it. However, for large values (>10000 ought to be sufficient), the OS buffer will fill up; within the tr command, calls to write will block, waiting for the buffer to be drained, and in turn, the buffer that the Java code is writing to will fill up (because tr is be blocked, preventing it from reading from its input), in turn blocking our calls to out.write, leading to a deadlock where both processes are waiting to write into full buffers that aren’t being actively read from.

This deadlock is not a bug in the OS, as limited buffer sizes for inter-process communication are a deliberate design decision. The alternative (unlimited buffer sizes) has some downsides:

• Can exhaust kernel memory
• To avoid the above, if buffers automatically “spill over” to disk, that can lead to unpredictable performance and can potentially fill up the disk.

---

As an aside, deadlock can also occur due to in-process buffers. Suppose that, to attempt to address the deadlock above, we changed our Java code to write one line, then read one line, alternatingly. However, it’s common for Linux processes to not flush after every line when they’re not writing directly to a terminal. So tr might read a line, and write it to its libc output buffer, and than block waiting for the next line to be written–and our Java code will block waiting for tr to output a line.