Question

I would like to execute arbitrary command-line application and read its standard output as it gets produced. I use CreateNamedPipe to create a pipe and then supply other end (open used CreateFile) to CreateProcess. Provided the target process doesn’t explicitly manipulates with standard output buffering is there a way to make sure that pipe in question is unbufferred or at least that the system minimum is used as buffer size?

Answer 1

You can’t really control the buffer sizes. You can pass in read and write buffer sizes of 1 to CreateNamedPipe, but the kernel will automatically increase those buffer sizes. Basically, the buffer will always be at least as large as the largest amount of data that has been ready to read at any given time. Put another way, the faster you respond to data being available, and the smaller the blocks of data written to the pipe, the smaller the buffer will remain.

The input and output buffer sizes are advisory. The actual buffer size reserved for each end of the named pipe is either the system default, the system minimum or maximum, or the specified size rounded up to the next allocation boundary. … Whenever a pipe write operation occurs, the system first tries to charge the memory against the pipe write quota. … If the remaining pipe write quota is too small to fulfill the request, the system will try to expand the buffers to accommodate the data using nonpaged pool reserved for the process.

However, I don’t think the buffer sizes are really important. Pipes do not delay the sending of data until the buffer is “full”, and there’s nothing equivalent to the “nagle” option for TCP, so maintaining a small buffer size will not improve your latency.

Keep in mind that when you connect a pipe to a console application’s stdout, the output is typically buffered by that application before it’s written to the pipe. If you want unbuffered output, you will need to use stderr.

Also, something to watch out for when using inherited pipe handles is that the spawned application will inherit all your handles, so if you have a file or a socket open, you spawn an application, then close that handle, the file/socket/etc. will remain open until the spawned child process stops, which can lead to unexpected sharing violations and other strange problems.