Question

I’m writing a wrapper that forks an execv() process. Output from the child is captured in stderr. When waitpid() releases, I can read the contents of stderr and report it back.

In my case, I want to dynamically allocate a buffer and write stderr into this buffer.

To size the buffer I can realloc(), but this is not very efficient and I have found it tends to bloat memory pool use. Rather, I’d like to know the size without altering its pointer.

Consider the following, and note that MyStderr is just a placeholder for stderr:

int size=0;
int ch=0;
// int MyStderr is the stderr fd
FILE *nCountFD = fdopen(MyStderr, "r");
while ((ch = getc(nCountFD)!=EOF)
{
    ++size;
}
printf("Size is %ld\n", size);

Here I get the size. However, now the file pointer for MyStderr is at the end of its buffer.

I tried using lseek.

lseek() fails against stderr, so I can’t use it here. At least, this is what my testing and stackoverflow searching indicates.

So …

1. Is there a way to get the size without moving MyStderr to eof ?

or

1. Is there an lseek method that will work with MyStderr ?

Note. Here is the only solution I can think of, using realloc..

char *buf=(char*)malloc(80);
char *NewBuf=NULL;
int n=80;
while ((ch = getc(nCountFD)!=EOF)
{
    buf[i]=ch;
    ++size;
    if (size>n)
    {
       n=n+80;
       NewBuf= realloc(buf, n);
       // some code to make sure it works here //
       buf=NewBuf;
    }
}
printf("Size is %ld\n", size);

---

And now the update

Rather than build the functionality to work around the fact that stderr is unbuffered, I determined to make the initial malloc of my result buffer sufficiently large so that realloc() would be unlikely in the majority of cases. And if realloc() happens, the original allocation is doubled in size for each realloc() as suggested.

In testing (100000 iterations) this works very well with no leaking or discernible bloating.

I owe a great deal to the Stack Overflow community. Thank you all.

The code below won’t run stand-alone. It’s placed here to illustrate what I did.

..
after all the code that parses the commandline, forks, does execv, and cleans up…

while (waitpid(nPID, &status, 0) != nPID)
    ;
i = 0;
nFD = fdopen(nErrFD, "r");
if (!nFD) {
    snprintf(cErrMsg, 80, "Cannot open fd[%i].  Failed to spaw process",
            nErrFD);
    cbuf = strcpy(cbuf, cErrMsg);
    goto NECerror;
}

close(nErrFD);


cbuf = calloc(nBufSz, sizeof(char));
memset(cbuf, 0x00, nBufSz);
i = 0;
while ((ch = getc(nFD)) != EOF) {
    cbuf[i] = (char) ch;
    ++size;
    ++i;
    if (size > nBufSz) {
        nBufSz = nBufSz + nBaseBufSz;
        NewBuf = realloc(cbuf, nBufSz);
        if (NewBuf == NULL) {
            snprintf(cErrMsg, 80,
                    "Internal error:cannot allocate [%i] bytes", nBufSz);
            cbuf = strcpy(cbuf, cErrMsg);
            fclose(nFD);
            goto NECerror;
        }
        cbuf = NewBuf;
        free(NewBuf);
    }

}
fclose(nFD);

Answer 1

Pipes are not seekable — if you’re reading data from a pipe that’s being provided by the other process, then you cannot seek its data stream. You have to store the data as its read.

The reason you’re probably seeing lots of memory thrashing is due to the way your expanding your buffer — you only expand it by a constant amount every time it’s exceeded. You’ll have much better success if you double its size when it’s exceeded: this is a well-known technique for dynamic arrays.

For example:

// Error checking omitted for expository purposes
size_t size = 4096;
char *buf = malloc(size);
char *bufPtr = buf;
ssize_t n;
while((n = read(MyStderr, bufPtr, 4096) > 0)
{
    bufPtr += n;

    // Make sure we always have at least 4096 bytes of free space in the
    // buffer for the next read
    if(size - (bufPtr - buf) < 4096)
    {
        size *= 2;  // Watch out for overflow!
        buf = realloc(buf, size);
    }
}

// The (bufPtr - buf) bytes of buf now hold the entirety of the process's stderr