Question

I need to read the total result from a C PIPE. I could just create a massive buffer and slowly fill it, but that seems inefficient and very memory hungry, so I’m wondering if anyone has any ideas how to do it.

Here is my code:

FILE* pipe = popen(command, "r");
if (!pipe)
{
    strcpy("FAIL", output);
    return -1;
}

char buffer[128];

while (!feof(pipe))
{
    if (fgets(buffer, 128, pipe) != NULL)
    {
        // should add buffer to main one here
    }
}

Any help would be greatly appreciated.

In C++ I could just add it to a String, but I’m not sure how I’d do that in C.

Edit : I don’t know the size of the final buffer.

Answer 1

I could just create a massive buffer and slowly fill it, but that
seems inefficient and very memory
hungry

If you need everything to be in the memory – that’s the only way to go. You can cache interim results in a temporary file, if it takes long time to gather all the data together, but eventually you’ll have to have that huge memory buffer filled one way or another. The interim caching can be either at the sender side or the receiver, up to you.

I’m not sure I can follow your dilemma…

edit

After some clarification it appears that the problem is that the amount of the needed memory is not known ahead of time.

For that realloc is indeed one of the ways to handle, using a file as the interim caching method will take care of that as well without the additional memory performance hit that realloc causes (but instead you get a time performance hit because disk I/O is much slower).