Question

I’m new to c. I have some code I’m trying to test out in Cuda but am having bit of trouble pulling the data in. My data resides in a file(19GB) and I basically plan to read certain number of lines save them to a list, send them to process and do this again for the whole file.

I’m at the very beginning of learning how to do this and having issue with C that I’m not sure about, when I run the program my memory keeps increasing(when I go to activity monitor on my mac), but it is doesn’t seem to be tied to the c program, it just shows inactive memory growing and growing. Even after the program stops the memory is still inactive(only way to get it back is to reboot). As far as I can tell it doesn’t impact the program but seems strange and I’m curious to know why and if/what I can do about it?

I have a little understanding of malloc and free(sorry I know Java/Python better and never had to do this) but I am not sure if I am suppose to do it in this code because I thought the line variable keeps getting overwritten.

Here’s the code:

int main() {
    printf("Starting..");
    char line[1024];
    FILE *fp = fopen("output.txt","r");

    if( fp == NULL ) {
        return 1;
    }
    int count = 0;
    while( fgets(line,1024,fp) ) {
        //printf("%s\n",line);
        count++;
    }
    printf(" num of lines is %i \n", count);

    return 0;
}

I appreciate any tips/suggestions on what’s going on here and if there’s a better way to do this?

Update: I’m sorry, I didn’t mention, the behavior I notice is while the program is running. As it runs the inactive memory just keeps growing and growing. I have about 4 gigs free and after 30 seconds its all full and rebooting is the only way to free it(even if the c program is killed memory is not freed).

Answer 1

This is perfectly normal. Making memory free takes effort. This effort is totally wasted because as soon as the memory is needed, the system just has to remove it from the free pool. It’s much more efficient to directly transition the memory from one use to another. So the system is being smart rather than stupid.

It’s not like if you use half as much memory today you can use twice as much tomorrow. So there is no benefit to making memory free.

When you go to read lines from the file, the operating system reads in entire blocks from the file. It tries to keep those blocks in memory, if possible, because that allows it to avoid having to read from the disk in the future. If it has no better use to put the memory to, it keeps those file blocks in memory. That both saves it the effort of having to make the memory free just to make the memory used again and it speeds up any future accesses to that same block of the file.

There is no benefit to making this memory free. It just takes effort on the part of the OS to make it free, effort on the part of the OS to make it used again in the future, and the OS loses the opportunity to avoid disk reads. So making the memory free would be complete stupidity.