Question

I have a problem with an application I’m currently developing. In this program I have to read huge amounts (billions) of data from text files and manage them consequently, but since it’s a two students project, the reading part will be developed by my mate. For testing reason I wrote a small procedures that generates pseudo-random structures to replace what my mate will do.

The problem is the following: a big amount of the generated data (due to redundancy) can be discarded in order to free its memory. But even invoking the free() function the memory usage keeps growing. So I tried to develop a debug application that simply generates a chunk of data and immediately frees it. And repeats that for thousands of times. Well, I can’t grasp the reason, but the memory allocated to the process grows to ~1.8 GB ram and then crashes. Why? The strangest thing, that makes me thing there’s a lot I’m not understanding well, is that when the process crashes the malloc does NOT return a NULL pointer, because the process always crashes when readCycles == 6008 and bypasses the NULL check.

I already read other related topics here on StackOverflow and I understood why free() doesn’t reduce the memory allocated to my process. That’s fine. But why the memory usage keeps growing? Shouldn’t the malloc allocate previously freed memory instead of constantly requesting new one?

This is the most relevant part of my code:

#define NREAD 1000
#define READCYCLES 10000
#define N_ALPHA_ILLUMINA 7
#define N_ALPHA_SOLID 5
#define SEQLEN 76

typedef struct{
    char* leftDNA;
    char* leftQuality;
    unsigned long int leftRow;
    char* rightDNA;
    char* rightQuality;
    unsigned long int rightRow;
} MatePair;


unsigned long int readCycles = 0;


MatePair* readStream(MatePair* inputStream, short* eof, unsigned long int* inputSize){

    double r;
    unsigned long int i, j;
    unsigned long int leftRow;
    int alphabet[] = {'A', 'C', 'G', 'T', 'N'};
    inputStream = (MatePair*) malloc (sizeof(MatePair) * (NREAD + 1));
    printf("%d\n", readCycles);
    if (inputStream == NULL){
        (*eof) = 1;
        return;
    }

    for (i = 0; i < NREAD; i++){
        leftRow = readCycles * NREAD + i;
        inputStream[i].leftDNA = (char*) malloc (SEQLEN);
        inputStream[i].rightDNA = (char*) malloc (SEQLEN);
        inputStream[i].leftQuality = (char*) malloc (SEQLEN);
        inputStream[i].rightQuality = (char*) malloc (SEQLEN);
        for (j = 0; j < SEQLEN; j++){
            r = rand() / (RAND_MAX + 1);
            inputStream[i].leftDNA[j] = alphabet[(int)(r * 5)];
            inputStream[i].rightDNA[j] = alphabet[(int)(r * 5)];
            inputStream[i].leftQuality[j] = (char) 64 + (int)(r * 60);
            inputStream[i].rightQuality[j] = (char) 64 + (int)(r * 60);
        }
        inputStream[i].leftDNA[SEQLEN - 1] = '\0';
        inputStream[i].rightDNA[SEQLEN - 1] = '\0';
        inputStream[i].leftQuality[SEQLEN - 1] = '\0';
        inputStream[i].rightQuality[SEQLEN - 1] = '\0';
        inputStream[i].leftRow = leftRow;
        inputStream[i].rightRow = leftRow;
    }

    inputStream[i].leftRow = -1;

    readCycles++;
    (*inputSize) = NREAD;
    (*eof) = readCycles > READCYCLES;

    return inputStream;

}


int main(int argc, char* argv[]){

    short eof = 0;
    unsigned long int inputSize = 0;
    MatePair* inputStream = NULL;

    while (!eof){
        inputStream = readStream(inputStream, &eof, &inputSize);
        free(inputStream);
        inputStream = NULL;
    }

    return 0;

}

I forgot to mention that, but before posting here, instead of calling free(inputStream), I tried invoking freeMemory(inputStream). Not sure if it’s the correct way of doing it, though.

void freeMemory(MatePair* memblock){

    for ( ; memblock->leftRow != 1; memblock++){
        free(memblock -> leftDNA);
        free(memblock -> leftQuality);
        free(memblock -> rightDNA);
        free(memblock -> rightQuality);
    }

}

Answer 1

You’re not freeing all members allocated within the read loop, hence you’re losing memory eahc time. Remember, you have to free everything you allocate with a malloc, not just your array.

Ok, Just look at your edit, and your freeMemory is still wrong. Try this;

void freeMemory(MatePair* inputStream)
{
    for (i = 0; i < NREAD; i++){
        free(inputStream[i].leftDNA);
        free(inputStream[i].leftQuality);
        free(inputStream[i].rightDNA);
        free(inputStream[i].rightQuality);
    }
    free (inputStream);
  }

Your free(memblock) was in the loop, which it shouldn’t have been, and I’d tend to use the same iteration sequence on freeing as mallocing. You also need to error check after each malloc, and decide what to do with a NULL at that point.