Question

I’m currently coding an implementation of the Burrows-Wheeler Transform which requires sorting a (large) array. Since I want to parallelize parts of the code of a recursive sort function, I decided to allocate some of the local variables in the heap (using malloc()) to prevent stack overflows or – at least- make the program die gracefully. That raised a whole new bunch of issues. I stripped the code down to the essential part (i.e., what causes the issues).

The following code compiles without errors. The resulting program works fine when compiled with icc, but it crashes (randomly) when compiled with gcc or tcc.

#include <stdlib.h>
#include <stdio.h>
#include <string.h>

unsigned char *block;
int *indexes, *indexes_copy;

void radixsort(int from, int to, int step)
{
    int *occurrences, *first_index_of, i;
    if((occurrences = malloc(1024)) == 0)
        exit(1);
    if((first_index_of = malloc(1024)) == 0)
        exit(1);
    memset(occurrences, 0, 1024);
    for(i = from; i < to; i++)
        occurrences[block[indexes[i] + step]]++;
    first_index_of[0] = from;
    for(i = 0; i < 256; i++)
        first_index_of[i + 1] = first_index_of[i] + occurrences[i];
    memset(occurrences, 0, 1024);
    memcpy(&indexes_copy[from], &indexes[from], 4 * (to - from));
    for(i = from; i < to; i++)
        indexes[first_index_of[block[indexes_copy[i] + step]] + occurrences[block[indexes_copy[i] + step]]++] = indexes_copy[i];
    for(i = 0; i < 256; i++)
    if(occurrences[i] > 1)
        radixsort(first_index_of[i], first_index_of[i] + occurrences[i], step + 1);
    free(occurrences);
    free(first_index_of);
}

int main(int argument_count, char *arguments[])
{
    int block_length, i;
    FILE *input_file = fopen(arguments[1], "rb");
    fseek(input_file, 0, SEEK_END);
    block_length = ftell(input_file);
    rewind(input_file);
    block = malloc(block_length);
    indexes = malloc(4 * block_length);
    indexes_copy = malloc(4 * block_length);
    fread(block, 1, block_length, input_file);
    for(i = 0; i < block_length; i++)
        indexes[i] = i;
    radixsort(0, block_length, 0);
    exit(0);
}

Even when the input is a very small text file (around 50 bytes), the program is very unstable with the latter two compilers. It works with ~50% probability. In the other cases, it crashes in the 2nd or 3rd iteration of radixsort when calling malloc(). It always crashes when the input file is bigger (around 1 MiB). Also in the 2nd or 3rd iteration…

Manually increasing the heap doesn’t do any good. Either way, it shouldn’t. If malloc() can’t allocate the memory, it should return a NULL pointer (and not crash).

Switching back from heap to stack makes the program work with either compiler (as long as the input file is small enough).

So, what am I missing / doing wrong?

Answer 1

if((occurrences = malloc(1024)) == 0)

make that:

if((occurrences = malloc(1024 * sizeof *occurences)) == 0)

But there are more problems …

UPDATE (the 1024 = 4 * 256 seems merely stylistic …)

for(i = 0; i < 256; i++)
    first_index_of[i + 1] = first_index_of[i] + occurrences[i];

The [i+1] index will address the array beyond its size;