Question

My program experiences a seg fault in the middle of the loop iteration. After calling the function intermediate, the inter_value prints until inter_value[199][208], and then, I have a seg fault.
To make sure it is not out of bound access, I print the array inter_value first and the array prints without any problem.

Is this a symbol of running out of memory? The array ct and inter_value are created by malloc, key_byte is a static array.

D = 200;
K = 256;
for(j = 0; j < D; j++)
    for(i = 0; i < K; i++)
        printf("inter_value[%i][%i] = %i\n", j, i, inter_value[j][i]);

for(j = 0; j < D; j++) {
    for(i = 0; i < K; i++) {
        intermediate(ct[j][0], key_byte[i], &inter_value[j][i]);
        printf("inter_value[%i][%i] = %i\n", j, i, inter_value[j][i]);
        fflush(stdout);
    }
}
printf("rex\n");


for(j = 0; j < D; j++) {
    for(i = 0; i < K; i++) {
        hamming_dist(ct[j][0], inter_value[j][i], &h[j][i]);
    }
}

Function intermediate is here

void intermediate(unsigned char ct, unsigned char key_byte, unsigned char *inter_value){
    *inter_value = getSBoxInvert(ct^key_byte);
}

---

Edit 1: Declaration of arrays.

//initialize different intermediate values
inter_value = (unsigned char**)malloc(D * sizeof(unsigned char*));
if(inter_value == NULL){
    fprintf(stderr, "out of memory\n");
    return 0;
}
for(i = 0; i < D; i++){
    inter_value[i] = (unsigned char *)malloc(K * sizeof(unsigned char)); // this is fix to key size
    if(inter_value[i] == NULL){
        fprintf(stderr, "out of memory\n");
        return 0;
    }
}


//ct = malloc(row * sizeof(unsigned char*));
ct = (unsigned char**)malloc(D * sizeof(unsigned char*));
if(ct == NULL){
    fprintf(stderr, "out of memory\n");
    return 0;
}
for(i = 0; i < D; i++){
    //ct[i] = malloc(column * sizeof(unsigned char));
    ct[i] = (unsigned char *)malloc(column * sizeof(unsigned char));
    if(ct[i] == NULL){
        fprintf(stderr, "out of memory\n");
        return 0;
    }
}

unsigned char key_byte[256] = {0};

Edit 2: The print out before seg fault.

inter_value[199][233] = 214
inter_value[199][234] = 119
inter_value[199][2

Edit 3: gdb output (It seems it is pointing to another function)

Program received signal SIGSEGV, Segmentation fault.
0x0804a3f5 in hamming_dist (ct=31 ‘\037’, inter_value=203 ‘\313’, h=0x2) at cpa.cpp:53
53 *h = c;

Edit 4: After issue backtrace command from gdb…

#0 0x0804a3f5 in hamming_dist (ct=31 ‘\037’, inter_value=203 ‘\313’, h=0x2) at cpa.cpp:53

#1 0x0804aff5 in main (argc=3, argv=0xbffff2f4) at cpa.cpp:266

Edit 5: Add the hamming_dist function call and a printf call before it.

Edit 6: Initialization of h

int **h;
h = (unsigned int**)malloc(D * sizeof(unsigned int*));
if(h == NULL){
    fprintf(stderr, "out of memory\n");
    return 0;
}
for(i = 0; i < D; i++){
    h[i] = (unsigned int*)malloc(K * sizeof(unsigned int)); // this is fix to key size
    if(h[i] == NULL){
        fprintf(stderr, "out of memory\n");
        return 0;
    }
}

Edit 7: The hamming_dist function declaration.

void hamming_dist(unsigned char ct, unsigned char inter_value, int *h){
    int temp;
    temp = ct ^ inter_value;
    //then count No. of ones
    int c; // c accumulates the total bits set in v
    for (c = 0; temp; c++)
        temp &= temp - 1; // clear the least significant bit set

    *h = c;
}

Answer 1

Your seg fault is because &h[j][i] is 0x2 when passed as the value of the hamming_dist parameter h, which attempts to dereference it and store into it. This is apparently because of an earlier out-of-bounds store that overwrote h[j].

Note that writing out of bounds in a malloced buffer can have effects that don’t show up until arbitrarily later in your program. They may not show up at all … until you release the code and some customer runs it with inputs that happen to trigger the bug.