Question

I’m trying to experiment with malloc and free in assembly code (NASM, 64 bit).

I have tried to malloc two arrays, each with space for 2 64 bit numbers. Now I would like to be able to write to their values (not sure if/how accessing them will work exactly) and then at the end of the whole program or in the case of an error at any point, free the memory.

What I have now works fine if there is one array but as soon as I add another, it fails on the first attempt to deallocate any memory 🙁

My code is currently the following:

extern printf, malloc, free


LINUX        equ     80H      ; interupt number for entering Linux kernel
EXIT         equ     60       ; Linux system call 1 i.e. exit ()

segment .text
    global      main

main:
    push dword 16       ; allocate 2 64 bit numbers
    call malloc
    add rsp, 4          ; Undo the push
    test  rax, rax      ; Check for malloc failure
    jz    malloc_fail
    mov r11, rax        ; Save base pointer for array

    ; DO SOME CODE/ACCESSES/OPERATIONS HERE

    push dword 16       ; allocate 2 64 bit numbers
    call malloc
    add rsp, 4          ; Undo the push
    test  rax, rax      ; Check for malloc failure
    jz    malloc_fail
    mov r12, rax        ; Save base pointer for array

    ; DO SOME CODE/ACCESSES/OPERATIONS HERE

malloc_fail:
    jmp dealloc

; Finish Up, deallocate memory and exit
dealloc:
    dealloc_1:
        test  r11, r11    ; Check that the memory was originally allocated
        jz    dealloc_2   ; If not, try the next block of memory
        push r11          ; push the address of the base of the array
        call free         ; Free this memory
        add rsp, 4
    dealloc_2:
        test  r12, r12
        jz    dealloc_end
        push r12
        call free
        add rsp, 4
dealloc_end:
    call os_return        ; Exit

os_return:
    mov  rax, EXIT
    mov  rdi, 0
    syscall

Answer 1

I’m assuming the above code is calling the C functions malloc() and free()…

If 1st malloc() fails, you arrive at dealloc_1 with whatever garbage is in r11 and r12 after returning from the malloc().

If 2nd malloc() fails, you arrive at dealloc_1 with whatever garbage is in r12 after returning from the malloc().

Therefore, you have to zero out r11 and r12 before doing the first allocation.

Since this is 64-bit mode, all pointers/addresses and sizes are normally 64-bit. When you pass one of those to a function, it has to be 64-bit. So, push dword 16 isn’t quite right. It should be push qword 16 instead. Likewise, when you are removing these parameters from the stack, you have to remove exactly as many bytes as you’ve put there, so add rsp, 4 must change to add rsp, 8.

Finally, I don’t know which registers malloc() and free() preserve and which they don’t. You may need to save and restore the so-called volatile registers (see your C compiler documentation). The same holds for the code not shown. It must preserve r11 and r12 so they can be used for deallocation. EDIT: And I’d check if it’s the right way of passing parameters through the stack (again, see your compiler documentation).

EDIT: you’re testing r11 for 0 right before 2nd free(). It should be r12. But free() doesn’t really mind receiving NULL pointers. So, these checks can be removed.

Pay attention to your code.