Question

My original problem, is that I have functions with a long list of arguments, that exceeded the memory that is allowed to be passed as an argument to a cuda kernel (I don’t remember how many bytes, because it’s been a while since I dealt with that). So, the way I bypassed this problem, was to define a new structure that its members are pointers pointing to other structures that I can dereference from within the kernel later.

… this is where the current problem begins: at the point where I’m trying to dereference the pointers (members of the structure I created earlier) from within the kernel, I get CUDA_EXCEPTION_5, Warp Out-of-range Address
…from the cuda-gdb. And of top of that, the kernel name and arguments (which are reported ‘not live at this point’ which cuda-gdb gives as the one with the error, is not one that I created in my code.

Now, for the more specifics :

here are the structures involved:

typedef struct {

    int strx;
    int stry;
    int strz;
    float* el;

} manmat;

typedef struct {

    manmat *x;
    manmat *y;
    manmat *z;

} manmatvec;

here’s how i’m trying to group the kernel’s arguments inside the main:

int main () {

...
...

    manmat resu0;
    resu0.strx = n+2;       resu0.stry = m+2;       resu0.strz = l+2;
    if (cudaMalloc((void**)&resu0.el,sizeof(float) * (n+2)*(m+2)*(l+2)) != cudaSuccess) cout << endl << " ERROR allocating memory for manmat resu0" << endl ;
    manmat resv0;
    resv0.strx = n+2;       resv0.stry = m+2;       resv0.strz = l+2;
    if (cudaMalloc((void**)&resv0.el,sizeof(float) * (n+2)*(m+2)*(l+2)) != cudaSuccess) cout << endl << " ERROR allocating memory for manmat resv0" << endl ;
    manmat resw0;
    resw0.strx = n+2;       resw0.stry = m+2;       resw0.strz = l+2;
    if (cudaMalloc((void**)&resw0.el,sizeof(float) * (n+2)*(m+2)*(l+2)) != cudaSuccess) cout << endl << " ERROR allocating memory for manmat resw0" << endl ;
    manmatvec residues0 ;

    residues0.x = &resu0;
    residues0.y = &resv0;
    residues0.z = &resw0;

    exec_res_std_2d <<<numBlocks2D, threadsPerBlock2D>>> (residues0, ......) ;

 .....
}

… and this is what happens in the kernel :

__global__ void exec_res_std_2d (manmatvec residues, ......) {

    int i = blockIdx.x * blockDim.x + threadIdx.x;
    int k = blockIdx.y * blockDim.y + threadIdx.y;

    manmat *resup;
    manmat *resvp;
    manmat *reswp;

    resup = residues.x;
    resvp = residues.y;
    reswp = residues.z;

    manmat resu, resv, resw ;

    resu.strx = (*resup).strx;     //LINE 1626
    resu.stry = (*resup).stry;
    resu.strz = (*resup).strz;
    resu.el = (*resup).el;

    resv = *resvp;
    resw = *reswp;

    .....
}

and finally, this is what cuda-gdb gives as output :

..................
[Launch of CUDA Kernel 1065 (exec_res_std_2d<<<(1,2,1),(32,16,1)>>>) on Device 0]
[Launch of CUDA Kernel 1066 (exec_res_bot_2d<<<(1,2,1),(32,16,1)>>>) on Device 0]

Program received signal CUDA_EXCEPTION_5, Warp Out-of-range Address.
[Switching focus to CUDA kernel 1065, grid 1066, block (0,0,0), thread (0,2,0), device 0, sm 0, warp 2, lane 0]
0x0000000003179020 in fdivide<<<(1,2,1),(32,16,1)>>> (a=warning: Variable is not live at this point. Value is undetermined.
..., pt=warning: Variable is not live at this point. Value is undetermined.
..., cells=warning: Variable is not live at this point. Value is undetermined.
...) at ola.cu:1626
1626    ola.cu: No such file or directory.
    in ola.cu

I have to note that I haven’t defined ANY function , __device__ or __global__ in my code called fdivide…..

Also, it might be important to say that, in the beginning of the run of the program inside the debugger, despite the fact that I compile my cuda c files with -arch=sm_20 -g -G -gencode arch=compute_20,code=sm_20, I get,

[New Thread 0x7ffff3b69700 (LWP 12465)]
[Context Create of context 0x1292340 on Device 0]
warning: no loadable sections found in added symbol-file /tmp/cuda-dbg/12456/session1/elf.1292340.1619c10.o.LkkWns
warning: no loadable sections found in added symbol-file /tmp/cuda-dbg/12456/session1/elf.1292340.1940ad0.o.aHtC7W
warning: no loadable sections found in added symbol-file /tmp/cuda-dbg/12456/session1/elf.1292340.2745680.o.bVXEWl
warning: no loadable sections found in added symbol-file /tmp/cuda-dbg/12456/session1/elf.1292340.2c438b0.o.cgUqiP
warning: no loadable sections found in added symbol-file /tmp/cuda-dbg/12456/session1/elf.1292340.2c43980.o.4diaQ4
warning: no loadable sections found in added symbol-file /tmp/cuda-dbg/12456/session1/elf.1292340.2dc9380.o.YYJAr5

Any answers or hints, or suggestions that could help me with this issue are very welcome!
please note that i’ve only recently started programming with cuda-c, and I’m not very experienced with cuda-gdb. Most of the debugging I did in C code I did it ‘manually’ by checking the output at various points of the code….

Also, this code is running on tesla M2090, and is also compiled to run on 2.0 architecture.

Answer 1

This will be a problem:

manmatvec residues0 ;

    residues0.x = &resu0;
    residues0.y = &resv0;
    residues0.z = &resw0;

The resu0, resv0, and resw0 variables are allocated in host memory – on the host stack. You’re putting host addresses into the manmatvec structure, then passing the manmatvec into the kernel. On the receiving end, the CUDA code cannot access the host memory addresses provided in the structure.

If you’re going to pass the addresses of the resu0, resv0, resw0 variables, you need to allocate them from device memory.

I don’t know if this is the entire problem, but I’m pretty sure it’s a top contributor.