Question

I am having some issues with cudaGetDeviceCount returning zero if used in mpirun with -np greater than 2. The portion of code from a much larger program is:

bool cpuInterfaces::checkGPUCount(int gpusPerMachine){

    int GPU_N;
    cudaGetDeviceCount(&GPU_N);

    //if the gpu count on this node does not equal what was given in fvSolution, return false
    return ((gpusPerMachine>GPU_N || gpusPerMachine < 1)? false : true);

}

The larger portion of code is located at cufflink-source file. If a run is executed using mpirun -np 2 somethingsomething then everything runs fine and cudaGetDeviceCount returns 4, where as if mpirun -np 4 somethingsomething will result in cudaGetDeviceCount returning zero. Are there special cases of using cudaGetDeviceCount in MPI that I’m not aware of? Any help or suggestions would be helpful.

Answer 1

Thanks to those that provided some answers. This turned out to be an issue with a driver. After waiting for a bit, a driver update resolved this question.