I am doing Finite Difference computation (Stencil Computation) on GPU (Fermi) using CUDA. When I tested my code using CUDA profiler, I found the occupany was 0.333. After I ordered my computation and increased the occupany to 0.677, the execution time of the kernel didn’t decrease but increased. In other words, there was a decrease in performance when the occupany got increased by 1/3.
My question is:
Does the performance of the kernel depend on the computation irrespective of the occupancy?
The answer is “it depends”, both on the characteristics of your workload and on how you define performance. Generally speaking, if your bottleneck is math throughput you’re often fine with a lower occupancy (12.5%-33%), but if your bottleneck is memory then you usually want a higher occupancy (66% or higher). This is just a rule of thumb, not an absolute rule. Most kernels fall somewhere in the middle but there are exceptions at both extremes.
Occupancy is the maximum number of threads of your kernel that can be active at once (limited by register count per thread or other resources) divided by the maximum number of threads the GPU can have active when not limited by other resources. Active means the thread has hardware resources assigned and is available for scheduling, not that it has any instructions executing on a given clock cycle.
After issuing instruction i for a thread, the instruction i+1 for that thread might not be able to run immediately, if it depends on the result of instruction i. If that instruction is a math instruction, the result will be available in a few clock cycles. If it’s a memory load instruction, it might be 100s of cycles. Rather than waiting, the GPU will issue instructions from some other thread who’s dependencies are satisfied.
So if you’re mostly doing math, you only need a few (few in GPU terms; on a CPU it would be considered many) threads to hide the few cycles of latency from math instructions, so you can get away with low occupancy. But if you’ve got a lot of memory traffic, you need more threads to ensure that some of them are ready to execute on every cycle, since each one spends a lot of time “sleeping” waiting for memory operations to complete.
If the algorithmic changes you made to increase occupancy also increased the amount of work done on each thread, and if you already had enough threads to keep the GPU busy, then the change will just slow you down. Increasing occupancy only improves performance up to the point where you have enough threads to keep the GPU busy.