I am using the GL selection buffer to implement mouse picking. Unfortunately, OpenGL is returning hits in the selection buffer even for objects that are entirely occluded. For example, if there is a man hidden behind a wall, the selection buffer will include a hit record for the man even though he is not visible.
Selection is implemented in roughly the way described in the OpenGL Programming Guide: switch to the GL_SELECT render mode — glRenderMode(GL_SELECT) — render the scene, and then parse the selection buffer. The depth buffer and depth testing are enabled, but GL seems to ignore depth settings in GL_SELECT mode.
Is it possible for OpenGL to do depth culling in GL_SELECT mode? Is there another way of discarding hit records for hidden objects without re-implementing selection using another method?
The selection buffer will give you all the objects that match your mouse position regardless of depth from the camera. It’s up to you to determine whether you want the closest, furthest or all objects. Remember the mouse only works in a 2D world and is trying to do selection for a 3D space. Imagine a ray shooting out into the -z direction at the x,y coordinate that you clicked your mouse. All the objects that intersect that ray are returned in the selection buffer. Sounds like you want to choose the closest one.
See jerome’s tutorial and nehe’s tutorial on selection.