I have some objects in the scene, some may occlude others. When I click the mouse or drag-select to get a selection rectangle, I want to select/pick only the objects that I can see from this perspective. The application currently uses GL_SELECT render mode but as we know, this selects occluded objects too. Also, I read that this is deprecated in OpenGL 3.
There are two methods that are currently appealing to me. First is Object Selection using the Back Buffer (Red book, chapter 14): setting the colour of each object to it’s object id and reading the colour of pixels from the back frame buffer. The second is occlusion queries (superbible, 4th ed, chap 13).
Other approaches I have ruled out are looking at the min/max z values in the selection buffer and doing custom ray/object detection outside of GL.
I have some questions:
1) If GL_SELECT is deprecated in recent OpenGL, what alternatives are developers supposed to be using?
2) I’ve only ever read about occlusion queries being employed to speed up rendering. Can they be used for selection/picking, and are there drawbacks?
3) The existing application has a handful of glColorXXX calls. If I went the back buffer route, and used glColorMask(FALSE,FALSE,FALSE,FALSE), will this effectively turn the glColourXXX calls into calls that have no effect, thereby letting me control the colour in a single place when rendering in select mode?
4) Which route is best/canonical?
I decided to implement the selection using the back buffer. Here’s my attempt to answer my questions:
If GL_SELECT is deprecated in recent OpenGL, what alternatives are developers supposed to be using?
I think it’s best to not employ OpenGL to do this task but to use spatial acceleration structures as user chamber85 suggested in comments to the original question.
I’ve only ever read about occlusion queries being employed to speed up rendering. Can they be used for selection/picking, and are there drawbacks?
I’m sure they could but one would need to know all the objects they want to query for occlusion before the draw. Using back buffer and colour selection, one can just see what is under the cursor or within a rectangular region and filter from there.
The existing application has a handful of glColorXXX calls. If I went the back buffer route, and used glColorMask(FALSE,FALSE,FALSE,FALSE), will this effectively turn the glColorXXX calls into calls that have no effect, thereby letting me control the colour in a single place when rendering in select mode?
The answer is no. Calling glColorMask() with all GL_FALSE parameters will not mean that glColor3ub() calls (for example) will not be honoured. It simply specifies a filter/mask for colours just before they are written to the colour buffer. The original thought was to set the colour to the object id, then call glColorMask() to ignore all subsequent glColorXXX() calls. This strategy is doomed as the colour representing the object id would also be masked out.
4) Which route is best/canonical?
I would say the back buffer colour selection is generally best as it doesn’t require setting up the occlusion queries before/during the draw.