I have the following code to draw an openGL scene
void QGLDiagramWidget::paintGL()
{
glClearColor(0.2f, 0.0f, 0.5f, 0.0f);
glEnable(GL_DEPTH_TEST);
glDepthFunc(GL_LESS);
glEnable(GL_MULTISAMPLE);
... load shaders...
... load buffers...
... load textures...
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
// Set viewport area
glViewport(0, 0, this->size().width(), this->size().height());
GLint view[4];
glGetIntegerv(GL_VIEWPORT, view);
glLoadIdentity();
gluPickMatrix(1, 1, 1, 1, view); // THIS IS THE RELEVANT CODE
// I don't use a fixed pipeline, so let's store this pickmatrix in a variable
Matrix pickingPerspMatrix;
glGetFloatv(GL_PROJECTION_MATRIX, pickingPerspMatrix); // This is successful in my code
// Create the projection matrix
gl_projection.setToIdentity();
gl_projection *= pickingPerspMatrix; // APPLY THE PICKING MATRIX
gl_projection.perspective(45.,((GLfloat)this->size().width())/((GLfloat)this->size().height()),0.1f,100.0f);
... now draw the entire scene with shaders, there's a camera matrix (view) and a model matrix (model) ...
}
As long as I know, this should take a pick matrix from the rect from bottom-left pixel 1,1 (the first of the scene) with width 1 and height 1 and multiply it by the identity matrix and then apply the perspective matrix over it.
The goal is: centering just my pixel area on the viewport and render whatever is really visible there.
Turns out that whatever value I put into the gluPickMatrix, the object in my viewport is ALWAYS visible, just shrinked or distorted or deformed, but always int he viewport area. And this is wrong since asking for the pick matrix from point (0;0) plus width 1 and height 1 should return an empty area.
This is the image that I obtain to help you understand the problem

Why is this happening? What is a pick matrix supposed to do? Translate the objects out of the scene? Since I use programmable pipelines in my shaders (they’re basically simple shaders to load a texture on the object and use phong lighting), what could cause that object deformation and its constant presence on the scene?
The pickmatrix is used to restrict rendering to a small part of the viewport. Read about gluPickMatrix here: http://www.opengl.org/sdk/docs/man/xhtml/gluPickMatrix.xml
Picking won’t work with your programmable pipeline. Picking works by associating each drawcall with a “name” (a number that represents the pickable object) and when rendering the scene with picking, it return any pixels/fragments.
The details are explained here:
http://content.gpwiki.org/index.php/OpenGL:Tutorials:Picking
and here
web.engr.oregonstate.edu/…/Picking/picking.pdf
You might be better off avoiding deprecated functionality and use color picking, render each object with a unique solid color, then use glReadPixels to find out whats under the mouse.
Colorpicking is well explained here: http://content.gpwiki.org/index.php/OpenGL_Selection_Using_Unique_Color_IDs
Another possibility is using ray-primitive tests for selection.