Question

Im using gluUnProject() to get the screen 2d coordinate in 3d world coordinate. I take 4 positions from each corner of the screen to get the area of visible objects.

How to check which points are inside that “rectangle” ?, i have no idea about the terms or anything. The image below shows what that “rectangle” looks like:

Answer 1

Are you trying to find which 3D point are visible by a camera? If so, you might find some interesting informations on this website: http://www.lighthouse3d.com/opengl/viewfrustum/.

In the following image, we can see the view frustum and your selection frustum (in red). Applying frustum visibility checks to your selection frustum should the same algorithm as the one used for frustum culling.

If you want a quick and non optimized solution:

GLdouble model_view[16];
glGetDoublev(GL_MODELVIEW_MATRIX, model_view);

GLdouble projection[16];
glGetDoublev(GL_PROJECTION_MATRIX, projection);

GLint viewport[4];
glGetIntegerv(GL_VIEWPORT, viewport);

for(unsigned i=0; i<points.size(); ++i){
  GLdouble winX, winY, winZ;
  gluProject(points[i].x, points[i].y, points[i].z, model_view, projection, viewport, &winX, &winY, &winZ);
  if(selectionMinX <= winX && winX <= selectionMaxX && selectionMinY <= winY && winY <= selectionMaxY && winZ>=0 && winZ<=1){
    /// point is selected
  }
}