I implemented a Clip-Space Frustum Culling on the CPU.
In a simple reduced case, I just create a rectangle based on 4 different points which I’m going to render in GL_LINES modes.
But sometimes, it seems to me, I get wrong results. Here is an example:
In this render pass, my frustum culling computation detects, all points would be outside the positive y coordinate in NDC coordinates.
Here is the input:
Points:
P1: -5000, 3, -5000
P2: -5000, 3, 5000
P3: 5000, 3, 5000
P4: 5000, 3, -5000
MVP (rounded):
1.0550 0.0000 -1.4521 1138.9092
-1.1700 1.9331 -0.8500 -6573.4885
-0.6481 -0.5993 -0.4708 -2129.3858
-0.6478 -0.5990 -0.4707 -2108.5455
And the calculations (MVP * Position)
P1 P2 P3 P4
3124 -11397 -847 13674
3532 -4968 -16668 -8168
3463 -1245 -7726 -3018
3482 -1225 -7703 -2996
And finally, transformed by the perspective divide (w-component)
P1 P2 P3 P4
0.897 9.304 0.110 -4.564
1.014 4.056 2.164 2.726
0.995 1.016 1.003 1.007
1.0 1.0 1.0 1.0
As you can see, all transformed points have their y component greater than 1 and should be outside the Viewing Frustum.
I already double checked my matrices. I also made a transform feedback in the Vertex Shader to be sure, I use the same matrices for computation on CPU and GPU. Even the result of MVP * Point is the same in my Vertex Shader as on the GPU. My rendering pipeline is as simple as possible.
The vertex Shader is
vColor = aColor;
gl_Position = MVP * aVertexPosition;
//With Transform Feedback enabled
//transOut = MVP * aVertexPosition
And the fragment Shader
FragColor = vColor;
So the Vertex Shader has the same results as my CPU computations.
But still, they are lines drawn on the screen!
Any Ideas why there are lines?
Do I do something wrong with the perspective divide?
What do I have to do, to detect, that this rectangle should not be culled, because there is at least one line visible (basically a strip of another line is visible as well in this example)
If it helps: The visible red line is the one between P1 and P2…
[Edit] Now I implemented a world space culling by computing the camera frustum normals and hesse normal form equations. This works fine with correct recognition. Sadly I do need correct computations in clip space, since I’m going to make other computations with that points. Someone any ideas?
Here is my computation code:
int outOfBoundArray[6] = {0, 0, 0, 0, 0, 0};
std::vector<glm::dvec4> tileBounds = activeElem->getTileBounds(); //Same as I use for world space culling
const glm::dmat4& viewProj = cam->getCameraTransformations().viewProjectionMatrix; //Same camera which I use for world space culling.
for (int i=0; i<tileBounds.size(); i++) {
//Apply ModelViewProjection Matrix, to Clip Space
glm::dvec4 transformVec = viewProj * tileBounds[i];
//To NDC space [-1,1]
transformVec = transformVec / transformVec[3];
//Culling test
if ( transformVec.x > 1.0 ) outOfBoundArray[0]++;
if ( transformVec.x < -1.0 ) outOfBoundArray[1]++;
if ( transformVec.y > 1.0 ) outOfBoundArray[2]++;
if ( transformVec.y < -1.0 ) outOfBoundArray[3]++;
if ( transformVec.z > 1.0 ) outOfBoundArray[4]++;
if ( transformVec.z < -1.0 ) outOfBoundArray[5]++;
//Other computations...
}
for (int i=0; i<6; i++) {
if (outOfBoundArray[i] == tileBounds.size()) {
return false;
}
}
return true;

The problem would appear that the sign of w (the third component) is different between P1 and P2. This will cause all kinds of trouble due to the nature of projective geometry.
Even though the points are on the same side of the NDC, the line that gets drawn actually goes through infinity: consider what happens when you linearly interpolate between P1 and P2 and do the division by w at each point separately; what happens is that when w approaches zero, the y value is not exactly zero and therefore the line zooms off to infinity. And then wraps around from the other side.
Projective geometry is a weird thing 🙂
But, for a solution, make sure that you clip those lines that cross the w=0 plane at the positive side of that plane, and you are set – your code should then work.