Question

In a pixel shader you can discard a pixel but I would imagine even a fast-fail shader called for every pixel takes a non-trivial time? Is there any way a vertex shader can discard an entire triangle… I am fairly sure a VS can’t access the primitive but are there any tricks by which we can get the same result?

Talking SM 3.0 here – for completeness discussion on newer versions is welcome.

Answer 1

The geometry shader can do that trivially as you probably know, but no such thing in SM3.

It’s harsh for the vertex shader to identify a triangle as such, since all the vertex shader sees is the individual vertices. But… in principle it’s possible.

If you set the w coordinate to zero, a point will be projected to infinity. Set the w coordinates of all points to zero, and the entire triangle is at infinity, so it won’t be rendered.

Alternatively, you could probably set gl_ClipDistance to zero or a value below zero (never really used those, but I guess this should work). That would mark the vertex as “behind” a clipping plane. If all three vertices are behind a clipping plane, the triangle is not visible.

You would need a way to identify the vertices that belong to the triangle that you want to discard, however… and that will not be easy unless you have a very special situation (such as for example knowing that you want to discard all triangles in some given bounding box, or all triangles in which the vertex attribute #5 is zero, or whatever).