I’m working on an OpenGL 3 renderer for a GUI toolkit called Gwen. I nearly have everything working, but I’m having some issues getting everything to blend correctly. I’ve sorted the triangles by which texture they use and packed them into a VBO, so with the Unit Test, it basically boils down into 3 layers: Filled Rects with no texture, Text, and the windows, buttons, etc that use a skin texture.
The Filled Rects are usually drawn on top of everything else and blended in, but the background behind everything is also a Filled Rect, so I can’t count on that. There is a Z-value conflict if you draw them last (ex: the windows have a textured shadow around the edges that turns black because the background fails the depth test) and a blending/z-value conflict if you draw them first (ex: some of the selection highlights get drawn on top of instead of blending like they’re supposed to).
I can’t count on being able to identify any specific layer except the Filled Rects. The different layers have a mix of z-values, so I can’t just draw them in a certain order to make things work. While writing this, I thought of a simple method of drawing the triangles sorted back to front, but it could mean lots of little draw calls, which I’m hoping to avoid. Is there some method that involves some voodoo magic blending that would let me keep my big batches of triangles?
You’re drawing a GUI; batching shouldn’t be your first priority for the simple fact that a GUI just doesn’t do much. A GUI will almost never be your performance bottleneck. This smells of premature optimization; first, get it to work. Then, if it’s too slow, make it work faster.
There is no simple mechanism for order-independent transparency. Your best bet is to just render things in the proper Z order.