Question

I want to set up multiple stencils in OpenGL and then draw into multiple combinations of them.

For example:
I have two rectangular stencils like this:

Then I want to draw into:

1. anywhere
2. left rectangle (blue+purple)
3. right rectangle (purple+red)
4. middle rectangle (purple)
5. whole colored area (blue+purple+red)

I have found that it is possible to declare multiple stencils in different bits of stencil buffer but I don’t know how.
How do I setup glStencilOp and glStencilFunc for this to work?

Or can I (should I) use glScissor for that?

Answer 1

I don’t currently know, if it’s possible to configure the stencil buffer for being able to do all the above 5 steps without making any changes to the stencil buffer between them. It would be easy if glStencilOp provided bitwise OR, but it doesn’t and with just using increment or decrement you would have to draw the rectangles multiple times.

But if the regions are always rectangles, why not just use the scissor test? So the first 3 steps (or actually 2 and 3) can be done by just setting the rectangle’s region with glScissor and enabling the scissor test (glEnable(GL_SCISSOR_TEST)).

For the middle one (step 4) you either compute the purple intersection rectangle yourself and use the scissor test again, or you use the stencil test:

glEnable(GL_STENCIL_TEST);
glStencilFunc(/*whatever*/);
glStencilOp(GL_INCR, GL_INCR, GL_INCR);    //increase the stencil value
//draw both rectangles

glStencilFunc(GL_EQUAL, 2, 0xFFFFFFFF);    //only draw where both rectangles are
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);    //don't change the stencil buffer
//draw things

So we first draw both rectangles and increase the stencil value everywhere they are drawn. Then we draw our things everywhere the stencil value is 2, meaning both rectangles were drawn.

For the 5th step you use the same stencil buffer, but with

glStencilFunc(GL_LEQUAL, 1, 0xFFFFFFFF);

for the second pass. This way you draw something everywhere the stencil buffer is at least 1, meaning at least one rectangle was drawn.

For more than two rectangles it can get more complicated and you need to play around a bit to find the most optimal way.