Question

I try to use the "stencil buffer" to display a part of my rendering from a texture, but my render is displayed without any mask effect.

It’s for a 2D iOS project, with OpenGL ES 2.0
This is the concerned part of my code :

glClear( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT );
glEnable( GL_STENCIL_TEST );


// mask rendering
glColorMask( GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE );
glStencilFunc( GL_ALWAYS, 1, 1 );
glStencilOp( GL_KEEP, GL_KEEP, GL_REPLACE );

glBindTexture(GL_TEXTURE_2D, _maskTexture);
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);  


// scene rendering
glColorMask( GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE ); 
glStencilFunc( GL_EQUAL, 1, 1 ); 
glStencilOp( GL_KEEP, GL_KEEP, GL_KEEP );  

glBindTexture(GL_TEXTURE_2D, _viewTexture);
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);  

Any help would be greatly appreciated !

(As usual for a French developer, sorry for my English !)

precision : "_maskTexture" is a black & white picture.

Solution :

---

I finnaly solved my problem with the indications of rotoglub and tim.
Thank you both.

1/ The stencil buffer has not been created correctly.
It should be initialized like this:

glBindRenderbufferOES(GL_RENDERBUFFER_OES, depthStencilRenderbuffer);
glRenderbufferStorageOES(GL_RENDERBUFFER_OES, 
                             GL_DEPTH24_STENCIL8_OES, 
                             backingWidth, 
                             backingHeight);

It’s the principal reason why my rendering was not affected by my mask.

2/ To be able to make use of a texture as a mask, I replaced the black color with an alpha channel and enable blending in my rendering.

My final rendering code looks like this :

glEnableClientState(GL_VERTEX_ARRAY);
glEnableClientState(GL_TEXTURE_COORD_ARRAY);
    
glVertexPointer(2, GL_FLOAT, 0, vertices);
glTexCoordPointer(2, GL_FLOAT, 0, texcoords);
    
glClearStencil(0); 
glClearColor (0.0,0.0,0.0,1);
glClear (GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT); 
    

// mask rendering
glColorMask( GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE );
glEnable(GL_STENCIL_TEST);
glEnable(GL_ALPHA_TEST);
glBlendFunc( GL_ONE, GL_ONE );
glAlphaFunc( GL_NOTEQUAL, 0.0 );
glStencilFunc(GL_ALWAYS, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
    
glBindTexture(GL_TEXTURE_2D, _mask);
glDrawArrays(GL_TRIANGLE_STRIP, 4, 4);  
    
// scene rendering
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE); 
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
    
glDisable(GL_STENCIL_TEST);
glDisable(GL_ALPHA_TEST);
glBindTexture(GL_TEXTURE_2D, _texture);
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);  

Answer 1

Simply, the problem is that you’re just drawing a texture to the scene without doing any testing of what’s in the texture. The stencil buffer doesn’t care about the colors in the texture, it just checks:

Did you draw a fragment ? (Update stencil buffer) : (Don't update stencil buffer);

You’re drawing a fragment for every pixel of your texture, so any mask effect in the texture is useless.

If you want to mask with a texture, you need to discard any fragments that you don’t want updated in the stencil buffer.

This is either done with discard keyword in shader, or glAlphaTest/glAlphaFunc in OpenGLES 1.1