Question

I’m wondering if it is possible to simulate the effect of looking through the keyhole in OpenGL.

I have my 3D scene drawn but I want to make everythig black everything except a central circle.

I tried this solution but its doing the completely opposite of what I want:

// here i draw my 3D scene 

// Begin 2D orthographic mode
glMatrixMode(GL_PROJECTION);
glPushMatrix();
glLoadIdentity();

GLint viewport [4];
glGetIntegerv(GL_VIEWPORT, viewport);

gluOrtho2D(0, viewport[2], viewport[3], 0);

glMatrixMode(GL_MODELVIEW);
glPushMatrix();
glLoadIdentity();

// Here I draw a circle in the center of the screen
float radius=50;
glBegin(GL_TRIANGLE_FAN);
glVertex2f(x, y);
for( int n = 0; n <= 100; ++n )
{
    float const t = 2*M_PI*(float)n/(float)100;
    glVertex2f(x + sin(t)*r, y + cos(t)*r);
}
glEnd();

// end orthographic 2D mode
glMatrixMode(GL_PROJECTION);
glPopMatrix();
glMatrixMode(GL_MODELVIEW);
glPopMatrix();

What I get is a circle drawn in the center, but I would like to obtain its complementary…

Answer 1

Like everything else in OpenGL, there are a few ways to do this. Here are two off the top of my head.

Use a circle texture: (recommended)

1. Draw the scene.

2. Switch to an orthographic projection, and draw a quad over the entire screen using a texture which has a white circle at the center. Use the appropriate blending function:

   glEnable(GL_BLEND);
   glBlendFunc(GL_ZERO, GL_SRC_COLOR);
   /* Draw a full-screen quad with a white circle at the center */
   

Alternatively, you can use a pixel shader to generate the circular shape.

Use a stencil test: (not recommended, but it may be easier if you don’t have textures or shaders)

1. Clear the stencil buffer, and draw the circle into it.

   glEnable(GL_STENCIL_TEST);
   glStencilFunc(GL_ALWAYS, 1, 1);
   glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);
   /* draw circle */
   

2. Enable the stencil test for the remainder of the scene.

   glEnable(GL_STENCIL_TEST)
   glStencilFunc(GL_EQUAL, 1, 1);
   glStencileOp(GL_KEEP, GL_KEEP, GL_KEEP);
   /* Draw the scene */
   

Footnote: I recommend avoiding use of immediate mode at any point in your code, and using arrays instead. This will improve the compatibility, maintainability, readibility, and performance of your code — a win in all areas.