Question

I’m having trouble mapping the iPhone camera output on a simple plane in OpenGLES 2.0. For some reason the OpenGL texture generated through the CVOpenGLESTextureCacheCreateTextureFromImage is flipped and rotated.

Here’s my situation: I made a Wavefront .obj loader that parses this simple plane object:

v -0.5 -0.5 0
v 0.5 -0.5 0
v -0.5 0.5 0
v 0.5 0.5 0

vt 0 0 0
vt 1 0 0
vt 0 1 0
vt 1 1 0

f 4/4 3/3 1/1 
f 2/2 4/4 1/1 

As far as I’m concerned my texture mapping works like it has to. I’ve verified this by mapping a simple png image onto the plane. It shows up correctly.

(Just for the record: I loaded the image in the iPhone coordinate system, flipped the image pixels to match the OpenGL’s top-left texture coordinate system and mapped everything). Result: works!

So, after shuffling my code around a bit and feeding the camera pixel buffer into my texture unit something goes wrong (or maybe this is intended behavior and I should counter that in one way or another? Any suggestions are welcome, as long as it isn’t CPU heavy code).

Here’s my code: some of it is inspired by Brad Larson‘s ColorTracking example and Apple’s GLCameraRipple example.

Code to capture the output:

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
    CVImageBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    [self.delegate processNewCameraFrame:pixelBuffer];
}

After passing around the image buffer ref it ends up here:

- (void)updateCameraFrame:(CVImageBufferRef)pCameraFrame
{
    if(cameraTextureMaterial)
    {
        cameraTextureMaterial->updateWithCameraFrame(pCameraFrame);
    }
}

CamerTextureMaterial:

void CameraTextureMaterial::updateWithCameraFrame(CVImageBufferRef pixelBuffer)
{
    if(!_videoTextureCache)
    {
        cout << "No video texture cache" << endl;
        return;
    }

    cleanup();

    CVReturn err;
    size_t width = CVPixelBufferGetWidth(pixelBuffer);
    size_t height = CVPixelBufferGetHeight(pixelBuffer);

    glActiveTexture(GL_TEXTURE0);

    err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
                                                       _videoTextureCache,
                                                       pixelBuffer,
                                                       NULL,
                                                       GL_TEXTURE_2D,
                                                       GL_RGBA,
                                                       width,
                                                       height,
                                                       GL_BGRA,
                                                       GL_UNSIGNED_BYTE,
                                                       0,
                                                       &_lumaTexture);
    if (err)
    {
        cout << "Error at CVOpenGLESTextureCacheCreateTextureFromImage" << endl;
    }

    glBindTexture(CVOpenGLESTextureGetTarget(_lumaTexture), CVOpenGLESTextureGetName(_lumaTexture));
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
}

I’m excluding initialization and cleanup of the texture cache since this doesn’t matter for my texture orientation problem.

Any suggestions on how to get the correct orientation for my texture?

Answer 1

Solving this issues was actually pretty easy by choosing your texture coordinates wisely. I rotated, flipped and cropped the camera texture by mapping it with the coordinates below and rendered the orthogonal scene to a new texture which I then use in my scene.

Coordinates I used:

void CameraClipPlaneObject3D::init()
{
    // Create plane object
    vertices = shared_ptr<Vertex3DVector>(new Vertex3DVector());
    vertices->push_back(Vertex3D(1, -1, 0));
    vertices->push_back(Vertex3D(1, 1, 0));
    vertices->push_back(Vertex3D(-1, 1, 0));
    vertices->push_back(Vertex3D(-1, -1, 0));

    // Create texture coords
    textureCoords = shared_ptr<Texture2DVector>(new Texture2DVector());
    textureCoords->push_back(Texture2D(0.875, 0));
    textureCoords->push_back(Texture2D(0.125, 0));
    textureCoords->push_back(Texture2D(0.125, 1));
    textureCoords->push_back(Texture2D(0.875, 1));

    // Create indices
    indices = shared_ptr<IndexVector>(new IndexVector());
    indices->push_back(0);
    indices->push_back(1);
    indices->push_back(2);
    indices->push_back(2);
    indices->push_back(3);
    indices->push_back(0);
}

Here’s how I render the plane to a new texture:

1. first create a texture like you normally do (don’t forget texture filtering)
2. then create a framebuffer like you normally do
3. attach the texture colorbuffer to the color attachment of this new framebuffer:
4. In your rendering loop, bind to this new framebuffer and render your scene.

Some sample code:

void TextureRenderTarget::init()
{
    // Create texture to render to
    texture = new RenderTexture(width, height, GL_LINEAR, GL_LINEAR);

    // Create framebuffer and attach texture to framebuffer
    framebuffer = new Framebuffer(width, height);
    framebuffer->attachTexture(texture);
}

Attach code:

void Framebuffer::attachTexture(Texture *texture) const
{
    glBindFramebuffer(GL_FRAMEBUFFER, handle);
    glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, texture->getHandle(), 0);
}