Question

In my native thread I have FFMpeg getting and decoding frames then putting them in a queue.

On Java side I have a GLSurfaceView and from Renderer.onDrawFrame I’m calling into the native code passing a bitmap (that bitmap I create only once, then pass it every time).

In the native code I get the head of the queue, copy data to the java bitmap using AndroidBitmap_xxx functions, then render that Bitmap on Java side as a texture.

I wonder is there a faster way to render video? Shall I do it entirely in the native code, if yes, why it will be faster?

Edit: I now don’t copy RGB frame pixels to the locked bitmap pixels, rather I decode YUV frame directly into the locked bitmap pixels. This makes rendering significantly faster (because no unneeded memcpy anymore) still the question remains.

Answer 1

The most effective technique to change the pixels in the texture is called Render-to-Texture and can be done in OpenGL/OpenGL ES via FBOs. On desktop OpenGL you can use pixel buffer objects (PBOs) to manipulate pixel data directly on GPU (but OpenGL ES does not support this yet).

On unextended OpenGL you can change the pixels in system memory and then update texture with glTexImage2D/glTexSubImage2D – but this is inefficient last resort solution and should be avoided if possible. glTexSubImage2D is usually much faster since it only updates pixel inside the existing texture, while glTexImage2D creates entirely new texture (as a benefit you can change the size and pixel format of the texture). On the other side, glTexSubImage2D allows to update only parts of the texture.

You say that you want it to work with OpenGL ES, so I would propose to do the following steps:

• replace glTexImage2D() with glTexSubImage2D() – if you gain enough performance that’s it, just let it be;
• implement render-to-texture with FBOs and shaders – it will require
  far more work to rewrite your code, but will give even better
  performance.

For FBOs the code can look like this:

// setup FBO
glGenFramebuffers( 1, &FFrameBuffer );
glBindFramebuffer( GL_FRAMEBUFFER, FFrameBuffer );
glFramebufferTexture2D( GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, YourTextureID, 0 );
glBindFramebuffer( GL_FRAMEBUFFER, 0 );

// render to FBO
glBindFramebuffer( GL_FRAMEBUFFER, FFrameBuffer );
glViewport( 0, 0, YourTextureWidth, YourTextureHeight );
your rendering code goes here - it will draw directly into the texture
glBindFramebuffer( GL_FRAMEBUFFER, 0 );

// cleanup
glBindFramebuffer( GL_FRAMEBUFFER, 0 );
glDeleteFramebuffers( 1, &FFrameBuffer );

Keep in mind that not all pixel formats can be rendered to. RGB/RGBA are usually fine.