Question

I have an OpenGL ES Shader fragment shader that is using the mix function to place an overlay over a videoframe. This is my shader:

#ifdef GL_ES
precision mediump float;
#endif

varying vec2 textureCoordinate;

uniform sampler2D videoFrameY; 
uniform sampler2D videoFrameUV;
uniform sampler2D overlay;

const mat3 yuv2rgb = mat3(
                          1,        1,          1,
                          0,        -.21482,    2.12798,
                          1.28033,  -.38059,    0
                          );

void main() {
    
    vec3 yuv;
    vec4 ovr;
    
    yuv.x = texture2D(videoFrameY, textureCoordinate).r;
    yuv.yz = texture2D(videoFrameUV, textureCoordinate).rg - vec2(0.5, 0.5);
    ovr = texture2D(overlay, textureCoordinate);
    
    vec3 rgb = yuv2rgb * yuv;
    
    gl_FragColor = mix(ovr, vec4(rgb, 1.0), ovr.a);
} 

Without the overlay texture, feeding gl_FragColor this:

gl_FragColor = vec4(rgb, 1.0);

works just fine and my video is displayed. Now I’m creating my overlay texture from a CATextLayer like this:

- (void)generateOverlay {
    CATextLayer *textLayer = [CATextLayer layer];
    [textLayer setString:@"Sample test string"];
    [textLayer setFont:(__bridge CFStringRef)@"Helvetica"];
    [textLayer setFontSize:(_videoHeight / 6)];
    [textLayer setAlignmentMode:kCAAlignmentLeft];
    [textLayer setBounds:CGRectMake(0, 0, _videoWidth, _videoHeight)];
    
    CGSize layerSize = textLayer.bounds.size;
    CGColorSpaceRef colorspace = CGColorSpaceCreateDeviceRGB();
    void *imageData = malloc(layerSize.height * layerSize.width * 4);
    CGContextRef context = CGBitmapContextCreate(imageData, layerSize.width, layerSize.height, 8, 4 * layerSize.width, colorspace, kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big);
    CGColorSpaceRelease(colorspace);
    CGContextClearRect(context, CGRectMake(0, 0, layerSize.width, layerSize.height));
    CGContextTranslateCTM(context, 0, layerSize.height - layerSize.height);
    [textLayer renderInContext:context];
    
    glActiveTexture(GL_TEXTURE2);
    glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, layerSize.width, layerSize.height, 0, GL_RGBA, GL_UNSIGNED_BYTE, imageData);
    
    CGContextRelease(context);
    free(imageData);

}

The problem is the results are inverted. So therefore they look like this:

_{(source: c.tro.pe)}

The fact is they are are inverted in two ways. First, instead of the blue being alpha’ed out and the video showing through, the text is the alpha and that is where the video is showing through. Second, the text is mirrored. The mirroring could be the results of the vertex values being used however the video is correct using the same coords. I’m sure this is a quick rearrange but I’m not sure what to tweak. Thanks!

Answer 1

For the mix, the mix(x, y, a) function interpolates between x and y based on a. a of 0 gives you all x, and a of 1.0 gives you all y. You’re keying off of the alpha of your text layer, so for the overlay you want, you need to reverse your ordering as follows:

gl_FragColor = mix(vec4(rgb, 1.0), ovr, ovr.a);

In regards to the rotation, remember that the iOS rear cameras are mounted landscape left and the front cameras landscape right, so for a portrait orientation you need to rotate the incoming video frames. You appear to be performing that rotation in either your vertices or texture coordinates. You’re going to need a second set of texture coordinates that aren’t rotated to use for sampling your overlay image, or you’ll need to draw your label at a matching landscape left rotation when generating its texture.