Question

I know how to code a shader that converts planar data (YUV420 image) to packed one (RGB image) using fragment shader, yet how do I code the RGB to YUV420.
I do not need the YUV420 image for rendering but to pass it to the h264 compression code.

Answer 1

From http://www.fourcc.org/fccyvrgb.php:

RGB to YUV Conversion

Y = (0.257 * R) + (0.504 * G) + (0.098 * B) + 16

Cr = V = (0.439 * R) – (0.368 * G) – (0.071 * B) + 128

Cb = U = -(0.148 * R) – (0.291 * G) + (0.439 * B) + 128

In a OpenGL shader those constant offsets at the end need to be multiplied by 1/256. Also if you look closely this is a vector-matrix multiplication. You can express this by

mat4 RGBtoYUV(0.257,  0.439, -0.148, 0.0,
              0.504, -0.368, -0.291, 0.0,
              0.098, -0.071,  0.439, 0.0,
              0.0625, 0.500,  0.500, 1.0 );

YUV = RGBtoYUV * RGB;

Some other formulas are given at http://en.wikipedia.org/wiki/YUV