Question

I want to get every OpenGL frame from an animation with glReadPixels() and convert the data to OpenCV::Mat. I know that glReadPixels() gets the data by rows from the lower one to upper one, from left to right. On the other hand, OpenCV stores the data differently.

Does anybody know any library or any tutorial/example that helps me to convert data from glReadPixels to a OpenCV:Mat in C++?

SUMMARY

 OpenGL frame      ----------------------->        CV::Mat

Data from left to right,                    Data from left to right,
bottom to top.                              top to bottom.

Answer 1

First we create an empty (or unititialized) cv::Mat for our data to be read into directly. This can be done once at startup, but on the other hand cv::Mat::create doesn’t really cost much when the image already has matching size and type. The type depends on your needs, usually it’s something like CV_8UC3 for a 24-bit color image.

cv::Mat img(height, width, CV_8UC3);

or

img.create(height, width, CV_8UC3);

Then you have to account for cv::Mat not neccessarily storing image rows contiguously. There might be a small padding value at the end of each row to make rows 4-byte aligned (or 8?). So you need to mess with the pixel storage modes:

//use fast 4-byte alignment (default anyway) if possible
glPixelStorei(GL_PACK_ALIGNMENT, (img.step & 3) ? 1 : 4);

//set length of one complete row in destination data (doesn't need to equal img.cols)
glPixelStorei(GL_PACK_ROW_LENGTH, img.step/img.elemSize());

Next, the type of the matrix influences the format and type parameters of glReadPixels. If you want color images you have to keep in mind that OpenCV usually stores color values in BGR order, so you need to use GL_BGR(A) (which were added with OpenGL 1.2) instead of GL_RGB(A). For one component images use either GL_LUMINANCE (which sums the individual color components) or GL_RED, GL_GREEN, … (to get an individual component). So for our CV_8UC3 image the final call to read it directly into the cv::Mat would be:

glReadPixels(0, 0, img.cols, img.rows, GL_BGR, GL_UNSIGNED_BYTE, img.data);

Finally, OpenCV stores images from top to bottom. So you may need to either flip them after getting them or render them flipped in OpenGL in the first place (this can be done by adjusting the projection matrix, but keep an eye on triangle orientation in this case). To flip a cv::Mat vertically, you can use cv::flip:

cv::flip(img, flipped, 0);

So to keep in mind OpenCV:

• stores images from top to bottom, left to right
• stores color images in BGR order
• might not store image rows tightly packed