Question

I am using the Windows Kinect SDK to obtain depth and RGB images from the sensor.

Since the depth image and the RGB images do not align, I would like to find a way of converting the coordinates of the RGB image to that of the depth image, since I want to use an image mask on the depth image I have obtained from some processing on the RGB image.

There is already a method for converting depth coordinates to the color space coordinates:

NuiImageGetColorPixelCoordinatesFromDepthPixel

unfortunately, the reverse does not exist. There is only an arcane call in INUICoordinateMapper:

HRESULT MapColorFrameToDepthFrame(
     NUI_IMAGE_RESOLUTION eColorResolution,
     NUI_IMAGE_RESOLUTION eDepthResolution,
     DWORD cDepthPixels,
     NUI_DEPTH_IMAGE_PIXEL *pDepthPixels,
     DWORD cDepthPoints,
     NUI_DEPTH_IMAGE_POINT *pDepthPoints
)

How this method works is not very well documented. Has anyone used it before?

I’m on the verge of performing a manual calibration myself to calculate a transformation matrix, so I would be very happy for a solution.

Answer 1

Thanks to commenter horristic, I got a link to msdn with some useful information (thanks also to T. Chen over at msdn for helping out with the API). Extracted from T. Chen’s post, here’s the code that will perform the mapping from RGB to depth coordinate space:

INuiCoordinateMapper* pMapper;

mNuiSensor->NuiGetCoordinateMapper(&pMapper);

pMapper->MapColorFrameToDepthFrame(
        NUI_IMAGE_TYPE_COLOR,
        NUI_IMAGE_RESOLUTION_640x480,
        NUI_IMAGE_RESOLUTION_640x480,
        640 * 480, 
        (NUI_DEPTH_IMAGE_PIXEL*)LockedRect.pBits,
        640 * 480, 
        depthPoints);

Note: the sensor needs to be initialized and a depth frame locked for this to work.

The transformed coordinates can, e.g., be queried as follows:

/// transform RGB coordinate point to a depth coordinate point 
cv::Point TransformRGBtoDepthCoords(cv::Point rgb_coords, NUI_DEPTH_IMAGE_POINT *    depthPoints)
{
    long index = rgb_coords.y * 640 + rgb_coords.x;
    NUI_DEPTH_IMAGE_POINT depthPointAtIndex = depthPoints[index];
    return cv::Point(depthPointAtIndex.x, depthPointAtIndex.y); 
}