Question

I’m trying to get the Kinect depth camera pixels to overlay onto the RGB camera. I am using the C++ Kinect 1.0 SDK with an Xbox Kinect, OpenCV and trying to use the new “NuiImageGetColorPixelCoordinateFrameFromDepthPixelFrameAtResolution” method.

I have watched the image render itself in slow motion and looks as if pixels are being drawn multiple times in the one frame. It first draws itself from the top and left borders, then it gets to a point (you can see a 45 degree angle in there) where it starts drawing weird.

I have been trying to base my code off of the C# code written by Adam Smith at the MSDN forums but no dice. I have stripped out the overlay stuff and just want to draw the depth normalized depth pixels where it “should” be in the RGB image.

The image on the left is what I’m getting when trying to fit the depth image to RGB space, and the image on the right is the “raw” depth image as I like to see it. I was hoping this my method would create a similar image to the one on the right with slight distortions.

This is the code and object definitions that I have at the moment:

// From initialization
INuiSensor *m_pNuiInstance;
NUI_IMAGE_RESOLUTION m_nuiResolution = NUI_IMAGE_RESOLUTION_640x480;
HANDLE m_pDepthStreamHandle;
IplImage *m_pIplDepthFrame;
IplImage *m_pIplFittedDepthFrame;


m_pIplDepthFrame = cvCreateImage(cvSize(640, 480), 8, 1);
m_pIplFittedDepthFrame = cvCreateImage(cvSize(640, 480), 8, 1);

// Method
IplImage *Kinect::GetRGBFittedDepthFrame() {
    static long *pMappedBits = NULL;

    if (!pMappedBits) {
         pMappedBits = new long[640*480*2];
    }

    NUI_IMAGE_FRAME pNuiFrame;
    NUI_LOCKED_RECT lockedRect;
    HRESULT hr = m_pNuiInstance->NuiImageStreamGetNextFrame(m_pDepthStreamHandle, 0, &pNuiFrame);

    if (FAILED(hr)) {
        // return the older frame
        return m_pIplFittedDepthFrame;
    }

    bool hasPlayerData = HasSkeletalEngine(m_pNuiInstance);

    INuiFrameTexture *pTexture = pNuiFrame.pFrameTexture;
    pTexture->LockRect(0, &lockedRect, NULL, 0);

    if (lockedRect.Pitch != 0) {
        cvZero(m_pIplFittedDepthFrame);

        hr = m_pNuiInstance->NuiImageGetColorPixelCoordinateFrameFromDepthPixelFrameAtResolution(
            m_nuiResolution,
            NUI_IMAGE_RESOLUTION_640x480,
            640 * 480, /* size is previous */ (unsigned short*) lockedRect.pBits,
            (640 * 480) * 2,  /* size is previous */ pMappedBits);

        if (FAILED(hr)) {
            return m_pIplFittedDepthFrame;
        }

    for (int i = 0; i < lockedRect.size; i++) {
            unsigned char* pBuf = (unsigned char*) lockedRect.pBits + i;
            unsigned short* pBufS = (unsigned short*) pBuf;
            unsigned short depth = hasPlayerData ? ((*pBufS) & 0xfff8) >> 3 :  ((*pBufS) & 0xffff);
            unsigned char intensity = depth > 0 ? 255 - (unsigned char) (256 * depth / 0x0fff) : 0;

            long
                x = pMappedBits[i], // tried with *(pMappedBits + (i * 2)),
                y = pMappedBits[i + 1]; // tried with *(pMappedBits + (i * 2) + 1);

            if (x >= 0 && x < m_pIplFittedDepthFrame->width && y >= 0 && y < m_pIplFittedDepthFrame->height) {
                m_pIplFittedDepthFrame->imageData[x + y * m_pIplFittedDepthFrame->widthStep] = intensity;
            }

        }
    }

    pTexture->UnlockRect(0);
    m_pNuiInstance->NuiImageStreamReleaseFrame(m_pDepthStreamHandle, &pNuiFrame);

    return(m_pIplFittedDepthFrame);
}

Thanks

Answer 1

I have found that the problem was that the loop,

for (int i = 0; i < lockedRect.size; i++) {
    // code
}

was iterating on a per-byte basis, not on a per-short (2 bytes) basis. Since lockedRect.size returns the number of bytes the fix was simply changing the increment to i += 2, even better would be changing it to sizeof(short), like so,

for (int i = 0; i < lockedRect.size; i += sizeof(short)) {
    // code
}