Question

Following scenario. Real time camera feed processing with openCV on ios using dispatch asnyc. Here is capture sampleBufferMethod that is converting the buffer to an IplImage and then using it.

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection    
{

    __block IplImage *image = 0;
    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    CVPixelBufferLockBaseAddress(imageBuffer, 0);

    // get information of the image in the buffer
    uint8_t *bufferBaseAddress = (uint8_t *)CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0);
    size_t bufferWidth = CVPixelBufferGetWidth(imageBuffer);
    size_t bufferHeight = CVPixelBufferGetHeight(imageBuffer);

    // create IplImage
    if (bufferBaseAddress) 
    {
        image = cvCreateImage(cvSize(bufferWidth, bufferHeight), IPL_DEPTH_8U, 4);
        image->imageData = (char*)bufferBaseAddress;
    }    
    // release memory
    CVPixelBufferUnlockBaseAddress(imageBuffer, 0);


    dispatch_async(dispatch_get_main_queue(), ^{
        IplImage *out=cvCreateImage(cvSize(568, 320), IPL_DEPTH_8U, 4);
        cvResize(image, out, 0);
        ...
 });
}

Pretty straight forward, except that this here:

        cvResize(image, out, 0);

gives me a EXC_BAD_ACCESS. I got a workaround which I found playing with it forever:

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection 
{

    IplImage *_image = 0;
    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    CVPixelBufferLockBaseAddress(imageBuffer, 0);

    // get information of the image in the buffer
    uint8_t *bufferBaseAddress = (uint8_t *)CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0);
    size_t bufferWidth = CVPixelBufferGetWidth(imageBuffer);
    size_t bufferHeight = CVPixelBufferGetHeight(imageBuffer);

    // create IplImage
    if (bufferBaseAddress) 
    {
        _image = cvCreateImage(cvSize(bufferWidth, bufferHeight), IPL_DEPTH_8U, 4);
        _image->imageData = (char*)bufferBaseAddress;
    }    
    // release memory
    CVPixelBufferUnlockBaseAddress(imageBuffer, 0);

    __block IplImage *image=cvCloneImage(_image);

    dispatch_async(dispatch_get_main_queue(), ^{
        IplImage *out=cvCreateImage(cvSize(568, 320), IPL_DEPTH_8U, 4);
        cvResize(image, out, 0);
        ...
 });
}

Key line:

    __block  IplImage *image=cvCloneImage(_image);

So what I don’t understand is why cvCloneImage makes the difference? What am I missing? I would like to get rid of that statement since the faster the better.

Answer 1

Without your workaround, imageBuffer might not be valid by the time the block executes. You are getting it externally from the framework, with no promises AFAIK regarding its continued lifetime once your handler finishes. Therefore you should copy it. Therefore, your clone makes the code work.

Another issue is in accessing memory which is allocated in the method’s stack frame.

You should remove the __block declaration from image – Otherwise the block is passed a pointer to the image struct pointer, instead of just a copy. Since the struct pointer is allocated on the stack, by the time your block runs, the memory where it was before is no longer valid.