Question

I’m trying to do some image processing on iPhone. I’m using http://developer.apple.com/library/ios/#qa/qa2010/qa1702.html to capture the camera frames.

My problem is that when I’m trying to access the captured buffer, the camera FPS drops from 30 to about 20. Does anybody knows how I can fix it?

I use the lowest capture quality I could find (AVCaptureSessionPresetLow = 192×144) in kCVPixelFormatType_32BGRA format. If anybody knows a lower quality I could use, I’m willing to try it.

When I do the same image access on other platforms, like Symbian, it works OK.

Here is my code:

#pragma mark -
#pragma mark AVCaptureSession delegate
- (void)captureOutput:(AVCaptureOutput *)captureOutput 
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer 
    fromConnection:(AVCaptureConnection *)connection 
{ 
 /*We create an autorelease pool because as we are not in the main_queue our code is
  not executed in the main thread. So we have to create an autorelease pool for the thread we are in*/
 NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];

    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
 //Lock the image buffer
    if (CVPixelBufferLockBaseAddress(imageBuffer, 0) == kCVReturnSuccess)
    {  

  // calculate FPS and display it using main thread
  [self performSelectorOnMainThread:@selector(updateFps:) withObject: (id) nil waitUntilDone:NO];


  UInt8 *base = (UInt8 *)CVPixelBufferGetBaseAddress(imageBuffer); //image buffer start address

  size_t width  = CVPixelBufferGetWidth(imageBuffer); 
  size_t height = CVPixelBufferGetHeight(imageBuffer);

  int size = (height*width);
  UInt8* pRGBtmp = m_pRGBimage;

        /*
        Here is the problem; m_pRGBimage is RGB image I want to process.
        In the 'for' loop I convert the image from BGRA to RGB. As a resault, the FPS drops to 20.
        */ 
  for (int i=0;i<size;i++)
  {   
    pRGBtmp[0] = base[2];
    pRGBtmp[1] = base[1];
    pRGBtmp[2] = base[0];
    base = base+4;
    pRGBtmp = pRGBtmp+3;     
  }


  // Display received action
  [self performSelectorOnMainThread:@selector(displayAction:) withObject: (id) nil waitUntilDone:NO];
  //[self displayAction:&eyePlayOutput];
  //saveFrame( imageBuffer );

  //unlock the  image buffer
  CVPixelBufferUnlockBaseAddress(imageBuffer,0);

 }


 [pool drain];
} 

As a follow-on to the answers, I need to process the image in realtime, it is being displayed.

I noticed that when I use AVCaptureSessionPresetHigh, the most simple thing I do, like:

 for (int i=0;i<size;i++)
    x = base[0];

causes the framerate to drop to 4-5 FPS. I guess its because an image in that size is not cached.

Basically I need 96×48 image. is there a simple way to downscale the camera output image, a way that uses hardware acceleration, so I could work with the small one?

Answer 1

Anything that iterates over every pixel in an image will be fairly slow on all but the fastest iOS devices. For example, I benchmarked iterating over every pixel in a 640 x 480 video frame (307,200 pixels) with a simple per-pixel color test and found that this only runs at around 4 FPS on an iPhone 4.

You’re looking at processing 27,648 pixels in your case, which should run fast enough to hit 30 FPS on an iPhone 4, but that’s a much faster processor than what was in the original iPhone and iPhone 3G. The iPhone 3G will probably still struggle with this processing load. You also don’t say how fast the processor was in your Symbian devices.

I’d suggest reworking your processing algorithm to avoid the colorspace conversion. There should be no need to reorder the color components in order to process them.

Additionally, you could selectively process only a few pixels by sampling at certain intervals within the rows and columns of the image.

Finally, if you are targeting the newer iOS devices that have support for OpenGL ES 2.0 (iPhone 3G S and newer), you might want to look at using a GLSL fragment shader to process the video frame entirely on the GPU. I describe the process here, along with sample code for realtime color-based object tracking. The GPU can handle this kind of processing 14 – 28 times faster than the CPU, in my benchmarks.