Question

I’m using AVCaptureSession to preview video in an augmented reality type app on iPhone. Since I’m also drawing OpenGL graphics on top of the video preview, the app is quite energy consuming. I want to minimize cpu usage to save battery.

When I check the app with Instruments/Energy usage, I see that a considerable portion (~20%) of the CPU is “wasted” on audio processing. If I remove my capture session, audio processing takes no CPU, as expected.

I don’t understand why the capture session is doing audio processing since I haven’t added any audio device input into it. Here’s how I set up the session:

if(!captureSession) {
  captureSession = [[AVCaptureSession alloc] init];
  AVCaptureDevice *videoDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
  if (videoDevice) {
    NSError *error;
    AVCaptureDeviceInput *videoIn = [AVCaptureDeviceInput deviceInputWithDevice:videoDevice error:&error];
    if (!error) {
      if ([captureSession canAddInput:videoIn]) {
        [captureSession addInput:videoIn];
      }
    }
  }
}

if(!previewLayer) {
  previewLayer = [[AVCaptureVideoPreviewLayer alloc] initWithSession:captureSession];
  [previewLayer setVideoGravity:AVLayerVideoGravityResizeAspectFill];
}

CGRect layerRect = [[viewBg layer] bounds];
[previewLayer setBounds:layerRect];
[previewLayer setPosition:CGPointMake(CGRectGetMidX(layerRect), CGRectGetMidY(layerRect))];
[[viewBg layer] addSublayer:previewLayer];

[captureSession startRunning];                                              

Is there a way to disable audio (input) altogether or how could I get rid of the audio processing CPU usage while previewing video input?

Answer 1

As an even larger performance optimization, may I suggest not using non-opaque OpenGL ES content overlaid on an AVCaptureVideoPreviewLayer? Instead, you’ll get much better rendering performance by grabbing your camera feed, uploading that as a texture to render behind your augmented reality content, then rendering your content in front of a screen-sized textured quad containing your camera texture.

From personal experience, rendering non-opaque OpenGL ES content causes a serious slowdown due to the compositing that needs to be performed in that case. Taking in your camera frames and displaying them as a background within your OpenGL ES scene will let you set your OpenGL ES hosting view to be opaque, which is far more efficient.

I have some sample code for this as part of an object tracking example, but a more efficient version of camera capture and uploading can be found within the GPUImageVideoCamera class in my open source GPUImage framework. Also, in my profiling of the code for that framework, I’ve not seen audio recording occurring without an audio input configured as part of the session, so you could examine what I do there.