Question

I’m writing an iOS app that streams video and audio over the network.

I am using AVCaptureSession to grab raw video frames using AVCaptureVideoDataOutput and encode them in software using x264. This works great.

I wanted to do the same for audio, only that I don’t need that much control on the audio side so I wanted to use the built in hardware encoder to produce an AAC stream. This meant using Audio Converter from the Audio Toolbox layer. In order to do so I put in a handler for AVCaptudeAudioDataOutput‘s audio frames:

- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
       fromConnection:(AVCaptureConnection *)connection 
{
    // get the audio samples into a common buffer _pcmBuffer
    CMBlockBufferRef blockBuffer = CMSampleBufferGetDataBuffer(sampleBuffer);
    CMBlockBufferGetDataPointer(blockBuffer, 0, NULL, &_pcmBufferSize, &_pcmBuffer);

    // use AudioConverter to
    UInt32 ouputPacketsCount = 1;
    AudioBufferList bufferList;
    bufferList.mNumberBuffers = 1;
    bufferList.mBuffers[0].mNumberChannels = 1;
    bufferList.mBuffers[0].mDataByteSize = sizeof(_aacBuffer);
    bufferList.mBuffers[0].mData = _aacBuffer;
    OSStatus st = AudioConverterFillComplexBuffer(_converter, converter_callback, (__bridge void *) self, &ouputPacketsCount, &bufferList, NULL);
    if (0 == st) {
        // ... send bufferList.mBuffers[0].mDataByteSize bytes from _aacBuffer...
    }
}

In this case the callback function for the audio converter is pretty simple (assuming packet sizes and counts are setup properly):

- (void) putPcmSamplesInBufferList:(AudioBufferList *)bufferList withCount:(UInt32 *)count
{
    bufferList->mBuffers[0].mData = _pcmBuffer;         
    bufferList->mBuffers[0].mDataByteSize = _pcmBufferSize;
}

And the setup for the audio converter looks like this:

{
    // ...
    AudioStreamBasicDescription pcmASBD = {0};
    pcmASBD.mSampleRate = ((AVAudioSession *) [AVAudioSession sharedInstance]).currentHardwareSampleRate;
    pcmASBD.mFormatID = kAudioFormatLinearPCM;
    pcmASBD.mFormatFlags = kAudioFormatFlagsCanonical;
    pcmASBD.mChannelsPerFrame = 1;
    pcmASBD.mBytesPerFrame = sizeof(AudioSampleType);
    pcmASBD.mFramesPerPacket = 1;
    pcmASBD.mBytesPerPacket = pcmASBD.mBytesPerFrame * pcmASBD.mFramesPerPacket;
    pcmASBD.mBitsPerChannel = 8 * pcmASBD.mBytesPerFrame;

    AudioStreamBasicDescription aacASBD = {0};
    aacASBD.mFormatID = kAudioFormatMPEG4AAC;
    aacASBD.mSampleRate = pcmASBD.mSampleRate;
    aacASBD.mChannelsPerFrame = pcmASBD.mChannelsPerFrame;
    size = sizeof(aacASBD);
    AudioFormatGetProperty(kAudioFormatProperty_FormatInfo, 0, NULL, &size, &aacASBD);

    AudioConverterNew(&pcmASBD, &aacASBD, &_converter);
    // ...
}

This seems pretty straight forward only the IT DOES NOT WORK. Once the AVCaptureSession is running, the audio converter (specifically AudioConverterFillComplexBuffer) returns an ‘hwiu’ (hardware in use) error. Conversion works fine if the session is stopped but then I can’t capture anything…

I was wondering if there was a way to get an AAC stream out of AVCaptureSession. The options I’m considering are:

1. Somehow using AVAssetWriterInput to encode audio samples into AAC and then get the encoded packets somehow (not through AVAssetWriter, which would only write to a file).

2. Reorganizing my app so that it uses AVCaptureSession only on the video side and uses Audio Queues on the audio side. This will make flow control (starting and stopping recording, responding to interruptions) more complicated and I’m afraid that it might cause synching problems between the audio and video. Also, it just doesn’t seem like a good design.

Does anyone know if getting the AAC out of AVCaptureSession is possible? Do I have to use Audio Queues here? Could this get me into synching or control problems?

Answer 1

I ended up asking Apple for advice (it turns out you can do that if you have a paid developer account).

It seems that AVCaptureSession grabs a hold of the AAC hardware encoder but only lets you use it to write directly to file.

You can use the software encoder but you have to ask for it specifically instead of using AudioConverterNew:

AudioClassDescription *description = [self
        getAudioClassDescriptionWithType:kAudioFormatMPEG4AAC
                        fromManufacturer:kAppleSoftwareAudioCodecManufacturer];
if (!description) {
    return false;
}
// see the question as for setting up pcmASBD and arc ASBD
OSStatus st = AudioConverterNewSpecific(&pcmASBD, &aacASBD, 1, description, &_converter);
if (st) {
    NSLog(@"error creating audio converter: %s", OSSTATUS(st));
    return false;
}

with

- (AudioClassDescription *)getAudioClassDescriptionWithType:(UInt32)type
                                           fromManufacturer:(UInt32)manufacturer
{
    static AudioClassDescription desc;

    UInt32 encoderSpecifier = type;
    OSStatus st;

    UInt32 size;
    st = AudioFormatGetPropertyInfo(kAudioFormatProperty_Encoders,
                                    sizeof(encoderSpecifier),
                                    &encoderSpecifier,
                                    &size);
    if (st) {
        NSLog(@"error getting audio format propery info: %s", OSSTATUS(st));
        return nil;
    }

    unsigned int count = size / sizeof(AudioClassDescription);
    AudioClassDescription descriptions[count];
    st = AudioFormatGetProperty(kAudioFormatProperty_Encoders,
                                sizeof(encoderSpecifier),
                                &encoderSpecifier,
                                &size,
                                descriptions);
    if (st) {
        NSLog(@"error getting audio format propery: %s", OSSTATUS(st));
        return nil;
    }

    for (unsigned int i = 0; i < count; i++) {
        if ((type == descriptions[i].mSubType) &&
            (manufacturer == descriptions[i].mManufacturer)) {
            memcpy(&desc, &(descriptions[i]), sizeof(desc));
            return &desc;
        }
    }

    return nil;
}

The software encoder will take up CPU resources, of course, but will get the job done.