Question

I am developing an iPad application that uses the AUSampler AudioUnit, in conjunction with other AudioUnits, to play audio. The AUSampler loads a preset from a SoundFont2 file.

When too many notes are played at once or notes are played too quickly, the audio starts dropping out briefly before dropping out completely. The sampler is then unusable until the preset is reloaded from the SoundFont. This only happens with some instruments (specifically presets 88, 90, and 94, which in General MIDI are Pad 1 (new age), Pad 3 (polysynth), and Pad 7 (halo), respectively)

By turning on the All Exceptions breakpoint in XCode, I can see that the issue occurs when AudioToolbox attempts to create a new VoiceZone for the note being played. However, this exception is caught somewhere lower in the call stack below my code, and AudioUnitRender doesn’t return any error.

My best guess is that the “pad” instruments use longer samples, and so playing many notes quickly/in conjunction uses more memory than available for AudioToolbox.

Has anyone encountered this issue before, and if so, is there a way to avoid the audio drop-outs, or to at least detect them so as to gracefully handle them?

Some details:

• The SoundFont is ~150 MB and contains many different presets/instruments that conform to General MIDI
• the AUSampler is at the head of a chain of effect units, which connect into a AUMultiChannelMixer (me.masterMixerUnit). The audioOutputCallback is set as the input callback of the output element of the RemoteIO unit

Backtrace (on the `AURemoteIO::IOThread):

* thread #14: tid = 0x3203, 0x368c8498 libc++abi.dylib`__cxa_throw, stop reason = breakpoint 1.2
frame #0: 0x368c8498 libc++abi.dylib`__cxa_throw
frame #1: 0x35d46a60 AudioToolbox`VoiceZone::operator new(unsigned long) + 340
frame #2: 0x35d46b6a AudioToolbox`VoiceZone::NewVoiceZone(SamplerNote*, ZoneState*, float, float, unsigned long) + 86
frame #3: 0x35d3f846 AudioToolbox`SamplerNote::Configure(InstrumentState*) + 1106
frame #4: 0x35d3ff2c AudioToolbox`non-virtual thunk to SamplerNote::Attack(MusicDeviceNoteParams const&) + 24
frame #5: 0x35d551ea AudioToolbox`SynthNote::AttackNote(SynthPartElement*, SynthGroupElement*, unsigned long, unsigned long long, unsigned long, MusicDeviceNoteParams const&) + 58
frame #6: 0x35d5485a AudioToolbox`SynthGroupElement::NoteOn(SynthNote*, SynthPartElement*, unsigned long, unsigned long, MusicDeviceNoteParams const&) + 70
frame #7: 0x35d3d220 AudioToolbox`SamplerElement::StartNote(SynthPartElement*, unsigned long, unsigned long, MusicDeviceNoteParams const&) + 344
frame #8: 0x35d3c4fa AudioToolbox`Sampler::RealTimeStartNote(SynthGroupElement*, unsigned long, unsigned long, MusicDeviceNoteParams const&) + 94
frame #9: 0x35d530a0 AudioToolbox`AUInstrumentBase::PerformEvents(AudioTimeStamp const&) + 164
frame #10: 0x35d5313c AudioToolbox`AUInstrumentBase::Render(unsigned long&, AudioTimeStamp const&, unsigned long) + 20
frame #11: 0x35d3c564 AudioToolbox`Sampler::Render(unsigned long&, AudioTimeStamp const&, unsigned long) + 68
frame #12: 0x35c2435a AudioToolbox`AUBase::DoRenderBus(unsigned long&, AudioTimeStamp const&, unsigned long, AUOutputElement*, unsigned long, AudioBufferList&) + 210
frame #13: 0x35c24184 AudioToolbox`AUBase::DoRender(unsigned long&, AudioTimeStamp const&, unsigned long, unsigned long, AudioBufferList&) + 496
frame #14: 0x35c23f8a AudioToolbox`AUMethodRender(void*, unsigned long*, AudioTimeStamp const*, unsigned long, unsigned long, AudioBufferList*) + 50

... render methods of other AudioUnits in chain ...

frame #32: 0x00081aaa SoundBrush 2`audioOutputCallback + 194 at SoundEngine.mm:1375

The exception is being thrown during the call to AudioUnitRender in my callback below:

static OSStatus audioOutputCallback(void                       *inRefCon,
                                    AudioUnitRenderActionFlags *ioActionFlags,
                                    const AudioTimeStamp       *inTimeStamp,
                                    UInt32                      inBusNumber,
                                    UInt32                      inNumberFrames,
                                    AudioBufferList            *ioData)
{
    //get a reference to the SoundEngine
    SoundEngine *me = (__bridge SoundEngine *)inRefCon;

    // render the sound
    OSStatus err = AudioUnitRender(me.masterMixerUnit,
                                   ioActionFlags,
                                   inTimeStamp,
                                   kAudioUnitBus_Output,
                                   inNumberFrames,
                                   ioData);
    if (err != noErr) NSLog(@"AudioUnitRender failed [%ld]", err);

    // ... do some processing ...

    return err;
}

Answer 1

If the sound quality is good under normal conditions, I would guess that the dropouts are not caused by a programming error in your AU chain, but rather from the OS not being able to process the block fast enough.

The easiest way to detect audio dropouts is to get the system time in microseconds before you start processing, and then measure it again after you have completed. By looking at the sample rate and buffer size, you should be able to determine if you were able to deliver the samples in the given amount of time.

If you find that you are indeed having dropouts caused by performance, the first thing you should do is to increase the buffer size. Following that, you should try to optimize your code in a few basic ways. Run the algorithm through Instruments (the profiler, that is) and look for any obvious bottlenecks. Avoid object allocation inside of the render callback; this will cause extra strain on the system.

Also I would suggest running the processing code on a desktop computer just to make sure that the issue is indeed caused by the iPad’s performance.