Tweet
Tweet

Slide 1

Slide 1 text

Low-latency sound in iOS Cyril Lashkevich iOS Solution Architect Viber Media Inc. Cyril Lashkevich Low-latency sound in iOS

Slide 2

Slide 2 text

iOS audio APIs AVAudioPlayer/AVAudioRecoder OpenAL AudioQueue AudioUnit Cyril Lashkevich Low-latency sound in iOS

Slide 3

Slide 3 text

Audio Unit Cyril Lashkevich Low-latency sound in iOS

Slide 4

Slide 4 text

VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit Cyril Lashkevich Low-latency sound in iOS

Slide 5

Slide 5 text

VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit BUS 1 from mic Get ASBD to inspect audio format being received from H/W; ASBD — AudioStreamBasicDescription Cyril Lashkevich Low-latency sound in iOS

Slide 6

Slide 6 text

VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit BUS 1 from mic BUS 0 to speaker Get ASBD to inspect audio format being sent to H/W; ASBD — AudioStreamBasicDescription Cyril Lashkevich Low-latency sound in iOS

Slide 7

Slide 7 text

VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit BUS 1 from mic BUS 0 to speaker BUS 0 from app Set ASBD to indicate what you’re providing for play-out; ASBD — AudioStreamBasicDescription Cyril Lashkevich Low-latency sound in iOS

Slide 8

Slide 8 text

VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit BUS 1 from mic BUS 0 to speaker BUS 0 from app BUS 1 to app Set ASBD to indicate what format you want your units to receive; ASBD — AudioStreamBasicDescription Cyril Lashkevich Low-latency sound in iOS

Slide 9

Slide 9 text

VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit BUS 1 from mic BUS 0 to speaker BUS 0 from app BUS 1 to app RenderCallback Cyril Lashkevich Low-latency sound in iOS

Slide 10

Slide 10 text

VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit BUS 1 from mic BUS 0 to speaker BUS 0 from app BUS 1 to app RenderCallback InputCallback Cyril Lashkevich Low-latency sound in iOS

Slide 11

Slide 11 text

AudioStreamBasicDescription Code struct AudioStreamBasicDescription { Float64 mSampleRate; .... UInt32 mChannelsPerFrame; .... }; Cyril Lashkevich Low-latency sound in iOS

Slide 12

Slide 12 text

Callbacks OSStatus AURenderCallback( void * inRefCon, AudioUnitRenderActionFlags * ioActionFlags, const AudioTimeStamp * inTimeStamp, Int32 inBusNumber, UInt32 inNumberFrames, AudioBufferList * ioData); Cyril Lashkevich Low-latency sound in iOS

Slide 13

Slide 13 text

Callbacks Used for both, input and output; Input callback calls AudioUnitRender which provides recorded data from microphone; Render callback fills thr ioData bu↵ers with inNumberFrames of data; inTimeStamp contains timstamp for provided/requested frames. Cyril Lashkevich Low-latency sound in iOS

Slide 14

Slide 14 text

AudioTimeStamp struct AudioTimeStamp { // The absolute sample frame time Float64 mSampleTime; // mach_absolute_time UInt64 mHostTime; .... // Which fields are valid UInt32 mFlags; }; Cyril Lashkevich Low-latency sound in iOS

Slide 15

Slide 15 text

Hardware bu↵er duration kAudioSessionProperty * Set: PreferredHardwareIOBufferDuration Get: CurrentHardwareIOBufferDuration; Minimal latency we have from the HW bu↵erization. Cyril Lashkevich Low-latency sound in iOS

Slide 16

Slide 16 text

Hardware bu↵er duration Default route, sample rate 44100 THW 2 {5.805 ms , 11.61 ms , 23.22 ms } Cyril Lashkevich Low-latency sound in iOS

Slide 17

Slide 17 text

Hardware bu↵er duration Default route, sample rate 44100 THW 2 {5.805 ms , 11.61 ms , 23.22 ms } Bluetooth route, sample rate 16000 THW 2 {16.0 ms , 32.0 ms } Cyril Lashkevich Low-latency sound in iOS

Slide 18

Slide 18 text

Hardware bu↵er duration Default route, sample rate 44100 THW 2 {5.805 ms , 11.61 ms , 23.22 ms } Bluetooth route, sample rate 16000 THW 2 {16.0 ms , 32.0 ms } Bu↵er duration in sec THW = N 256 mSampleRateHW ; N 2 {1, 2, 3, . . .} Cyril Lashkevich Low-latency sound in iOS

Slide 19

Slide 19 text

Internal resampling Default settings mSampleRateHW = 44100 Cyril Lashkevich Low-latency sound in iOS

Slide 20

Slide 20 text

Internal resampling Default settings mSampleRateHW = 44100 bu↵SizeHW = 1024 Cyril Lashkevich Low-latency sound in iOS

Slide 21

Slide 21 text

Internal resampling Default settings mSampleRateHW = 44100 bu↵SizeHW = 1024 mSampleRateAPP = 16000 Cyril Lashkevich Low-latency sound in iOS

Slide 22

Slide 22 text

Internal resampling Default settings mSampleRateHW = 44100 bu↵SizeHW = 1024 mSampleRateAPP = 16000 THW = 1024 44100 = 23.22 ms Cyril Lashkevich Low-latency sound in iOS

Slide 23

Slide 23 text

Internal resampling Default settings mSampleRateHW = 44100 bu↵SizeHW = 1024 mSampleRateAPP = 16000 THW = 1024 44100 = 23.22 ms bu↵SizeAPP = 1024 · 16000 44100 = 371.52 bytes Cyril Lashkevich Low-latency sound in iOS

Slide 24

Slide 24 text

Internal resampling 1.024 2.048 3.072 4.096 Cyril Lashkevich Low-latency sound in iOS

Slide 25

Slide 25 text

Internal resampling 1.024 2.048 3.072 4.096 Cyril Lashkevich Low-latency sound in iOS

Slide 26

Slide 26 text

Internal resampling 1.024 2.048 3.072 4.096 Cyril Lashkevich Low-latency sound in iOS

Slide 27

Slide 27 text

Internal resampling 1.024 2.048 3.072 4.096 Cyril Lashkevich Low-latency sound in iOS 3

Slide 28

Slide 28 text

Internal resampling 1.024 2.048 3.072 4.096 Cyril Lashkevich Low-latency sound in iOS

Slide 29

Slide 29 text

Internal resampling 1.024 2.048 3.072 4.096 Cyril Lashkevich Low-latency sound in iOS

Slide 30

Slide 30 text

Internal resampling 1.024 2.048 3.072 4.096 Cyril Lashkevich Low-latency sound in iOS

Slide 31

Slide 31 text

Internal resampling 1.024 2.048 3.072 4.096 Cyril Lashkevich Low-latency sound in iOS

Slide 32

Slide 32 text

Internal resampling 1.024 2.048 3.072 4.096 Cyril Lashkevich Low-latency sound in iOS

Slide 33

Slide 33 text

Frames loss, route changes Frames loss Wrong thread syncronisation/design; High CPU load; Up to 50ms missed frames. Cyril Lashkevich Low-latency sound in iOS

Slide 34

Slide 34 text

Frames loss, route changes Frames loss Wrong thread syncronisation/design; High CPU load; Up to 50ms missed frames. inTimeStamp->mSampleTime gives the correct number of missed samples. Cyril Lashkevich Low-latency sound in iOS

Slide 35

Slide 35 text

Frames loss, route changes Headset/Speaker/Headphones route switch Up to 600ms missed samples; Bu↵er duration is unchanged; Cyril Lashkevich Low-latency sound in iOS

Slide 36

Slide 36 text

Frames loss, route changes Headset/Speaker/Headphones route switch Up to 600ms missed samples; Bu↵er duration is unchanged; inTimeStamp->mSampleTime still gives the correct number of missed samples. Cyril Lashkevich Low-latency sound in iOS

Slide 37

Slide 37 text

Route changes Bluetooth/USB/30-pin route switch Up to 3000ms missed samples; Bu↵er duration can change; Cyril Lashkevich Low-latency sound in iOS

Slide 38

Slide 38 text

Route changes Bluetooth/USB/30-pin route switch Up to 3000ms missed samples; Bu↵er duration can change; inTimeStamp->mSampleTime is useless. inTimeStamp->mHostTime should be used for aproximate missed samples calculation. Cyril Lashkevich Low-latency sound in iOS

Slide 39

Slide 39 text

Conclusion Process on HW sample rate if possible. Cyril Lashkevich Low-latency sound in iOS

Slide 40

Slide 40 text

Conclusion Process on HW sample rate if possible. Circular bu↵ers are your friends. Cyril Lashkevich Low-latency sound in iOS

Slide 41

Slide 41 text

Conclusion Process on HW sample rate if possible. Circular bu↵ers are your friends. Lock-free rocks! Cyril Lashkevich Low-latency sound in iOS

Slide 42

Slide 42 text

Conclusion Process on HW sample rate if possible. Circular bu↵ers are your friends. Lock-free rocks! Say ’NO’ to heavy calculations in callbacks. Cyril Lashkevich Low-latency sound in iOS

Slide 43

Slide 43 text

Conclusion Process on HW sample rate if possible. Circular bu↵ers are your friends. Lock-free rocks! Say ’NO’ to heavy calculations in callbacks. Be careful with routes switch. Cyril Lashkevich Low-latency sound in iOS