Low-latency sound in iOS

Transcript

1. Low-latency sound in iOS Cyril Lashkevich iOS Solution Architect Viber Media Inc. Cyril Lashkevich Low-latency sound in iOS
2. iOS audio APIs AVAudioPlayer/AVAudioRecoder OpenAL AudioQueue AudioUnit Cyril Lashkevich Low-latency sound in iOS
3. Audio Unit Cyril Lashkevich Low-latency sound in iOS
4. VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit Cyril Lashkevich Low-latency sound in iOS
5. VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit BUS 1 from mic Get ASBD to inspect audio format being received from H/W; ASBD — AudioStreamBasicDescription Cyril Lashkevich Low-latency sound in iOS
6. VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit BUS 1 from mic BUS 0 to speaker Get ASBD to inspect audio format being sent to H/W; ASBD — AudioStreamBasicDescription Cyril Lashkevich Low-latency sound in iOS
7. VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit BUS 1 from mic BUS 0 to speaker BUS 0 from app Set ASBD to indicate what you’re providing for play-out; ASBD — AudioStreamBasicDescription Cyril Lashkevich Low-latency sound in iOS
8. VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit BUS 1 from mic BUS 0 to speaker BUS 0 from app BUS 1 to app Set ASBD to indicate what format you want your units to receive; ASBD — AudioStreamBasicDescription Cyril Lashkevich Low-latency sound in iOS
9. VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit BUS 1 from mic BUS 0 to speaker BUS 0 from app BUS 1 to app RenderCallback Cyril Lashkevich Low-latency sound in iOS
10. VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit BUS 1 from mic BUS 0 to speaker BUS 0 from app BUS 1 to app RenderCallback InputCallback Cyril Lashkevich Low-latency sound in iOS
11. AudioStreamBasicDescription Code struct AudioStreamBasicDescription { Float64 mSampleRate; .... UInt32 mChannelsPerFrame; .... }; Cyril Lashkevich Low-latency sound in iOS
12. Callbacks OSStatus AURenderCallback( void * inRefCon, AudioUnitRenderActionFlags * ioActionFlags, const AudioTimeStamp * inTimeStamp, Int32 inBusNumber, UInt32 inNumberFrames, AudioBufferList * ioData); Cyril Lashkevich Low-latency sound in iOS
13. Callbacks Used for both, input and output; Input callback calls AudioUnitRender which provides recorded data from microphone; Render callback fills thr ioData bu↵ers with inNumberFrames of data; inTimeStamp contains timstamp for provided/requested frames. Cyril Lashkevich Low-latency sound in iOS
14. AudioTimeStamp struct AudioTimeStamp { // The absolute sample frame time Float64 mSampleTime; // mach_absolute_time UInt64 mHostTime; .... // Which fields are valid UInt32 mFlags; }; Cyril Lashkevich Low-latency sound in iOS
15. Hardware bu↵er duration kAudioSessionProperty * Set: PreferredHardwareIOBufferDuration Get: CurrentHardwareIOBufferDuration; Minimal latency we have from the HW bu↵erization. Cyril Lashkevich Low-latency sound in iOS
16. Hardware bu↵er duration Default route, sample rate 44100 THW 2 {5.805 ms , 11.61 ms , 23.22 ms } Cyril Lashkevich Low-latency sound in iOS
17. Hardware bu↵er duration Default route, sample rate 44100 THW 2 {5.805 ms , 11.61 ms , 23.22 ms } Bluetooth route, sample rate 16000 THW 2 {16.0 ms , 32.0 ms } Cyril Lashkevich Low-latency sound in iOS
18. Hardware bu↵er duration Default route, sample rate 44100 THW 2 {5.805 ms , 11.61 ms , 23.22 ms } Bluetooth route, sample rate 16000 THW 2 {16.0 ms , 32.0 ms } Bu↵er duration in sec THW = N 256 mSampleRateHW ; N 2 {1, 2, 3, . . .} Cyril Lashkevich Low-latency sound in iOS
19. Internal resampling Default settings mSampleRateHW = 44100 Cyril Lashkevich Low-latency sound in iOS
20. Internal resampling Default settings mSampleRateHW = 44100 bu↵SizeHW = 1024 Cyril Lashkevich Low-latency sound in iOS
21. Internal resampling Default settings mSampleRateHW = 44100 bu↵SizeHW = 1024 mSampleRateAPP = 16000 Cyril Lashkevich Low-latency sound in iOS
22. Internal resampling Default settings mSampleRateHW = 44100 bu↵SizeHW = 1024 mSampleRateAPP = 16000 THW = 1024 44100 = 23.22 ms Cyril Lashkevich Low-latency sound in iOS
23. Internal resampling Default settings mSampleRateHW = 44100 bu↵SizeHW = 1024 mSampleRateAPP = 16000 THW = 1024 44100 = 23.22 ms bu↵SizeAPP = 1024 · 16000 44100 = 371.52 bytes Cyril Lashkevich Low-latency sound in iOS
24. Internal resampling 1.024 2.048 3.072 4.096 Cyril Lashkevich Low-latency sound in iOS
25. Internal resampling 1.024 2.048 3.072 4.096 Cyril Lashkevich Low-latency sound in iOS
26. Internal resampling 1.024 2.048 3.072 4.096 Cyril Lashkevich Low-latency sound in iOS
27. Internal resampling 1.024 2.048 3.072 4.096 Cyril Lashkevich Low-latency sound in iOS 3
28. Internal resampling 1.024 2.048 3.072 4.096 Cyril Lashkevich Low-latency sound in iOS
29. Internal resampling 1.024 2.048 3.072 4.096 Cyril Lashkevich Low-latency sound in iOS
30. Internal resampling 1.024 2.048 3.072 4.096 Cyril Lashkevich Low-latency sound in iOS
31. Internal resampling 1.024 2.048 3.072 4.096 Cyril Lashkevich Low-latency sound in iOS
32. Internal resampling 1.024 2.048 3.072 4.096 Cyril Lashkevich Low-latency sound in iOS
33. Frames loss, route changes Frames loss Wrong thread syncronisation/design; High CPU load; Up to 50ms missed frames. Cyril Lashkevich Low-latency sound in iOS
34. Frames loss, route changes Frames loss Wrong thread syncronisation/design; High CPU load; Up to 50ms missed frames. inTimeStamp->mSampleTime gives the correct number of missed samples. Cyril Lashkevich Low-latency sound in iOS
35. Frames loss, route changes Headset/Speaker/Headphones route switch Up to 600ms missed samples; Bu↵er duration is unchanged; Cyril Lashkevich Low-latency sound in iOS
36. Frames loss, route changes Headset/Speaker/Headphones route switch Up to 600ms missed samples; Bu↵er duration is unchanged; inTimeStamp->mSampleTime still gives the correct number of missed samples. Cyril Lashkevich Low-latency sound in iOS
37. Route changes Bluetooth/USB/30-pin route switch Up to 3000ms missed samples; Bu↵er duration can change; Cyril Lashkevich Low-latency sound in iOS
38. Route changes Bluetooth/USB/30-pin route switch Up to 3000ms missed samples; Bu↵er duration can change; inTimeStamp->mSampleTime is useless. inTimeStamp->mHostTime should be used for aproximate missed samples calculation. Cyril Lashkevich Low-latency sound in iOS
39. Conclusion Process on HW sample rate if possible. Cyril Lashkevich Low-latency sound in iOS
40. Conclusion Process on HW sample rate if possible. Circular bu↵ers are your friends. Cyril Lashkevich Low-latency sound in iOS
41. Conclusion Process on HW sample rate if possible. Circular bu↵ers are your friends. Lock-free rocks! Cyril Lashkevich Low-latency sound in iOS
42. Conclusion Process on HW sample rate if possible. Circular bu↵ers are your friends. Lock-free rocks! Say ’NO’ to heavy calculations in callbacks. Cyril Lashkevich Low-latency sound in iOS
43. Conclusion Process on HW sample rate if possible. Circular bu↵ers are your friends. Lock-free rocks! Say ’NO’ to heavy calculations in callbacks. Be careful with routes switch. Cyril Lashkevich Low-latency sound in iOS