Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Low-latency sound in iOS

Low-latency sound in iOS

Cyril Lashkevich

March 13, 2014
Tweet

More Decks by Cyril Lashkevich

Other Decks in Programming

Transcript

  1. Low-latency sound in iOS Cyril Lashkevich iOS Solution Architect Viber

    Media Inc. Cyril Lashkevich Low-latency sound in iOS
  2. VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit

    Cyril Lashkevich Low-latency sound in iOS
  3. VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit

    BUS 1 from mic Get ASBD to inspect audio format being received from H/W; ASBD — AudioStreamBasicDescription Cyril Lashkevich Low-latency sound in iOS
  4. VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit

    BUS 1 from mic BUS 0 to speaker Get ASBD to inspect audio format being sent to H/W; ASBD — AudioStreamBasicDescription Cyril Lashkevich Low-latency sound in iOS
  5. VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit

    BUS 1 from mic BUS 0 to speaker BUS 0 from app Set ASBD to indicate what you’re providing for play-out; ASBD — AudioStreamBasicDescription Cyril Lashkevich Low-latency sound in iOS
  6. VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit

    BUS 1 from mic BUS 0 to speaker BUS 0 from app BUS 1 to app Set ASBD to indicate what format you want your units to receive; ASBD — AudioStreamBasicDescription Cyril Lashkevich Low-latency sound in iOS
  7. VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit

    BUS 1 from mic BUS 0 to speaker BUS 0 from app BUS 1 to app RenderCallback Cyril Lashkevich Low-latency sound in iOS
  8. VoiceProcessingIO Unit I N O U T RemoteIO Audio Unit

    BUS 1 from mic BUS 0 to speaker BUS 0 from app BUS 1 to app RenderCallback InputCallback Cyril Lashkevich Low-latency sound in iOS
  9. Callbacks OSStatus AURenderCallback( void * inRefCon, AudioUnitRenderActionFlags * ioActionFlags, const

    AudioTimeStamp * inTimeStamp, Int32 inBusNumber, UInt32 inNumberFrames, AudioBufferList * ioData); Cyril Lashkevich Low-latency sound in iOS
  10. Callbacks Used for both, input and output; Input callback calls

    AudioUnitRender which provides recorded data from microphone; Render callback fills thr ioData bu↵ers with inNumberFrames of data; inTimeStamp contains timstamp for provided/requested frames. Cyril Lashkevich Low-latency sound in iOS
  11. AudioTimeStamp struct AudioTimeStamp { // The absolute sample frame time

    Float64 mSampleTime; // mach_absolute_time UInt64 mHostTime; .... // Which fields are valid UInt32 mFlags; }; Cyril Lashkevich Low-latency sound in iOS
  12. Hardware bu↵er duration Default route, sample rate 44100 THW 2

    {5.805 ms , 11.61 ms , 23.22 ms } Cyril Lashkevich Low-latency sound in iOS
  13. Hardware bu↵er duration Default route, sample rate 44100 THW 2

    {5.805 ms , 11.61 ms , 23.22 ms } Bluetooth route, sample rate 16000 THW 2 {16.0 ms , 32.0 ms } Cyril Lashkevich Low-latency sound in iOS
  14. Hardware bu↵er duration Default route, sample rate 44100 THW 2

    {5.805 ms , 11.61 ms , 23.22 ms } Bluetooth route, sample rate 16000 THW 2 {16.0 ms , 32.0 ms } Bu↵er duration in sec THW = N 256 mSampleRateHW ; N 2 {1, 2, 3, . . .} Cyril Lashkevich Low-latency sound in iOS
  15. Internal resampling Default settings mSampleRateHW = 44100 bu↵SizeHW = 1024

    mSampleRateAPP = 16000 Cyril Lashkevich Low-latency sound in iOS
  16. Internal resampling Default settings mSampleRateHW = 44100 bu↵SizeHW = 1024

    mSampleRateAPP = 16000 THW = 1024 44100 = 23.22 ms Cyril Lashkevich Low-latency sound in iOS
  17. Internal resampling Default settings mSampleRateHW = 44100 bu↵SizeHW = 1024

    mSampleRateAPP = 16000 THW = 1024 44100 = 23.22 ms bu↵SizeAPP = 1024 · 16000 44100 = 371.52 bytes Cyril Lashkevich Low-latency sound in iOS
  18. Frames loss, route changes Frames loss Wrong thread syncronisation/design; High

    CPU load; Up to 50ms missed frames. Cyril Lashkevich Low-latency sound in iOS
  19. Frames loss, route changes Frames loss Wrong thread syncronisation/design; High

    CPU load; Up to 50ms missed frames. inTimeStamp->mSampleTime gives the correct number of missed samples. Cyril Lashkevich Low-latency sound in iOS
  20. Frames loss, route changes Headset/Speaker/Headphones route switch Up to 600ms

    missed samples; Bu↵er duration is unchanged; Cyril Lashkevich Low-latency sound in iOS
  21. Frames loss, route changes Headset/Speaker/Headphones route switch Up to 600ms

    missed samples; Bu↵er duration is unchanged; inTimeStamp->mSampleTime still gives the correct number of missed samples. Cyril Lashkevich Low-latency sound in iOS
  22. Route changes Bluetooth/USB/30-pin route switch Up to 3000ms missed samples;

    Bu↵er duration can change; Cyril Lashkevich Low-latency sound in iOS
  23. Route changes Bluetooth/USB/30-pin route switch Up to 3000ms missed samples;

    Bu↵er duration can change; inTimeStamp->mSampleTime is useless. inTimeStamp->mHostTime should be used for aproximate missed samples calculation. Cyril Lashkevich Low-latency sound in iOS
  24. Conclusion Process on HW sample rate if possible. Circular bu↵ers

    are your friends. Cyril Lashkevich Low-latency sound in iOS
  25. Conclusion Process on HW sample rate if possible. Circular bu↵ers

    are your friends. Lock-free rocks! Cyril Lashkevich Low-latency sound in iOS
  26. Conclusion Process on HW sample rate if possible. Circular bu↵ers

    are your friends. Lock-free rocks! Say ’NO’ to heavy calculations in callbacks. Cyril Lashkevich Low-latency sound in iOS
  27. Conclusion Process on HW sample rate if possible. Circular bu↵ers

    are your friends. Lock-free rocks! Say ’NO’ to heavy calculations in callbacks. Be careful with routes switch. Cyril Lashkevich Low-latency sound in iOS