A Brief History of Real Time Audio (or Wasn’t That API Deprecated?)

Real Time Audio has been the desire of those building games or audio tools with mixed success throughout the history of Android. This talk will cover the ups and downs and various API changes for those interested in high performance audio, starting with JetPlayer (yes it still exists) going through assistive projects like Oboe and the newest stuff in Android Q.
Presented at Droidcon NYC 2019

Transcript

1. A Brief History of Real Time Audio on Android (or wait wasn’t that API deprecated?) Morrison Chang @codeledger https://speakerdeck.com/codeledger Droidcon NYC 2019
2. What is Real-Time Audio? • Not Regular Audio / Media Playback • Low latency Audio from Input to Output • Music Instrument Digital Interface (MIDI)
3. What Android was originally competing against CC BY-SA 2.0, https://commons.wikimedia.org/w/index.php?curid=1398765
4. “Normal Audio” • SoundPool (API 1) • MediaPlayer (API 1) • AudioRecord (API 3) • AudioTrack (API 3) [static or streaming PCM]
5. Sonivox Emebedded Audio System • Licensed to Open Handset Alliance in Nov 2007 • Branded JetPlayer on Android 1.5 API 3 Cupcake (Apr 2009) • MIDI synthesizer engine with authoring tooling • Intended for games & other dynamic audio based on prebuilt samples
6. Sonivox EAS • Tooling used Python 2.5.4 and WXWidgets for UI • Audition (hear content) in Windows & Mac, could only build with Linux • Open Sourced, engine & tools in Android Open Source Project (AOSP) • Tooling not maintained beyond original version
7. Sonivox EAS • JetPlayer API still in Android • Documentation is still available: https:// developer.android.com/guide/topics/media/jet/ jetcreator_manual • Used as a base for other open source libraries like: https://github.com/billthefarmer/mididriver (this version has C/C++ API)
8. OpenSL ES • Khronos API (same organization as OpenGL ES) • Introduced in API 9 Gingerbread (Dec 2010) • NDK access only • PWM capability added in API 14 ICS (Dec 2011)
9. • Primarily Audio Playback • Some Audio recording • Limited Mixing and Effects • Exposes audio features similar to those in the MediaPlayer and MediaRecorder APIs • FEATURE_AUDIO_LOW_LATENCY
10. OpenMAX AL • Added in API 14 Ice Cream Sandwich • Standardized NDK access to multimedia APIs
11. Desires of Audio Developers • Autotune Apps / Entertainment Apps • Digital Audio Workstation • DJ Apps • Synthesizers • Emulators • VOIP
12. Desires of Audio Developers • Low latency audio stack • https://issuetracker.google.com/issues/36908622 [also known as issue 3434] • Opened in 2009 • Closed in 2017/2018
13. Reality Morris Animal Refuge (@morrisanimal)
14. • Desired latency < 20 ms for full round trip! • Android audio had latencies of 100 ms with a round trip latency of 200 ms! • Android Compatibility Definition Document on Audio Latency since API 9 - lots of SHOULDs • Mobile Audio Developer community went to Apple (GarageBand for iOS released in 2011)
15. Issues in Android’s Audio Stack • Variable Buffer Size • Variable Sample Rate • Priority Inversion • Callback timing for Audio thread (OpenSL ES) • Blocking and non-blocking behavior
16. Issues with Android’s Audio Stack Google IO 2013 High Performance Audio
17. Necessary Changes to Android for Low Latency Audio • Rearchitecting AudioFlinger and AudioTrack/ AudioRecord starting in Android 4.1 (July 2012) through Android 5.0 (Nov 2014) • Primarily for scheduling & priority inversion • Adding in blocking & non-blocking algorithms • See: https://source.android.com/devices/audio/ avoiding_pi
18. Adding to AOSP documentation of testing hardware/software • FEATURE_AUDIO_PRO added in API 23 • Android Open Source Project Documentation on the Audio stack - https://source.android.com/devices/ audio • Audio Loopback dongle - https:// source.android.com/devices/audio/latency/loopback • WALT Latency Timer - https://github.com/google/walt
19. Light testing circuit Using differences between screen flashes and audio for delay measurement. (Suggested tool) https://source.android.com/devices/audio/latency/testing_circuit
20. Loop back circuit https://source.android.com/devices/audio/latency/loopback
21. AAudio • Available in Android 8.0 (Aug 2017) • Additional latency reductions with drivers in Android 8.1 • Android’s low latency audio path, use instead of OpenSL ES • Managing Audio Streams and Buffers • Latency tuning
22. Project Oboe • Open source wrapper for OpenSL ES and AAudio in C/C++ • JellyBean API 16 and up • Uses AAudio or OpenSL ES fallback • Doesn’t do everything, does handle workarounds
23. New Stuff in Q (Android 10) • Sharing Audio Input (app & assistant or accessibility app) • Capture Audio (similar to screen capture but for Audio - should not affect latency)
24. New Stuff in Q • AMidi - native interface for MIDI APIs in MidiManager (API 21 Lollipop)
25. Takeaways • Low-latency Audio is available on Android • Need to have FEATURE_AUDIO_PRO • Limited to flagship devices (Samsung at one point had their own Audio SDK)
26. Challenges • USB Audio (loopback circuit?) • Bluetooth Audio • Android Compatibility Definition Document on Audio Latency is mostly [Strongly Recommend] with [Must Not] identify for devices which fall short.
27. Q & A
28. Video Resources • High Performance Audio on Android (I/O 2013): https://youtu.be/d3kfEeMZ65c • Building great multi-media experiences in Android (I/O 2014): https://youtu.be/92fgcUNCHic • Audio latency: buffer sizes (100 Days of Google Dev 2015): https://youtu.be/PnDK17zP9BI
29. • Android high-performance audio (I/O 2016): https://youtu.be/F2ZDp-eNrh4 • Best Practices for Android Audio (I/O 2017): https://youtu.be/C0BPXZIvG-Q • Low Latency Audio - Because Your Ears Are Worth It (Android Dev Summit 2018): https:// youtu.be/8vOf_fDtur4 • Sonic Boom! Audio Programming on Android and Chrome (I/O 2019): https://youtu.be/EeQr0zfvF04
30. Misc Resources • https://developer.android.com/ndk/guides/audio • Android Open Source Project - Audio section: https://source.android.com/devices/audio • Sonivox EAS source code: https:// android.googlesource.com/platform/external/ sonivox/
31. Misc Resources • Sound Amplifier and new Dynamic Processing Effect (I/O 2018): https://youtu.be/_hPlNoF1Tyc • https://github.com/googlesamples/android-audio- high-performance • Google IO 2013 presentation code - https:// github.com/gkasten/high-performance-audio • ADC 2017 - Build a synth in Android - https:// youtu.be/FjwMHIWP0eY?t=25353
32. Alternate Libraries • SuperPowered: https://superpowered.com/ • MWEngine: https://github.com/igorski/MWEngine