Android Audio: Beyond Winning On It

DroidKaigi 2025

Transcript

1. Android Audio: Beyond Winning On It atsushieno The slides are available at https://speakerdeck.com/atsushieno
2. Audio apps? Should we care about them? • kind of yes ◦ Every app potentially features audio: games, calls, alerts, ... ◦ The Android platform itself features it a lot too • music and audio apps - still, kind of yes ◦ iTunes, Spotify, YouTube, TikTok, etc. • creative music apps - maybe not many for you We're focusing on the "not many" part today😅
3. We're behind iOS When we find Audio product XXX for mobile on the web, we click the link to find a link only to AppStore, or we search on PlayStore to find nothing󰷹 That should change󰙢
4. We're behind iOS, in what? Okay, not everything. app category Media apps 👍 ... ... Creative apps 󰻷
5. But why is Android not a first citizen in the first place?
6. 1: "Android audio is laggy" Still not ideal (<10 msec.), it is not too much nowaday. (It used to be a lot.) 2021: https://android-developers.googleblog.com/2021/03/an-update-on-androids-audio-latency.html
7. 1: "Android audio is laggy" Pro Audio compliant devices achieve < 20 msec. latency https://chromium.googlesource.com/external/github.com/google/oboe/+/devicelist/docs/Devices.md
8. What is "audio latency", by the way? • Audio program should be realtime-safe (special kind of coding) ◦ Keeping deadline is the only requirement ◦ RT-safety is required only on audio threads (e.g. GUI is not RT-safe) ◦ If anyone involved broke the deadline ▪ it may result in audio glitches (noises) 😖 ▪ some pro musicians notice the lag • annoying to play together • uncomfortable recordings, retakes
9. Android audio framework is "done" there • AAudio API (since Android 8.1) has solid support for low latency audio ◦ at best, an audio device can be exclusively occupied by an app (usually it is shared) ◦ Oboe is more popular, user-friendly API (also works for <8.0 platforms) • Android CDD section 5.6 Audio Latency and 5.10 Professional Audio ◦ ProAudio round-trip latency must be within 20 msec. (to analog) or 25 msec. (to USB) If you are serious about latency, get a Pro-Audio device (Pixel, some Galaxy, ROG...)
10. Should we have to care about audio latency? Media players like YouTube / YT Music use Jetpack Media3 (ExoPlayer) and ExoPlayer does not even use AAudio (and not laggy) But creative music apps are lively interactive and latency matters.
11. 2: "Android uses Java (or Kotlin)" • RT-safe code must be written in RT-safe language (C++, C, Rust, Zig...) • We do write audio code in C++ on Android too • It should not matter because RT safety is required only on audio thread
12. GUI: There's more than one way to do it and any approach is alright Desktop, Android, iOS... Android Audio in C++ Audio in C++ x-plat UI, C++ Compose, Android View iOS Audio in C++ SwiftUI, UIKit Desktop Audio in C++ native C++ Desktop Audio in C++ native UI C#, Swift, ... Desktop, Android, iOS... Audio in C++ x-plat UI C#, Dart, Web UI, ... It is particularly popular in audio (JUCE, OpenGL, Qt, etc.)
13. How can we write C++ Android audio code? Winning on Android [ADC 2018] But - winning on Android is not enough. Real-time processing on Android [ADC 2019] We need Android itself winning 🦾
14. 3. "iOS provides more audio features. Android doesn't" Android actually brought in a lot of audio features: • MIDI 1.0 and 2.0, in Java and C++ (Android 6, 10, 13, 15) • Spatializer and hardware-accelerated head tracker in Android 12 • LE Audio support and codec enhancements (e.g. Pixel Buds) • OEM audio plugin services for Android Auto in Android 14 Can you name any other features that are required on Android platform?
15. Android still misses audio plugin system(s) audio plugins: framework for instruments (synths) and effects. Apple platforms have "AudioUnit V3"
16. Android still misses audio plugin system(s) audio plugins: framework for instruments (synths) and effects. Apple platforms have "AudioUnit V3"
17. 4: "Mobile platforms are hard" • Limited computing resources • Small screens • One app per screen • Isolated apps We can't "fix" those issues ➔ Have different product goals for mobiles. (iOS is hard too.) cf. new desktop mode on Android tablets.
18. What are audio plugins?
19. DAW (plugin host) We typically use this kind of complicated software for trackmaking (or "DTM" in Japanglish) Take it like IDE for composers.
20. Audio Plugins: instruments / synthesizers / effects There are many instrument vendors like real (physical) instruments Take them like libraries on programming language runtimes.
21. Audio Plugin Formats (APIs) There must be some API to play instruments ➔ audio plugin format (MIDI 1.0 is not expressive enough) There are some common formats (VST3, AudioUnit, LV2, CLAP, AAX, ...) They are different but similar. Take them like programming language runtimes.
22. iOS: AudioUnit V3 Apple has AudioUnit V3 which works nicely on iOS AudioUnit Format (interface)
23. Why does it matter? Designing instruments is some artistic work. Those synth devs are irreplaceable. A plugin format builds its ecosystem. More supporting host DAWs => expands every plugin's market, and vice versa. If an instrument cannot be used on a DAW, do you purchase it? If a DAW cannot load others' instruments, do you purchase it? This is Android now. This should change.
24. Why does it matter? "The plug-ins Steinberg offer do not compare to the plethora of VSTs, AUs that iOS users can install. This needs to change in all fairness to make it a complete DAW." (Steinberg responds) "Unfortunately the given limitations on Android, prevents the app to show up with the same features compared to iOS"
25. Why does it matter? (latency is not an issue but) we need audio plugins on Android! 2023: https://www.reddit.com/r/androiddev/comments/16rpp9z/android_low_latency_high_performance_audio_in_2023/
26. Can't we build audio plugin systems on Android?
27. It is possible, with a handful of technical hurdles I have partially successful experiments. Take it like iOS AudioBus (product).
28. Host and plugins, on desktop On desktop platforms, music apps load "plugins" as libraries (`dlopen()` etc.) Host Plugin
29. Host and plugin, over inter-app audio system • iOS and Android: we can't load dynamic libs from another app ◦ Technically we can on Android, but it violates Google Play Store policy. • We have to use IPC (code) and shared memory (data) instead 🎧 🎧
30. Android Binder (Service) • Android system had the most advanced IPC foundation • C++ API available (Android 10) • supports realtime priority inheritance at Linux kernel level (Android 8) ◦ it usually does not happen because "RT-safety == no system calls" ◦ but unavailable in framework domain... (i.e. vendor-only) 😖
31. GUI overlay on Android It is usually impossible, but we can do something like these Compose UI on native UI (JUCE) Native UI (JUCE) on Compose UI
32. options for GUI solutions • SurfaceControlViewHost (since Android 11) ◦ any View can be rendered on host SurfaceView (host instantiates them) ◦ gives almost full UI control to the plugin process itself. ◦ hard to deal with it, hard to debug (e.g. I cannot fully control JUCE UI) • Web UI ◦ loadable on the hosting (DAW) process ◦ build plugin UI has to be written in Web technology ◦ UI has to interact with non-UI (DSP) via IPC ◦ might become a thing depending on how audio app ecosystem accepts Web UI • System Alert Window - kind of deprecated
33. How about AudioUnit V3? Apple had AudioUnit V2, then created a mobile-friendly V3. based on App Extensions technology (AudioUnit extensions) Even released Logic Pro for iPad (2023) (and of course other DAWs can use AUv3) Their GUI can show up as "compact view" https://www.apple.com/logic-pro-for-ipad/
34. How about iOS? iOS brought RT priority inheritance to userspace as XNU turnstiles (iOS 12) and Audio Workgroups (iOS 14). DAWs and AudioUnits appreciate them. Meet Audio Workgroups (WWDC 2020) (no need if your app and plugins work with just one audio thread)
35. Make changes
36. Platform: loading DSPs in-process? It seems "open up RT priority inheritance to everyone" is likely refused󰢄 by the Android platform team (fair enough...) Android 16 supports in-process media codecs (vendor codecs loaded in user apps) • only "safe" codecs can be loaded in-process (default: sandboxed) • safe == written in Rust etc. • verified when their APEX package is loaded • not like any app developer on Play Store can distribute (vendor-only)
37. Platform: Can we give priority on verified plugin code? • Run arbitrary plugin code in process for verified code? ➔ hard ◦ How do you design data sharing between in-process DSP code and out-of-process UI code? • Give IPC priority inheritance for verified code? ➔ maybe? But there is no official way to write "safe" code right now: • Rust is not even an official NDK language • Those "artistic" works are already done in C++... "safe C++" kind of? • Standardized safety verification? We think about them in the future 🫠
38. Community: grow audio app ecosystems • Build inter-app audio ecosystems (plugin formats). • Improve Android support in x-plat music C++ libraries ◦ Most of existing apps are written in x-plat code w/ those libraries ▪ I indeed ported ~30 plugins from x-plat OSS projects to Android • Support and enhance x-plat technologies for easier porting from desktop • MIDI 2.0 offers a lot of audio plugin feature parity • Web UI, JavaScript and WebAssembly
39. Summary Android platform achieved a lot of audio features except for audio plugins and that's why we're behind iOS in this market We can build audio plugin ecosystems - without realtime safety (yet) We can improve the ecosystem by ourselves!
40. Thanks! Links • https://github.com/atsushieno/aap-core - Audio Plugins For Android (AAP) • https://speakerdeck.com/atsushieno - similar slides from past session talks Contacts • https://g0v.social/@atsushieno (Mastodon / en) • https://fedibird.com/@atsushieno (Mastodon / ja) • https://bsky.app/profile/atsushieno.bsky.social (ATproto / ja) Android Audio Devs discord https://discord.gg/G8br7X2m