Slide 1

Slide 1 text

Android Audio: Beyond Winning On It atsushieno The slides are available at https://speakerdeck.com/atsushieno

Slide 2

Slide 2 text

Audio apps? Should we care about them? ● kind of yes ○ Every app potentially features audio: games, calls, alerts, ... ○ The Android platform itself features it a lot too ● music and audio apps - still, kind of yes ○ iTunes, Spotify, YouTube, TikTok, etc. ● creative music apps - maybe not many for you We're focusing on the "not many" part today😅

Slide 3

Slide 3 text

We're behind iOS When we find Audio product XXX for mobile on the web, we click the link to find a link only to AppStore, or we search on PlayStore to find nothing󰷹 That should change󰙢

Slide 4

Slide 4 text

We're behind iOS, in what? Okay, not everything. app category Media apps 👍 ... ... Creative apps 󰻷

Slide 5

Slide 5 text

But why is Android not a first citizen in the first place?

Slide 6

Slide 6 text

1: "Android audio is laggy" Still not ideal (<10 msec.), it is not too much nowaday. (It used to be a lot.) 2021: https://android-developers.googleblog.com/2021/03/an-update-on-androids-audio-latency.html

Slide 7

Slide 7 text

1: "Android audio is laggy" Pro Audio compliant devices achieve < 20 msec. latency https://chromium.googlesource.com/external/github.com/google/oboe/+/devicelist/docs/Devices.md

Slide 8

Slide 8 text

What is "audio latency", by the way? ● Audio program should be realtime-safe (special kind of coding) ○ Keeping deadline is the only requirement ○ RT-safety is required only on audio threads (e.g. GUI is not RT-safe) ○ If anyone involved broke the deadline ■ it may result in audio glitches (noises) 😖 ■ some pro musicians notice the lag ● annoying to play together ● uncomfortable recordings, retakes

Slide 9

Slide 9 text

Android audio framework is "done" there ● AAudio API (since Android 8.1) has solid support for low latency audio ○ at best, an audio device can be exclusively occupied by an app (usually it is shared) ○ Oboe is more popular, user-friendly API (also works for <8.0 platforms) ● Android CDD section 5.6 Audio Latency and 5.10 Professional Audio ○ ProAudio round-trip latency must be within 20 msec. (to analog) or 25 msec. (to USB) If you are serious about latency, get a Pro-Audio device (Pixel, some Galaxy, ROG...)

Slide 10

Slide 10 text

Should we have to care about audio latency? Media players like YouTube / YT Music use Jetpack Media3 (ExoPlayer) and ExoPlayer does not even use AAudio (and not laggy) But creative music apps are lively interactive and latency matters.

Slide 11

Slide 11 text

2: "Android uses Java (or Kotlin)" ● RT-safe code must be written in RT-safe language (C++, C, Rust, Zig...) ● We do write audio code in C++ on Android too ● It should not matter because RT safety is required only on audio thread

Slide 12

Slide 12 text

GUI: There's more than one way to do it and any approach is alright Desktop, Android, iOS... Android Audio in C++ Audio in C++ x-plat UI, C++ Compose, Android View iOS Audio in C++ SwiftUI, UIKit Desktop Audio in C++ native C++ Desktop Audio in C++ native UI C#, Swift, ... Desktop, Android, iOS... Audio in C++ x-plat UI C#, Dart, Web UI, ... It is particularly popular in audio (JUCE, OpenGL, Qt, etc.)

Slide 13

Slide 13 text

How can we write C++ Android audio code? Winning on Android [ADC 2018] But - winning on Android is not enough. Real-time processing on Android [ADC 2019] We need Android itself winning 🦾

Slide 14

Slide 14 text

3. "iOS provides more audio features. Android doesn't" Android actually brought in a lot of audio features: ● MIDI 1.0 and 2.0, in Java and C++ (Android 6, 10, 13, 15) ● Spatializer and hardware-accelerated head tracker in Android 12 ● LE Audio support and codec enhancements (e.g. Pixel Buds) ● OEM audio plugin services for Android Auto in Android 14 Can you name any other features that are required on Android platform?

Slide 15

Slide 15 text

Android still misses audio plugin system(s) audio plugins: framework for instruments (synths) and effects. Apple platforms have "AudioUnit V3"

Slide 16

Slide 16 text

Android still misses audio plugin system(s) audio plugins: framework for instruments (synths) and effects. Apple platforms have "AudioUnit V3"

Slide 17

Slide 17 text

4: "Mobile platforms are hard" ● Limited computing resources ● Small screens ● One app per screen ● Isolated apps We can't "fix" those issues ➔ Have different product goals for mobiles. (iOS is hard too.) cf. new desktop mode on Android tablets.

Slide 18

Slide 18 text

What are audio plugins?

Slide 19

Slide 19 text

DAW (plugin host) We typically use this kind of complicated software for trackmaking (or "DTM" in Japanglish) Take it like IDE for composers.

Slide 20

Slide 20 text

Audio Plugins: instruments / synthesizers / effects There are many instrument vendors like real (physical) instruments Take them like libraries on programming language runtimes.

Slide 21

Slide 21 text

Audio Plugin Formats (APIs) There must be some API to play instruments ➔ audio plugin format (MIDI 1.0 is not expressive enough) There are some common formats (VST3, AudioUnit, LV2, CLAP, AAX, ...) They are different but similar. Take them like programming language runtimes.

Slide 22

Slide 22 text

iOS: AudioUnit V3 Apple has AudioUnit V3 which works nicely on iOS AudioUnit Format (interface)

Slide 23

Slide 23 text

Why does it matter? Designing instruments is some artistic work. Those synth devs are irreplaceable. A plugin format builds its ecosystem. More supporting host DAWs => expands every plugin's market, and vice versa. If an instrument cannot be used on a DAW, do you purchase it? If a DAW cannot load others' instruments, do you purchase it? This is Android now. This should change.

Slide 24

Slide 24 text

Why does it matter? "The plug-ins Steinberg offer do not compare to the plethora of VSTs, AUs that iOS users can install. This needs to change in all fairness to make it a complete DAW." (Steinberg responds) "Unfortunately the given limitations on Android, prevents the app to show up with the same features compared to iOS"

Slide 25

Slide 25 text

Why does it matter? (latency is not an issue but) we need audio plugins on Android! 2023: https://www.reddit.com/r/androiddev/comments/16rpp9z/android_low_latency_high_performance_audio_in_2023/

Slide 26

Slide 26 text

Can't we build audio plugin systems on Android?

Slide 27

Slide 27 text

It is possible, with a handful of technical hurdles I have partially successful experiments. Take it like iOS AudioBus (product).

Slide 28

Slide 28 text

Host and plugins, on desktop On desktop platforms, music apps load "plugins" as libraries (`dlopen()` etc.) Host Plugin

Slide 29

Slide 29 text

Host and plugin, over inter-app audio system ● iOS and Android: we can't load dynamic libs from another app ○ Technically we can on Android, but it violates Google Play Store policy. ● We have to use IPC (code) and shared memory (data) instead 🎧 🎧

Slide 30

Slide 30 text

Android Binder (Service) ● Android system had the most advanced IPC foundation ● C++ API available (Android 10) ● supports realtime priority inheritance at Linux kernel level (Android 8) ○ it usually does not happen because "RT-safety == no system calls" ○ but unavailable in framework domain... (i.e. vendor-only) 😖

Slide 31

Slide 31 text

GUI overlay on Android It is usually impossible, but we can do something like these Compose UI on native UI (JUCE) Native UI (JUCE) on Compose UI

Slide 32

Slide 32 text

options for GUI solutions ● SurfaceControlViewHost (since Android 11) ○ any View can be rendered on host SurfaceView (host instantiates them) ○ gives almost full UI control to the plugin process itself. ○ hard to deal with it, hard to debug (e.g. I cannot fully control JUCE UI) ● Web UI ○ loadable on the hosting (DAW) process ○ build plugin UI has to be written in Web technology ○ UI has to interact with non-UI (DSP) via IPC ○ might become a thing depending on how audio app ecosystem accepts Web UI ● System Alert Window - kind of deprecated

Slide 33

Slide 33 text

How about AudioUnit V3? Apple had AudioUnit V2, then created a mobile-friendly V3. based on App Extensions technology (AudioUnit extensions) Even released Logic Pro for iPad (2023) (and of course other DAWs can use AUv3) Their GUI can show up as "compact view" https://www.apple.com/logic-pro-for-ipad/

Slide 34

Slide 34 text

How about iOS? iOS brought RT priority inheritance to userspace as XNU turnstiles (iOS 12) and Audio Workgroups (iOS 14). DAWs and AudioUnits appreciate them. Meet Audio Workgroups (WWDC 2020) (no need if your app and plugins work with just one audio thread)

Slide 35

Slide 35 text

Make changes

Slide 36

Slide 36 text

Platform: loading DSPs in-process? It seems "open up RT priority inheritance to everyone" is likely refused󰢄 by the Android platform team (fair enough...) Android 16 supports in-process media codecs (vendor codecs loaded in user apps) ● only "safe" codecs can be loaded in-process (default: sandboxed) ● safe == written in Rust etc. ● verified when their APEX package is loaded ● not like any app developer on Play Store can distribute (vendor-only)

Slide 37

Slide 37 text

Platform: Can we give priority on verified plugin code? ● Run arbitrary plugin code in process for verified code? ➔ hard ○ How do you design data sharing between in-process DSP code and out-of-process UI code? ● Give IPC priority inheritance for verified code? ➔ maybe? But there is no official way to write "safe" code right now: ● Rust is not even an official NDK language ● Those "artistic" works are already done in C++... "safe C++" kind of? ● Standardized safety verification? We think about them in the future 🫠

Slide 38

Slide 38 text

Community: grow audio app ecosystems ● Build inter-app audio ecosystems (plugin formats). ● Improve Android support in x-plat music C++ libraries ○ Most of existing apps are written in x-plat code w/ those libraries ■ I indeed ported ~30 plugins from x-plat OSS projects to Android ● Support and enhance x-plat technologies for easier porting from desktop ● MIDI 2.0 offers a lot of audio plugin feature parity ● Web UI, JavaScript and WebAssembly

Slide 39

Slide 39 text

Summary Android platform achieved a lot of audio features except for audio plugins and that's why we're behind iOS in this market We can build audio plugin ecosystems - without realtime safety (yet) We can improve the ecosystem by ourselves!

Slide 40

Slide 40 text

Thanks! Links ● https://github.com/atsushieno/aap-core - Audio Plugins For Android (AAP) ● https://speakerdeck.com/atsushieno - similar slides from past session talks Contacts ● https://g0v.social/@atsushieno (Mastodon / en) ● https://fedibird.com/@atsushieno (Mastodon / ja) ● https://bsky.app/profile/atsushieno.bsky.social (ATproto / ja) Android Audio Devs discord https://discord.gg/G8br7X2m