Building Audio Plugin Ecosystem for Android in 2026

DroidKnights 2026

Transcript

1. ✦ DroidKnights 2026 Building Audio Plugin Ecosystem for Android in 2026 atsushieno androidaudioplugin.org 2026.11.02 · ST Center, Seoul
2. 01 intro
3. ✦ intro Android and Audio: a rare crossing Android devs: almost no need to explain Audio devs: almost no need to explain • app / activity / service lifecycle • trackmaking on DAWs, instruments and effects • how mobile UI works, Material Design, etc. • MIDI (1.0), audio plugins, VST3, AudioUnit, CLAP • Kotlin and Jetpack Compose • C++, CMake, JUCE, etc. • sandboxed app environment • realtime safety, latency seealso my session slides @ ADC Japan 2026
4. ✦ intro Creating Music Audio apps meetings assistants Music (and Video) apps games play create ...
5. ✦ intro Creating Music on Android Creating Music On iOS On PC (No AI related arguments here) On Android This should change!
6. ✦ intro History of digital trackmaking on PC 80s code 90s 00s 10s MIDI sequencers DAWs (paper scores) trackers (TUI/GUI)
7. ✦ intro History of PC-controlled digital instruments 80s 90s 00s 10s audio plugins (VST, AU, ...) (analog) FM, PSG, ... MIDI modules
8. ✦ Intro DAW (digital audio workstation) A DAW is like an IDE for musicians. • You create a song project which contains ◦ Audio tracks: edit recorded audio ◦ Instrument tracks: edit scores (like MIDI)
9. ✦ Intro Audio plugin • They are like libraries / components on an IDE project ◦ • Music producers install a lot of audio plugins (100+ on average) Two categories: ◦ Instruments: transform commands (like MIDI messages) to audio outputs ◦ • Effects: transform audio inputs to audio outputs Audio track: effect plugins Instrument track: an instrument plugin + effect plugins
10. ✦ Intro Audio plugin • Audio plugin software is kind of artistic ◦ like real instruments: Steinway, Gibson/Fender, Stradivari... ◦ "I want to use synth (XYZ)" is kind of frequent request I hear.
11. ✦ Audio Plugins Ecosystem Audio plugin formats • An audio plugin conforms to some plugin format • Any DAW can use any audio plugin as long as they speak the same language (API), and vice versa • There are multiple industrial standards: VST3, AudioUnit v2, AudioUnit v3, LV2, CLAP, AAX... • Typical DAWs and plugins support multiple formats
12. PC DAW demo hopefully we have time for that
13. 02 Audio Plugins Ecosystem
14. ✦ Audio Plugins Ecosystem Audio Plugins Ecosystem consists of... • Audio Plugin • DAW (plugin host) • Audio Plugin Format
15. ✦ Audio Plugins Ecosystem What should DAWs implement? They have massive workloads, but to name a few • plugin hosting • audio graph routing for each track and mixing • editable project document model • audio engine: playback (play/pause/stop/jump) • editors (piano roll, step sequencer, etc.) • macro / scripting / MCP (modern features) Too much work to explain today...!
16. ✦ Audio Plugins Ecosystem What should audio plugins implement? • process audio (with MIDI) some-audio-plugin-api.h • • take float parameter changes (in realtime manner) save and load "states" ◦ as part of saving/loading of a DAW project typedef struct audio_plugin { int (*prepare) (struct audio_plugin, ...); int (*activate) (struct audio_plugin); int (*process) (struct audio_plugin, ...); ◦ • states: parameters and non-float properties provide a set of "presets" (set of well-defined states) int (*get_state) (struct audio_plugin, ...); int (*load_preset) (struct audio_plugin, ...); int (*createUI) (struct audio_plugin, ...); • provide custom GUI controller (not going to explain them today...!) ... } audio_plugin_t;
17. ✦ Audio Plugins Ecosystem How about audio plugin formats? • They are just a C API ◦ • They usually come up with some SDK (implementation) ◦ • They must be in native language, and good enough to keep ABI compatibility across compilers in C++ or higher-level languages (such as Objective-C) defining API is easy. Market adoption is super hard. ◦ DAWs won't support a plugin format if there is no plugin. ◦ Plugins won't support a plugin format if there is no DAW.
18. ✦ Audio Plugins Ecosystem plugin formats and platforms VST3, LV2, CLAP Windows macOS Linux iOS Web Android (WebCLAP) AudioUnit v2 AudioUnit v3
19. ✦ Audio Plugins Ecosystem What happens if there is no plugin ecosystem? • We still have many Android DAWs, but they can only use their own instruments ◦ or SoundFont (which is like software MIDI 1.0) from '00s This does not happen We are around here
20. ✦ Audio Plugins Ecosystem Why are plugin formats missing on Android? • Because no one built one somehow...! • Who built other plugin formats? ◦ DAW developers (VST3, CLAP) ◦ platform developers (AudioUnit v2/v3) ◦ dev. community (LV2)
21. 03 AAP
22. ✦ AAP AAP: Audio Plugins For Android If we don't have any, why not build one! AAP fundamental principles: • liberal open source license, minimum deps • can be released to Google Play Store (no root) • ABI must be preserved • port existing plugins because they are artistic work (DAWs: mobile OSS DAWs don't exist...)
23. ✦ AAP Are we audio plugin ecosystem yet? AAP has • a fully featured plugin format API • ~30 plugins apps, 100+ plugins (or 600+...) ◦ popular OSS plugins ported to Android • a few plugin hosts • small community of a few devs/users • no commercial products • no business plan (!)
24. ✦ AAP Porting existing audio plugins to AAP • • Plugin devs use C++, JUCE framework (like 75% in the market...) ◦ supports [multi-platforms] x [multi-formats] ◦ Most devs build cross-platform UI in C++(!) In 2026, most of them can be ported to AAP by ◦ patching JUCE (in aap-juce) ◦ custom JUCE module for AAP support (in aap-juce) ◦ patching each app for Android (in every port) Similar work is done for other (non-JUCE) plugin ports too. JUCE Apps patched AAP-patched JUCE JUCE juceaap::* AAP VST, AU, ...
25. ✦ AAP How about DAWs? We have our own (there was no OSS DAWs dedicated to Android) Maybe not really a "DAW" yet
26. AAP demo
27. 04 Android pitfalls
28. ✦ Android pitfalls Android: many pitfalls • designing for IPC-based plugin API • native remote GUI • dealing with plugin deaths • threading • storage access • performance vs. battery balance • ...
29. ✦ Android pitfalls IPC-based plugin API • We cannot dlopen("libmyplugin.so"), so we use Android Binder IPC. • Binder even supports realtime-safe calls at the kernel level (but N/A in the framework domain...!)
30. ✦ Android pitfalls IPC-based plugin API • The most important rule in API: keep ABI compatibility But there are always requests for new features. • Therefore, most plugin features are defined as "extensions" (state extension, parameters extension, presets extension, GUI extension, ...) • AAP avoids extensions API in Binder AIDL because its ABI maintainability is uncertain. • Instead, they are defined in a binary protocol (like gRPC) over two transports: ◦ Binder (non-realtime) ◦ MIDI 2.0 data buffer in audio processing (realtime)
31. ✦ Android pitfalls GUI options Host (DAW) runs the UI • Host can provide somewhat more consistent UX • Host cannot run native executable code • UI by scripting e.g. Web UI Common to Web tech, but very few plugins use Web UI System Alert Window: special permission required Plugin runs the UI: SurfaceControlViewHost • The mechanism goes complicated • only available since Android 11+ (15+ for IME) • can run any UI: Compose, JUCE, Web, ... It is the only practical option in 2026 Bubbles, Activity Embedding: workflow not quite there 05
32. ✦ Android pitfalls Performance • ADPF @ Oboe wiki Low Latency Android Audio with improved CPU Performance - Phil Burk - ADC 2025 Just one liner: (oboeStream)->setPerformanceHintEnabled(true);
33. 05 going forward
34. ✦ going forward Fixing Android platform defects • Google should provide realtime IPC channels for audio threads https://issuetracker.google.com/issues/531511924 ◦ realtime IPC is technically available to device vendors at Linux kernel level ◦ If you build AOSP-based devices, maybe you can create "advantage" over Google: "We support realtime priority inheritance over AAudio thread"
35. ✦ going forwards AAP-specific visions • AAP 1.0 Stable Release at some stage • More plugins on Google Play Store (they blocked me after publishing many plugins though!) ◦ • All the supported AAP plugins (OSS) are published to GitHub Releases first. support in-host-process plugin audio processor like (some) Web Audio plugin formats (no RT-safety concern, but people don't builds such plugins yet)
36. ✦ going forwards Recap • People use DAWs and audio plugins for creating music in 2026 and those plugins are artistic work • We need some audio plugin format/ecosystem for Android for modern creative activities ◦ • We (format developers / app developers) have to deal with some platform limitations ◦ • and hopefully AAP can be a savior especially when porting apps from desktop But we can do it!
37. Speaker @atsushieno (on GitHub, g0v.social, Bsky, speakerdeck, youtube, soundcloud, ...) • AAP: Audio Plugins For Android • UAPMD, uapmd-kmp : MIDI 2.0 native DAW/host engine • ktmidi, compose-audio-controls, etc. • used to work on XML, WS, g11n, RX, msbuild, Android API in C#, etc. at mono / xamarin team (~2018)
38. ✦ Q&A Thanks! Shoot questions now and/or @atsushieno or at atsushieno/aap-core ! Contact · SNS goes here