Share
Embed
Transcript
38 slides
-
✦ DroidKnights 2026 Building Audio Plugin Ecosystem for Android in
2026 atsushieno androidaudioplugin.org 2026.11.02 · ST Center, Seoul -
✦ intro Android and Audio: a rare crossing Android devs:
almost no need to explain Audio devs: almost no need to explain • app / activity / service lifecycle • trackmaking on DAWs, instruments and effects • how mobile UI works, Material Design, etc. • MIDI (1.0), audio plugins, VST3, AudioUnit, CLAP • Kotlin and Jetpack Compose • C++, CMake, JUCE, etc. • sandboxed app environment • realtime safety, latency seealso my session slides @ ADC Japan 2026 -
✦ intro Creating Music Audio apps meetings assistants Music (and
Video) apps games play create ... -
✦ intro Creating Music on Android Creating Music On iOS
On PC (No AI related arguments here) On Android This should change! -
✦ intro History of digital trackmaking on PC 80s code
90s 00s 10s MIDI sequencers DAWs (paper scores) trackers (TUI/GUI) -
✦ intro History of PC-controlled digital instruments 80s 90s 00s
10s audio plugins (VST, AU, ...) (analog) FM, PSG, ... MIDI modules -
✦ Intro DAW (digital audio workstation) A DAW is like
an IDE for musicians. • You create a song project which contains ◦ Audio tracks: edit recorded audio ◦ Instrument tracks: edit scores (like MIDI) -
✦ Intro Audio plugin • They are like libraries /
components on an IDE project ◦ • Music producers install a lot of audio plugins (100+ on average) Two categories: ◦ Instruments: transform commands (like MIDI messages) to audio outputs ◦ • Effects: transform audio inputs to audio outputs Audio track: effect plugins Instrument track: an instrument plugin + effect plugins -
✦ Intro Audio plugin • Audio plugin software is kind
of artistic ◦ like real instruments: Steinway, Gibson/Fender, Stradivari... ◦ "I want to use synth (XYZ)" is kind of frequent request I hear. -
✦ Audio Plugins Ecosystem Audio plugin formats • An audio
plugin conforms to some plugin format • Any DAW can use any audio plugin as long as they speak the same language (API), and vice versa • There are multiple industrial standards: VST3, AudioUnit v2, AudioUnit v3, LV2, CLAP, AAX... • Typical DAWs and plugins support multiple formats -
✦ Audio Plugins Ecosystem Audio Plugins Ecosystem consists of... •
Audio Plugin • DAW (plugin host) • Audio Plugin Format -
✦ Audio Plugins Ecosystem What should DAWs implement? They have
massive workloads, but to name a few • plugin hosting • audio graph routing for each track and mixing • editable project document model • audio engine: playback (play/pause/stop/jump) • editors (piano roll, step sequencer, etc.) • macro / scripting / MCP (modern features) Too much work to explain today...! -
✦ Audio Plugins Ecosystem What should audio plugins implement? •
process audio (with MIDI) some-audio-plugin-api.h • • take float parameter changes (in realtime manner) save and load "states" ◦ as part of saving/loading of a DAW project typedef struct audio_plugin { int (*prepare) (struct audio_plugin, ...); int (*activate) (struct audio_plugin); int (*process) (struct audio_plugin, ...); ◦ • states: parameters and non-float properties provide a set of "presets" (set of well-defined states) int (*get_state) (struct audio_plugin, ...); int (*load_preset) (struct audio_plugin, ...); int (*createUI) (struct audio_plugin, ...); • provide custom GUI controller (not going to explain them today...!) ... } audio_plugin_t; -
✦ Audio Plugins Ecosystem How about audio plugin formats? •
They are just a C API ◦ • They usually come up with some SDK (implementation) ◦ • They must be in native language, and good enough to keep ABI compatibility across compilers in C++ or higher-level languages (such as Objective-C) defining API is easy. Market adoption is super hard. ◦ DAWs won't support a plugin format if there is no plugin. ◦ Plugins won't support a plugin format if there is no DAW. -
✦ Audio Plugins Ecosystem plugin formats and platforms VST3, LV2,
CLAP Windows macOS Linux iOS Web Android (WebCLAP) AudioUnit v2 AudioUnit v3 -
✦ Audio Plugins Ecosystem What happens if there is no
plugin ecosystem? • We still have many Android DAWs, but they can only use their own instruments ◦ or SoundFont (which is like software MIDI 1.0) from '00s This does not happen We are around here -
✦ Audio Plugins Ecosystem Why are plugin formats missing on
Android? • Because no one built one somehow...! • Who built other plugin formats? ◦ DAW developers (VST3, CLAP) ◦ platform developers (AudioUnit v2/v3) ◦ dev. community (LV2) -
✦ AAP AAP: Audio Plugins For Android If we don't
have any, why not build one! AAP fundamental principles: • liberal open source license, minimum deps • can be released to Google Play Store (no root) • ABI must be preserved • port existing plugins because they are artistic work (DAWs: mobile OSS DAWs don't exist...) -
✦ AAP Are we audio plugin ecosystem yet? AAP has
• a fully featured plugin format API • ~30 plugins apps, 100+ plugins (or 600+...) ◦ popular OSS plugins ported to Android • a few plugin hosts • small community of a few devs/users • no commercial products • no business plan (!) -
✦ AAP Porting existing audio plugins to AAP • •
Plugin devs use C++, JUCE framework (like 75% in the market...) ◦ supports [multi-platforms] x [multi-formats] ◦ Most devs build cross-platform UI in C++(!) In 2026, most of them can be ported to AAP by ◦ patching JUCE (in aap-juce) ◦ custom JUCE module for AAP support (in aap-juce) ◦ patching each app for Android (in every port) Similar work is done for other (non-JUCE) plugin ports too. JUCE Apps patched AAP-patched JUCE JUCE juceaap::* AAP VST, AU, ... -
✦ AAP How about DAWs? We have our own (there
was no OSS DAWs dedicated to Android) Maybe not really a "DAW" yet -
✦ Android pitfalls Android: many pitfalls • designing for IPC-based
plugin API • native remote GUI • dealing with plugin deaths • threading • storage access • performance vs. battery balance • ... -
✦ Android pitfalls IPC-based plugin API • We cannot dlopen("libmyplugin.so"),
so we use Android Binder IPC. • Binder even supports realtime-safe calls at the kernel level (but N/A in the framework domain...!) -
✦ Android pitfalls IPC-based plugin API • The most important
rule in API: keep ABI compatibility But there are always requests for new features. • Therefore, most plugin features are defined as "extensions" (state extension, parameters extension, presets extension, GUI extension, ...) • AAP avoids extensions API in Binder AIDL because its ABI maintainability is uncertain. • Instead, they are defined in a binary protocol (like gRPC) over two transports: ◦ Binder (non-realtime) ◦ MIDI 2.0 data buffer in audio processing (realtime) -
✦ Android pitfalls GUI options Host (DAW) runs the UI
• Host can provide somewhat more consistent UX • Host cannot run native executable code • UI by scripting e.g. Web UI Common to Web tech, but very few plugins use Web UI System Alert Window: special permission required Plugin runs the UI: SurfaceControlViewHost • The mechanism goes complicated • only available since Android 11+ (15+ for IME) • can run any UI: Compose, JUCE, Web, ... It is the only practical option in 2026 Bubbles, Activity Embedding: workflow not quite there 05 -
✦ Android pitfalls Performance • ADPF @ Oboe wiki Low
Latency Android Audio with improved CPU Performance - Phil Burk - ADC 2025 Just one liner: (oboeStream)->setPerformanceHintEnabled(true); -
✦ going forward Fixing Android platform defects • Google should
provide realtime IPC channels for audio threads https://issuetracker.google.com/issues/531511924 ◦ realtime IPC is technically available to device vendors at Linux kernel level ◦ If you build AOSP-based devices, maybe you can create "advantage" over Google: "We support realtime priority inheritance over AAudio thread" -
✦ going forwards AAP-specific visions • AAP 1.0 Stable Release
at some stage • More plugins on Google Play Store (they blocked me after publishing many plugins though!) ◦ • All the supported AAP plugins (OSS) are published to GitHub Releases first. support in-host-process plugin audio processor like (some) Web Audio plugin formats (no RT-safety concern, but people don't builds such plugins yet) -
✦ going forwards Recap • People use DAWs and audio
plugins for creating music in 2026 and those plugins are artistic work • We need some audio plugin format/ecosystem for Android for modern creative activities ◦ • We (format developers / app developers) have to deal with some platform limitations ◦ • and hopefully AAP can be a savior especially when porting apps from desktop But we can do it! -
Speaker @atsushieno (on GitHub, g0v.social, Bsky, speakerdeck, youtube, soundcloud, ...)
• AAP: Audio Plugins For Android • UAPMD, uapmd-kmp : MIDI 2.0 native DAW/host engine • ktmidi, compose-audio-controls, etc. • used to work on XML, WS, g11n, RX, msbuild, Android API in C#, etc. at mono / xamarin team (~2018) -
✦ Q&A Thanks! Shoot questions now and/or @atsushieno or at
atsushieno/aap-core ! Contact · SNS goes here