talk about it) An experimental plugin format and implementation that is designed to work on Android platform, which - users can combine any DAW with any plugin from different app packages - Your favorite plugin vendor does not have to "partner" with your DAW vendor (and vice versa) - can show plugin UI (Compose, JUCE, Web...) - some plugins and hosts ported from LV2 (aap-lv2) and JUCE (aap-juce)
of features we could achieve on Android how we can realize these features how to use AAP (*my* plugin format), AAP API details 🙆 what AAP ecosystem does NOT provide so far
Android audio dev. talk is about • how to build C++ app using Android NDK with Java/Kotlin • how to use Oboe • how to integrate MIDI (Android MIDI API or AMidi NDK API) • how to build Android app UI based on { Android view / Jetpack Compose } This session does not cover any of above(!) See DonTurner-ADC18 on those topics.
Plugin Format itself is NOT an SDK 1. Its first priority is to preserve ABI compatibility. 2. Ease of development is just a nice to have. Low priority. 3. It must be kept in C ABI or more generic level, with RT safety. 4. The scope of the API should be limited to interaction between host and plugin Should a plugin format API specify messaging between plugin DSP and plugin UI? LV2: YES / CLAP: NO (cf. WebCLAP?)
⇛ Google then? Can this be open? • Steinberg owns VST3 and Bitwig owns CLAP ⇛ DAW vendors then? Any of those on Android? ◦ actually there was: AAP is NOT the first attempt for audio plugin format. n-track AudioRoute SDK (~2021) • In principle, anyone should be able to build ones
JUCE API is in C++, expecting static linking, which does not preserve ABI. • JUCE is a plugin/host SDK-or-framework (among many features). C++ for ABI does not work. (We learned the lesson from VCVRack.)
is the lowest level down to kernel level Binder itself is capable of realtime priority inheritance, but it is unavailable to app devs... (Binder calls from audio thread indeed does not inherit priority.) cf. iOS AudioUnit v3 audio processors work in RT-safe manner (XNU Turnstiles) https://source.android.com/docs/core/architecture/hidl/binder-ipc
can be performed in RT-safe manner, interaction between host and plugin should be kept minimum in audio processing. There will be a lot of context switches. • Callbacks from plugin to host within audio processing should be avoided too. ◦ CLAP thread-pool extension (microthreading) ◦ AUv2 HostCallback_GetBeatAndTempo() etc.
GUI...) • plugin-developer-defined controllers that may not fit well with knob or slider • properties, non-float parameters ◦ LV2 supports non-float parameters (Atoms), but typical multi-format DAWs and plugins don't support them
GUI...) • unmanaged presets by the plugin format API Most of existing plugins don't expose presets via presets API (!) Even though GUI is not mandatory, plugins without GUI drops a lot...
good, but still problematic ◦ IME does not work with SVCH as far as I tried (it matters in every language, not just CJKV) • Web UI - not much less ability, but needs JS-DSP interaction • Notification Bubbles - awkward displaying, but IME would work ("chat sample") • system alert window - deprecated platform feature, hard to manage AAP supports SCVH and Web UI in principle
expect one global display (`addToDesktop()`) - PopupMenu etc. JUCE dialogs sync API is unavailable on Android JUCE expects that a launcher Activity within the app itself (not true for Service). aap-juce comes up with a handful of JUCE patches for handful of versions🙃
per-app data storage. We have to use platform Document Provider API / Storage Access Framework. Where I was stuck without files: - Presets - Sampler files - DSP scripts (e.g. JSFX in aap-juce-ysfx) They are bundled as assets, extracted to local filesDir at first run.
GUI thread requirement + synchronous API etc. - currently state API itself is sync, but the SDK part uses it async way. - expecting synchronous execution over IPC is error-prone: Fabian Renn-ADC24 - synchronous state API on main thread can block more often cf. clap issue #484 (asynchronous state API) I'm quite unsure if VST3 and CLAP APIs are safe to reuse. (JUCE plugins seem to be working though.)
VST3 and CLAP require too much main thread work that tend to block. • LV2 and CLAP require custom UI type. (VST3 too but the API is abstract) ◦ We don't go GUI-less. GUI matters (for good or not). • WebCLAP "Wasm as common ABI" is an interesting solution ◦ actually even Android NDK team once thought of that (ndk/issues/1771) ◦ I might add dual DSP bundle (WASM for host, native for plugin) in the future. • adopting WebCLAP / WebView UI practically means GUI-less nowaday. ◦ Can we integrate juce_emscripten? For now I find AAP defining its own API is more practical.
calls on Binder IPC: • create(), prepare(), activate(), deactivate(), process(), extension() • when they change, that means essential format change (like VST2 -> VST3) AAP extensions (states, presets, UI, etc.) are designed as binary protocol Transport is either Binder (non-RT) or MIDI 2.0 SysEx on process() (RT). AAP State extension can work like Get and Set Device State MIDI-CI Properties.
is Polished and Stable ◦ Plugins: Build LV2 plugin or JUCE plugin and then port to Android ◦ Hosts: Use aap-juce (aap_audio_plugin_client) or atsushieno/uapmd (still not as easy as clap-juce-extensions) • Really useful DAW kind of hosts ◦ atsushieno/uapmd on Android: confirmed 6 tracks without glitches with 1024 bytes buffer. (uapmd is primarily a desktop plugin host and sequencer...) • Released Apps on Play Store ◦ Use atsushieno/android-ci-package-installer or Obtainium ◦ Hopefully release soon-ish • Business Plans💸
the first citizen. ◦ AAP is the only viable one so far. • Android has its own difficulty to achieve plugin features in: ◦ cross process architecture (namely it is not RT-safe) ◦ GUI ◦ file systems ◦ asynchronous API If you are to build a plugin format for Android, you will be hit by these issues😈
atsushieno
ezaki
civitaspo
iwashi86
nari_ex
chanyou0311
kairim0
ham0215
sheharyar
yuyatakeyama
ysd113
cyberagentdevelopers
sonic
scottboms
eileencodes
aleyda
uxyall
jonrohan
davidcarrasco
jonoalderson
seathinner
colly
codingconduct
mlcsv
