Taking trends in music app development into the future mobile ecosystem atsushieno 

Agenda Today I will be focusing on - audio plugin system (effects and synths) for audio apps - on mobile platforms - taking best from open technology and FLOSS - along with audio developers' tech. trends (kinda) images on the slides mostly has links to the originals (website, product, etc.) 

Tech Trends in Audio "development" ? They do not mean "majority" (more like "hot topics") ● AI-based composition (Suno, Udio, Yue, DiffRhythm, etc.) ● AI-assisted tooling (trackmaking / production tools / virtual singers) ● spatial audio ● WebView as GUI ● CLAP plugin format ● MIDI 2.0 ● C++ I check those trends at: audio.dev, theaudioprogrammer.com etc. 

Some introduction to audio/music app development 

many different kinds of audio / music software 

many different kinds of audio / music software music & video player (files, streaming) 

many different kinds of audio / music software games (music, sound effects, character voices. interactive audio) 

many different kinds of audio / music software video and voice calls (or: karaoke, sessions) 

many different kinds of audio / music software audio recorder & editor, streaming content authoring 

many different kinds of audio / music software creative tools: DAW, DJ tools, audio plugins (instruments + effects) 

How LIVE is your audio effects? (C++ or not) Music player: user may change effects interactively Games: apply effects as user controls DAW: apply effects as user plays Live streaming: change effects while user speaks They are all LIVE (some are even real-time, some are even low-latency) (also their slides) 

How LIVE is your audio app? (C++ or not) accept glitches and delays? Use C++ (or C or Rust or Zig...) Use any language! (JS, Java, Python, Go...) YES GC JIT dynamic types interpreter is it live? NO YES NO 

How LIVE is your audio app? (C++ or not) accept glitches and delays? Use any language! (JS, Java, Python, Go...) YES NO GC JIT dynamic types interpreter is it live? NO YES Use C++ (or C or Rust or Zig...) 

should we write everything in C++ (etc.) ? not really. in audio I/O thread? Use C++ etc. Use any language! YES NO UI malloc locks I/O React/JS Flutter/Dart Unity/C# 

should we write everything in C++ (etc.) ? not really. in audio I/O thread? Use C++ etc. Use any language! YES NO UI malloc locks I/O React/JS Flutter/Dart Unity/C# 

...do they use any other language? well, actually... it is changing though (WebView)　　　 does it deal with audio I/O? Use C++ etc. Use C++ regardless 😛 YES NO Qt JUCE GUI OpenGL VSTGUI Trend 

Explaining some trends AI technology: audio developers care about realtime safety ● in RT-safe code: RTNeural etc. ● in non-RT-safe code: OnnxRuntime etc. with RT thread C++: how we can achieve RT safety ● RT-safe algorithms ● memory operations (allocation, atomic shared pointers / hazard pointers) ● related toolchains and stdlibs e.g. RealtimeSanitizer 

Understanding audio plugins 

What are audio plugins? effects? synths? Audio Plugins = Effects + Synths (+ misc.) 

What are audio plugins? effects? synths? Effects Synths 

What are audio plugins? effects? synths? Effects Synths Control audio outputs by parameters and - audio inputs (effects) - instrument inputs (MIDI keyboard etc.) Not very different 

What are audio plugins? effects? synths? What can be controlled? What can be an effect? ● gain (volume), pan, envelope ● pitch shifter, tuner ● reverb ● delay echo ● equalizer ● distortion ● phazer ● vocoder ● ... What are instruments? ● samplers ● synthesizers ○ wavetable ○ additive ○ subtractive ○ FM ○ ... 

Two kinds of audio effect systems ● Platform (OS) internal effects ● Open audio plugin formats / apps 

Platform (OS) Internal Effects ● achieve basic effects: "bass boost", "equalizer", "reverb", "denoise" ... ● only platform and driver vendor provide them (mobiles: phone vendor provides) ● can be hardware-accelerated ● can be specific to platform features e.g. VR headset (spatializer) ● can be controlled by parameters, but no exact/precise audio outputs ● no GUI ● consumers: recorder, video and voice calls, streaming apps etc. (not very suitable for DAWs) e.g. android.media.audiofx API for apps, AOSP Audio effect HAL for devices Trend 

Open Audio Plugin Formats with common interface, any host can load any plugin Popular formats: ● Steinberg VST3 ● Apple AudioUnit ● Linux community LV2 ● Bitwig CLAP No one can break the ABI Music software always drives audio plugins ecosystem 

Open Audio Plugin Formats without plugin format, you can still ship a DAW, but... 

Open Audio Plugin Formats without open plugin formats: you can still ship a DAW, but... "The plug-ins Steinberg offer do not compare to the plethora of VSTs, AUs that iOS users can install. This needs to change in all fairness to make it a complete DAW." "Unfortunately the given limitations on Android, prevents the app to show up with the same features compared to iOS" (more of those voices at FL Studio Mobile reviews) 

Cross-Platform development ● VST3, LV2, and CLAP are all cross-platform (except for UI) ● Users want plugins available on multiple platforms + multiple plugin formats ○ in consistent behavior ● Plugins are (usually) native apps based on C ABI (application binary interface) 

Cross-Platform development ● VST3, LV2, and CLAP are all cross-platform (except for UI) ● Users want plugins available on multiple platforms + multiple plugin formats ○ in consistent behavior ● Plugins are (usually) native apps based on C ABI (application binary interface) ● developers tend to write cross-platform code for audio plugins (in C++) ○ using JUCE, DPF, iPlug2 etc. -or- using clap-wrapper ("CLAP first") ● many of existing plugin sources are cross-platform code using ^ Trend 

Building Audio Software Ecosystem on Mobiles 

Desktop vs. Mobile: totally different ecosystems Do we run desktop software on mobiles? No... 

Desktop vs. Mobile: totally different ecosystems Desktop Mobile how we run apps many windows show plugin within DAW one app on entire screen remote UI is restricted screen size big, show everything small, cannot show all load and save files usual awkward input events mouse + keyboard touches 

Desktop vs. Mobile: totally different ecosystems Desktop Mobile app installation sloppy: download from anywhere app can load third-party libs strict: usually via app stores each app has user context how plugins are used load as a library connect as a different app JIT compilation possible prohibited on iOS standard plugin formats a few options only AUv3 on iOS nothing on Android (On macOS, Mac App Store apps are similarly strict) 

What should we achieve on "mobile version" ? ● Integration (interoperable) with desktop apps ● build and run subset of desktop version ○ not to achieve the same workflow, but make project data round-trip (shareable with desktop) example: Logic Pro 

How Apple ecosystem works ● consistent API with desktop (CoreAudio, CoreMIDI, AudioToolbox etc.) ● audio plugin format exists: AudioUnit v3 ○ AUv3 works on both macOS and iOS ○ interoperable with AUv2 (desktop in-process only) : migration without rewrite ● with audio plugins, audio apps gain more features It is not loved by desktop devs, but on secure platform everyone needs to behave. 

AUv3 technical aspects Works well with iOS app model ● based on App Extension ● privileged IPC on real-time audio thread using AudioWorkgroup 

AUv3 technical aspects UI App Extension ● host can launch plugin UI ● has "compact view" that shows small UI 

AUv3 technical aspects Smooth migration for AUv2 developers ● AudioToolbox provides consistent hosting API between AUv2 and AUv3 ● AUv2 plugin can be loaded as AUv3 using AUAudioUnitV2Bridge 

Apple: tight MIDI integration MIDI is integrated from driver level (CoreMIDI) to AudioUnits. The earliest party to have adopted MIDI 2.0. Internally CoreMIDI is based on MIDI 2.0, downtranslates to 1.0 only if needed. Still not complete, but the most advanced MIDI platform. 

AAP: Audio Plugins For Android 

An experimental audio plugin format for Android Audio apps should be ubiquitously available None of existing formats works well with these constraints on Android There should be something comparable to AUv3 

Crossing app border (IPC) ● Android has fairly good Binder IPC with real-time priority inheritance ● but not exposed to framework binder domain ● there is nothing like Audio Workgroup 

GUI Options Option 1: SurfaceControlViewHost embed UI from another app (using SurfaceView) It is doable because host app instantiates them (security risk otherwise) Option 2: Web UI loaded by host WebView (there may be communication limitations) 

Reuse existing code If we create a new plugin format, will people write code for it? => Very unlikely 😥 Chicken and Egg - no plugins, then no hosts. No hosts, then no plugins Why not reuse existing plugins then! 

Reuse existing code This is how AAP achieved those bunch of plugins: ● aap-juce build JUCE plugins (and hosts) as AAP ● aap-lv2 wrap LV2 plugins (of non-JUCE plugins) as AAP In the future, CLAP wrapper is also doable (when many CLAP API based OSS plugins show up) 

Reuse existing code Not all features are imported ● LV2 UI does not support Android ● JUCE UI does not work as native UI (yet) ○ it likely needs significant rewrite of JUCE IMO there should be different UI for mobiles Ideally this should work... 

A bunch of ports ● 30+ FLOSS repos, 200+ plugins 

Some design details AAP makes full use of MIDI 2.0 UMP ● Events (note-on/off etc.) are fully based on UMP ● Parameter changes are MIDI 2.0 Assignable Controllers (NRPNs) ● Supports per-note expression and pitchbend ● Extension API is built on top of SysEx messages ● Future: state binaries and param./prog. metadata in MIDI-CI for sharing with potential MIDI-CI compatible desktop plugins AudioUnit supports UMP as well (parameters API is different) JUCE 9 (next major ver.) will support UMP in its plugin API too Trend 

Alternative Ideas WebAssembly DSP + WASM runtime + Web UI ? (assuming we can JIT compile) ● pros can run DSP in process without privilege inheritance host can be simple WebView ● cons WASM may not perform well interop between DSP and WebView in separate process can be tricky less possibility to port existing plugins (must be emscripten-compatible) 

Wrap-up 

Wrap-up Many audio apps potentially make use of audio effects Bringing in audio plugins ecosystem to mobile is hard (app process model, UI) Port existing plugin apps to build up our own plugin API ecosystem Make use of open technology to make things easier and useful