Designing Audio Plugin Framework for Android

At COSCUP 2020. https://coscup.org/2020/zh-TW/agenda/JST3EH

Transcript

1. Designing Audio Plugin Framework for Android atsushieno @ github, speakerdeck, google, facebook, twitter(ja)
2. How do we compose music?　　　
3. How do we compose music on PC? DAW audio plugins MIDI devices
4. What is Audio Plugin? Instrument (Synth) Effector
5. What is Audio Plugin "Framework" ? A common interface makes them interoperable Audio Plugin Fx Hosts (DAWs) Plugins
6. Existing Audio Plugin Frameworks Linux Windows Mac iOS Android VST Audio Unit LADSPA / LV2 !?
7. Linux Windows Mac iOS VST Filling the Gap Audio Unit LADSPA / LV2 Android !! AAP
8. Demo (recorded) (Audio plugin devs don't develop and test with DAWs. Instead, we use simple "host" for testing.)
9. Keys to successful audio plugin framework? bigger ecosystem API/ABI stability and extensibility Meet audio plugin requirements
10. Keys to successful audio plugin framework? bigger ecosystem API/ABI stability and extensibility Meet realtime audio equirements
11. Know how plugins are developed 01 02 Audio processing (DSP) Design Plugin 03 04 Plugin UI Package Plugin Bundles How the plugin works ? (especially regarding user inputs)
12. Know how plugins are developed it is mostly about byte (float) array processing 01 02 Audio processing (DSP) Design Plugin 03 04 Plugin UI Package Plugin Bundles void applyPlugin( vector<uint8_t> midiMessages, vector<float*> audioInputs, vector<float*> audioOutputs) { switch(current_program) { case PIANO: piano(midiMessages, audioOutputs); break; case GUITAR: ... } }
13. Know how plugins are developed 01 02 Design Plugin Audio processing (DSP) 03 04 Plugin UI Package Plugin Bundles cross-platform? per platform? Only for Win/Mac?
14. Know how plugins are developed 01 02 Design Plugin Audio processing (DSP) 03 04 Plugin UI Package Plugin Bundles VST? AU? LV2?
15. Easily import existing plugins VST3 AU LV2 AAP they often use cross-platform, cross-plugin frameworks... JUCE ...or, they can often "wrap" foreign plugins. LV2
16. Keys to successful audio plugin framework? bigger ecosystem API/ABI stability and extensibility Meet realtime audio requirements
17. What if we lose compatibility... incompatible versions = smaller ecosystem :( Audio Plugin Fx Hosts (DAWs) Plugins
18. Keys to successful audio plugin framework? bigger ecosystem API/ABI stability and extensibility Meet realtime audio requirements
19. Realtime Audio Requirements Audio processing has to be done in "realtime", in 10ms-ish Realtime != high performance processing. WCET (worst case exec. time) matters. Realtime blockers: GC, JIT, I/O, thread blocking, many system calls... non-RT? audio glitches, out-of-sync performance, dropouts = noise
20. Realtime Audio Requirements blocking syscalls for example... seealso: Real-time 101 videos @Audio Developers Conference 2019
21. Android-specific issues? DAWs cannot load plugins in-process Plugins (kinda) have to be distributed via Google Play either only DAW or only plugin app can show the UI NO dlopen() App Distribution Single Activity
22. Process isolation Process Process Desktop in general Process Host (DAW) Plugin Plugin Android Process Host (DAW) Plugin Plugin
23. IPC-based plugin chains need to use ASharedMemory (ashmem) NdkBinder: native Binder API - minSdkVersion 29 "Android IPC mechanism" by jserv Though it's still full of mutex...
24. UI architecture (draft stage) It's nicer if plugin UI can be loaded on the working DAW But UI code (Java or native) from the plugin app cannot be loaded on DAW Otherwise application/Activity has to switch around too often Will popup via WINDOW_MANAGER service work? (needs investigation) What if the plugin UI is in HTML so that WebView can load it??
25. UI architecture - web UI (idea only) my desktop PoC (LV2) >>> Web-tech based UI is also in... • iPlug2 WebUI • mod-ui • blueprint (JUCE/React) • juce_emscripten ongoing work anyways, issue #34
26. Further thoughts / discussions... Should we ever invent another plugin framework? Are existing code really useful on mobile? because mobile is too different... Maybe UIs aren't. Audio parts are.
27. Thanks! Repos • android-audio-plugin-framework • aap-lv2 • aap-juce
28. Some annoying extra but inevitable slides follow...
29. CREDITS This is where you give credit to the ones who are part of this project. • Presentation template by Slidesgo • Icons by Flaticon • Infographics by Freepik • Images created by Freepik • Author introduction slide photo created by Freepik • Text & Image slide photo created by Freepik
30. RESOURCES Did you like the resources on this template? Get them for free at our other websites. PHOTOS • Young businessman with hands in his pocket talking on mobile phone • Smiling ethnic woman with gray laptop • Smiling young man busy in texting message to someone VECTORS • Flat computer engineering concept • Circuit of user experience • Timeline infographic • Logotype collection • Flat social media multimedia player templa • Gradient technology logo template collection • Digital marketing landing page template • Colorful responsive landing page • Modern web design concept with hand drawn style • App development concept with flat design
31. Instructions for use In order to use this template, you must credit Slidesgo by keeping the Credits slide. You are allowed to: - Modify this template. - Use it for both personal and commercial projects. You are not allowed to: - Sublicense, sell or rent any of Slidesgo Content (or a modified version of Slidesgo Content). - Distribute Slidesgo Content unless it has been expressly authorized by Slidesgo. - Include Slidesgo Content in an online or offline database or file. - Offer Slidesgo templates (or modified versions of Slidesgo templates) for download. - Acquire the copyright of Slidesgo Content. For more information about editing slides, please read our FAQs: https://slidesgo.com/faqs