Designing Audio Plugin Framework for Android

Designing Audio Plugin Framework for Android

24837993455f54c957883ba1f1db7f2d?s=128

Atsushi Eno

August 02, 2020
Tweet

Transcript

  1. Designing Audio Plugin Framework for Android atsushieno @ github, speakerdeck,

    google, facebook, twitter(ja)
  2. How do we compose music?   

  3. How do we compose music on PC? DAW audio plugins

    MIDI devices
  4. What is Audio Plugin? Instrument (Synth) Effector

  5. What is Audio Plugin "Framework" ? A common interface makes

    them interoperable Audio Plugin Fx Hosts (DAWs) Plugins
  6. Existing Audio Plugin Frameworks Linux Windows Mac iOS Android VST

    Audio Unit LADSPA / LV2 !?
  7. Linux Windows Mac iOS VST Filling the Gap Audio Unit

    LADSPA / LV2 Android !! AAP
  8. Demo (recorded) (Audio plugin devs don't develop and test with

    DAWs. Instead, we use simple "host" for testing.)
  9. Keys to successful audio plugin framework? bigger ecosystem API/ABI stability

    and extensibility Meet audio plugin requirements
  10. Keys to successful audio plugin framework? bigger ecosystem API/ABI stability

    and extensibility Meet realtime audio equirements
  11. Know how plugins are developed 01 02 Audio processing (DSP)

    Design Plugin 03 04 Plugin UI Package Plugin Bundles How the plugin works ? (especially regarding user inputs)
  12. Know how plugins are developed it is mostly about byte

    (float) array processing 01 02 Audio processing (DSP) Design Plugin 03 04 Plugin UI Package Plugin Bundles void applyPlugin( vector<uint8_t> midiMessages, vector<float*> audioInputs, vector<float*> audioOutputs) { switch(current_program) { case PIANO: piano(midiMessages, audioOutputs); break; case GUITAR: ... } }
  13. Know how plugins are developed 01 02 Design Plugin Audio

    processing (DSP) 03 04 Plugin UI Package Plugin Bundles cross-platform? per platform? Only for Win/Mac?
  14. Know how plugins are developed 01 02 Design Plugin Audio

    processing (DSP) 03 04 Plugin UI Package Plugin Bundles VST? AU? LV2?
  15. Easily import existing plugins VST3 AU LV2 AAP they often

    use cross-platform, cross-plugin frameworks... JUCE ...or, they can often "wrap" foreign plugins. LV2
  16. Keys to successful audio plugin framework? bigger ecosystem API/ABI stability

    and extensibility Meet realtime audio requirements
  17. What if we lose compatibility... incompatible versions = smaller ecosystem

    :( Audio Plugin Fx Hosts (DAWs) Plugins
  18. Keys to successful audio plugin framework? bigger ecosystem API/ABI stability

    and extensibility Meet realtime audio requirements
  19. Realtime Audio Requirements Audio processing has to be done in

    "realtime", in 10ms-ish Realtime != high performance processing. WCET (worst case exec. time) matters. Realtime blockers: GC, JIT, I/O, thread blocking, many system calls... non-RT? audio glitches, out-of-sync performance, dropouts = noise
  20. Realtime Audio Requirements blocking syscalls for example... seealso: Real-time 101

    videos @Audio Developers Conference 2019
  21. Android-specific issues? DAWs cannot load plugins in-process Plugins (kinda) have

    to be distributed via Google Play either only DAW or only plugin app can show the UI NO dlopen() App Distribution Single Activity
  22. Process isolation Process Process Desktop in general Process Host (DAW)

    Plugin Plugin Android Process Host (DAW) Plugin Plugin
  23. IPC-based plugin chains need to use ASharedMemory (ashmem) NdkBinder: native

    Binder API - minSdkVersion 29 "Android IPC mechanism" by jserv Though it's still full of mutex...
  24. UI architecture (draft stage) It's nicer if plugin UI can

    be loaded on the working DAW But UI code (Java or native) from the plugin app cannot be loaded on DAW Otherwise application/Activity has to switch around too often Will popup via WINDOW_MANAGER service work? (needs investigation) What if the plugin UI is in HTML so that WebView can load it??
  25. UI architecture - web UI (idea only) my desktop PoC

    (LV2) >>> Web-tech based UI is also in... • iPlug2 WebUI • mod-ui • blueprint (JUCE/React) • juce_emscripten ongoing work anyways, issue #34
  26. Further thoughts / discussions... Should we ever invent another plugin

    framework? Are existing code really useful on mobile? because mobile is too different... Maybe UIs aren't. Audio parts are.
  27. Thanks! Repos • android-audio-plugin-framework • aap-lv2 • aap-juce

  28. Some annoying extra but inevitable slides follow...

  29. CREDITS This is where you give credit to the ones

    who are part of this project. • Presentation template by Slidesgo • Icons by Flaticon • Infographics by Freepik • Images created by Freepik • Author introduction slide photo created by Freepik • Text & Image slide photo created by Freepik
  30. RESOURCES Did you like the resources on this template? Get

    them for free at our other websites. PHOTOS • Young businessman with hands in his pocket talking on mobile phone • Smiling ethnic woman with gray laptop • Smiling young man busy in texting message to someone VECTORS • Flat computer engineering concept • Circuit of user experience • Timeline infographic • Logotype collection • Flat social media multimedia player templa • Gradient technology logo template collection • Digital marketing landing page template • Colorful responsive landing page • Modern web design concept with hand drawn style • App development concept with flat design
  31. Instructions for use In order to use this template, you

    must credit Slidesgo by keeping the Credits slide. You are allowed to: - Modify this template. - Use it for both personal and commercial projects. You are not allowed to: - Sublicense, sell or rent any of Slidesgo Content (or a modified version of Slidesgo Content). - Distribute Slidesgo Content unless it has been expressly authorized by Slidesgo. - Include Slidesgo Content in an online or offline database or file. - Offer Slidesgo templates (or modified versions of Slidesgo templates) for download. - Acquire the copyright of Slidesgo Content. For more information about editing slides, please read our FAQs: https://slidesgo.com/faqs