Designing Audio Plugin Framework for Android

At COSCUP 2020. https://coscup.org/2020/zh-TW/agenda/JST3EH

Transcript

1. Designing
   Audio Plugin Framework
   for Android
   atsushieno @ github, speakerdeck, google, facebook, twitter(ja)
   

   View Slide

2. How do we compose music?　　　
   

   View Slide

3. How do we compose music on PC?
   DAW
   audio plugins
   MIDI devices
   

   View Slide

4. What is Audio Plugin?
   Instrument (Synth) Effector
   

   View Slide

5. What is Audio Plugin "Framework" ?
   A common interface
   makes them interoperable
   Audio Plugin Fx
   Hosts (DAWs)
   Plugins
   

   View Slide

6. Existing Audio Plugin Frameworks
   Linux
   Windows
   Mac
   iOS
   Android
   VST
   Audio
   Unit
   LADSPA /
   LV2
   !?
   

   View Slide

7. Linux
   Windows
   Mac
   iOS
   VST
   Filling the Gap
   Audio
   Unit
   LADSPA /
   LV2
   Android !!
   AAP
   

   View Slide

8. Demo (recorded)
   (Audio plugin devs don't
   develop and test with
   DAWs. Instead, we use
   simple "host" for testing.)
   

   View Slide

9. Keys to successful audio plugin framework?
   bigger ecosystem
   API/ABI stability and extensibility
   Meet audio plugin requirements
   

   View Slide

10. Keys to successful audio plugin framework?
    bigger ecosystem
    API/ABI stability and extensibility
    Meet realtime audio equirements
    

    View Slide

11. Know how plugins are developed
    01
    02
    Audio processing (DSP)
    Design Plugin
    03
    04
    Plugin UI
    Package Plugin Bundles
    How the plugin works ?
    (especially regarding user inputs)
    

    View Slide

12. Know how plugins are developed
    it is mostly about byte (float) array processing
    01
    02
    Audio processing (DSP)
    Design Plugin
    03
    04
    Plugin UI
    Package Plugin Bundles
    void applyPlugin(
    vector midiMessages,
    vector audioInputs,
    vector audioOutputs)
    {
    switch(current_program) {
    case PIANO:
    piano(midiMessages, audioOutputs);
    break;
    case GUITAR: ...
    }
    }
    

    View Slide

13. Know how plugins are developed
    01
    02
    Design Plugin
    Audio processing (DSP)
    03
    04
    Plugin UI
    Package Plugin Bundles
    cross-platform? per platform?
    Only for Win/Mac?
    

    View Slide

14. Know how plugins are developed
    01
    02
    Design Plugin
    Audio processing (DSP)
    03
    04
    Plugin UI
    Package Plugin Bundles
    VST? AU? LV2?
    

    View Slide

15. Easily import existing plugins
    VST3
    AU
    LV2
    AAP
    they often use cross-platform,
    cross-plugin frameworks... JUCE
    ...or, they can often "wrap" foreign plugins. LV2
    

    View Slide

16. Keys to successful audio plugin framework?
    bigger ecosystem
    API/ABI stability and extensibility
    Meet realtime audio requirements
    

    View Slide

17. What if we lose compatibility...
    incompatible versions =
    smaller ecosystem :(
    Audio Plugin Fx
    Hosts (DAWs)
    Plugins
    

    View Slide

18. Keys to successful audio plugin framework?
    bigger ecosystem
    API/ABI stability and extensibility
    Meet realtime audio requirements
    

    View Slide

19. Realtime Audio Requirements
    Audio processing has to be done in "realtime", in 10ms-ish
    Realtime != high performance processing. WCET (worst case exec. time) matters.
    Realtime blockers: GC, JIT, I/O, thread blocking, many system calls...
    non-RT? audio glitches, out-of-sync performance, dropouts = noise
    

    View Slide

20. Realtime Audio Requirements
    blocking syscalls for example...
    seealso: Real-time 101 videos @Audio Developers Conference 2019
    

    View Slide

21. Android-specific issues?
    DAWs cannot load
    plugins in-process
    Plugins (kinda) have to be
    distributed via Google Play
    either only DAW or only
    plugin app can show the UI
    NO dlopen()
    App Distribution Single Activity
    

    View Slide

22. Process isolation
    Process
    Process
    Desktop in general
    Process
    Host (DAW)
    Plugin Plugin
    
    Android
    Process
    Host (DAW)
    Plugin Plugin
    
    

    View Slide

23. IPC-based plugin chains
    need to use ASharedMemory (ashmem)
    NdkBinder: native Binder API - minSdkVersion 29
    "Android IPC mechanism" by jserv
    Though it's still full of mutex...
    

    View Slide

24. UI architecture (draft stage)
    It's nicer if plugin UI can be loaded on the working DAW
    But UI code (Java or native) from the plugin app cannot be loaded on DAW
    Otherwise application/Activity has to switch around too often
    Will popup via WINDOW_MANAGER service work? (needs investigation)
    What if the plugin UI is in HTML so that WebView can load it??
    

    View Slide

25. UI architecture - web UI (idea only)
    my desktop PoC (LV2) >>>
    Web-tech based UI is also in...
    ● iPlug2 WebUI
    ● mod-ui
    ● blueprint (JUCE/React)
    ● juce_emscripten
    ongoing work anyways, issue #34
    

    View Slide

26. Further thoughts / discussions...
    Should we ever invent another plugin framework?
    Are existing code really useful on mobile?
    because mobile is too different...
    Maybe UIs aren't. Audio parts are.
    

    View Slide

27. Thanks!
    Repos
    ● android-audio-plugin-framework
    ● aap-lv2
    ● aap-juce
    

    View Slide

28. Some annoying extra but inevitable slides follow...
    

    View Slide

29. CREDITS
    This is where you give credit to the ones who are part of this project.
    ● Presentation template by Slidesgo
    ● Icons by Flaticon
    ● Infographics by Freepik
    ● Images created by Freepik
    ● Author introduction slide photo created by Freepik
    ● Text & Image slide photo created by Freepik
    

    View Slide

30. RESOURCES
    Did you like the resources on this template? Get them for free at our other websites.
    PHOTOS
    ● Young businessman with hands in his pocket talking on mobile phone
    ● Smiling ethnic woman with gray laptop
    ● Smiling young man busy in texting message to someone
    VECTORS
    ● Flat computer engineering concept
    ● Circuit of user experience
    ● Timeline infographic
    ● Logotype collection
    ● Flat social media multimedia player templa
    ● Gradient technology logo template collection
    ● Digital marketing landing page template
    ● Colorful responsive landing page
    ● Modern web design concept with hand drawn style
    ● App development concept with flat design
    

    View Slide

31. Instructions for use
    In order to use this template, you must credit Slidesgo by keeping the Credits slide.
    You are allowed to:
    - Modify this template.
    - Use it for both personal and commercial projects.
    You are not allowed to:
    - Sublicense, sell or rent any of Slidesgo Content (or a modified version of Slidesgo Content).
    - Distribute Slidesgo Content unless it has been expressly authorized by Slidesgo.
    - Include Slidesgo Content in an online or offline database or file.
    - Offer Slidesgo templates (or modified versions of Slidesgo templates) for download.
    - Acquire the copyright of Slidesgo Content.
    For more information about editing slides, please read our FAQs:
    https://slidesgo.com/faqs
    

    View Slide