Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Designing Audio Plugin Framework for Android

Designing Audio Plugin Framework for Android

Atsushi Eno

August 02, 2020
Tweet

More Decks by Atsushi Eno

Other Decks in Technology

Transcript

  1. Designing
    Audio Plugin Framework
    for Android
    atsushieno @ github, speakerdeck, google, facebook, twitter(ja)

    View Slide

  2. How do we compose music?   

    View Slide

  3. How do we compose music on PC?
    DAW
    audio plugins
    MIDI devices

    View Slide

  4. What is Audio Plugin?
    Instrument (Synth) Effector

    View Slide

  5. What is Audio Plugin "Framework" ?
    A common interface
    makes them interoperable
    Audio Plugin Fx
    Hosts (DAWs)
    Plugins

    View Slide

  6. Existing Audio Plugin Frameworks
    Linux
    Windows
    Mac
    iOS
    Android
    VST
    Audio
    Unit
    LADSPA /
    LV2
    !?

    View Slide

  7. Linux
    Windows
    Mac
    iOS
    VST
    Filling the Gap
    Audio
    Unit
    LADSPA /
    LV2
    Android !!
    AAP

    View Slide

  8. Demo (recorded)
    (Audio plugin devs don't
    develop and test with
    DAWs. Instead, we use
    simple "host" for testing.)

    View Slide

  9. Keys to successful audio plugin framework?
    bigger ecosystem
    API/ABI stability and extensibility
    Meet audio plugin requirements

    View Slide

  10. Keys to successful audio plugin framework?
    bigger ecosystem
    API/ABI stability and extensibility
    Meet realtime audio equirements

    View Slide

  11. Know how plugins are developed
    01
    02
    Audio processing (DSP)
    Design Plugin
    03
    04
    Plugin UI
    Package Plugin Bundles
    How the plugin works ?
    (especially regarding user inputs)

    View Slide

  12. Know how plugins are developed
    it is mostly about byte (float) array processing
    01
    02
    Audio processing (DSP)
    Design Plugin
    03
    04
    Plugin UI
    Package Plugin Bundles
    void applyPlugin(
    vector midiMessages,
    vector audioInputs,
    vector audioOutputs)
    {
    switch(current_program) {
    case PIANO:
    piano(midiMessages, audioOutputs);
    break;
    case GUITAR: ...
    }
    }

    View Slide

  13. Know how plugins are developed
    01
    02
    Design Plugin
    Audio processing (DSP)
    03
    04
    Plugin UI
    Package Plugin Bundles
    cross-platform? per platform?
    Only for Win/Mac?

    View Slide

  14. Know how plugins are developed
    01
    02
    Design Plugin
    Audio processing (DSP)
    03
    04
    Plugin UI
    Package Plugin Bundles
    VST? AU? LV2?

    View Slide

  15. Easily import existing plugins
    VST3
    AU
    LV2
    AAP
    they often use cross-platform,
    cross-plugin frameworks... JUCE
    ...or, they can often "wrap" foreign plugins. LV2

    View Slide

  16. Keys to successful audio plugin framework?
    bigger ecosystem
    API/ABI stability and extensibility
    Meet realtime audio requirements

    View Slide

  17. What if we lose compatibility...
    incompatible versions =
    smaller ecosystem :(
    Audio Plugin Fx
    Hosts (DAWs)
    Plugins

    View Slide

  18. Keys to successful audio plugin framework?
    bigger ecosystem
    API/ABI stability and extensibility
    Meet realtime audio requirements

    View Slide

  19. Realtime Audio Requirements
    Audio processing has to be done in "realtime", in 10ms-ish
    Realtime != high performance processing. WCET (worst case exec. time) matters.
    Realtime blockers: GC, JIT, I/O, thread blocking, many system calls...
    non-RT? audio glitches, out-of-sync performance, dropouts = noise

    View Slide

  20. Realtime Audio Requirements
    blocking syscalls for example...
    seealso: Real-time 101 videos @Audio Developers Conference 2019

    View Slide

  21. Android-specific issues?
    DAWs cannot load
    plugins in-process
    Plugins (kinda) have to be
    distributed via Google Play
    either only DAW or only
    plugin app can show the UI
    NO dlopen()
    App Distribution Single Activity

    View Slide

  22. Process isolation
    Process
    Process
    Desktop in general
    Process
    Host (DAW)
    Plugin Plugin

    Android
    Process
    Host (DAW)
    Plugin Plugin

    View Slide

  23. IPC-based plugin chains
    need to use ASharedMemory (ashmem)
    NdkBinder: native Binder API - minSdkVersion 29
    "Android IPC mechanism" by jserv
    Though it's still full of mutex...

    View Slide

  24. UI architecture (draft stage)
    It's nicer if plugin UI can be loaded on the working DAW
    But UI code (Java or native) from the plugin app cannot be loaded on DAW
    Otherwise application/Activity has to switch around too often
    Will popup via WINDOW_MANAGER service work? (needs investigation)
    What if the plugin UI is in HTML so that WebView can load it??

    View Slide

  25. UI architecture - web UI (idea only)
    my desktop PoC (LV2) >>>
    Web-tech based UI is also in...
    ● iPlug2 WebUI
    ● mod-ui
    ● blueprint (JUCE/React)
    ● juce_emscripten
    ongoing work anyways, issue #34

    View Slide

  26. Further thoughts / discussions...
    Should we ever invent another plugin framework?
    Are existing code really useful on mobile?
    because mobile is too different...
    Maybe UIs aren't. Audio parts are.

    View Slide

  27. Thanks!
    Repos
    ● android-audio-plugin-framework
    ● aap-lv2
    ● aap-juce

    View Slide

  28. Some annoying extra but inevitable slides follow...

    View Slide

  29. CREDITS
    This is where you give credit to the ones who are part of this project.
    ● Presentation template by Slidesgo
    ● Icons by Flaticon
    ● Infographics by Freepik
    ● Images created by Freepik
    ● Author introduction slide photo created by Freepik
    ● Text & Image slide photo created by Freepik

    View Slide

  30. RESOURCES
    Did you like the resources on this template? Get them for free at our other websites.
    PHOTOS
    ● Young businessman with hands in his pocket talking on mobile phone
    ● Smiling ethnic woman with gray laptop
    ● Smiling young man busy in texting message to someone
    VECTORS
    ● Flat computer engineering concept
    ● Circuit of user experience
    ● Timeline infographic
    ● Logotype collection
    ● Flat social media multimedia player templa
    ● Gradient technology logo template collection
    ● Digital marketing landing page template
    ● Colorful responsive landing page
    ● Modern web design concept with hand drawn style
    ● App development concept with flat design

    View Slide

  31. Instructions for use
    In order to use this template, you must credit Slidesgo by keeping the Credits slide.
    You are allowed to:
    - Modify this template.
    - Use it for both personal and commercial projects.
    You are not allowed to:
    - Sublicense, sell or rent any of Slidesgo Content (or a modified version of Slidesgo Content).
    - Distribute Slidesgo Content unless it has been expressly authorized by Slidesgo.
    - Include Slidesgo Content in an online or offline database or file.
    - Offer Slidesgo templates (or modified versions of Slidesgo templates) for download.
    - Acquire the copyright of Slidesgo Content.
    For more information about editing slides, please read our FAQs:
    https://slidesgo.com/faqs

    View Slide