Can we build better music software ecosystems for more generative era?

COSCUP 2025 https://coscup.org/2025/sessions/YJURHJ

Transcript

1. Can we build better music software ecosystems for more generative era? atsushieno
2. Audio application "ecosystem" • We use DAWs and plugins (instruments and effects) in general • To "play" instruments We use something like MIDI (or exactly MIDI)
3. The primary topic in this session • Can we bring our DAW projects to my new PC? ◦ between different platforms? ◦ can we maintain modern music production quality? ◦ on some sustainable ecosystem? The answer should become YES✊
4. Some backgrounds on modern music production on PCs • Plugin formats: VST3, AudioUnit, LV2, CLAP, etc. ◦ none of them are ideal (yet) • DAW project files are not portable ◦ and DAWs aren't solely faulty • At the moment, we only share the rendered production (mp3, flac, ogg, etc.) There should be more generative music software ecosystem mp3, ogg, flac mp3, ogg, flac mp3, ogg, flac project files project files Not Useful
5. Generative? "The Generative Internet" by Jonathan Zittrain (2006) (in short) "Apple II was generative and you could run whatever you want. iPhone is the opposite" shareable (among track-makers and listeners), adaptable (portable), user can decide what to use The Internet: we can read HTML and learn how to reproduce it = "generative" Can we share and let others learn how to produce music in data? (not in YouTube how-to videos) like we used to do with MIDI files
6. remain productive, expressive, and usable in modern ways • we don't want to go back to traditional MIDI 1.0 music files (SMF) ◦ MIDI was meant for instrument-agnostic representations • we should keep consistent instrumental experience wherever possible (as long as the same instruments are available)
7. Licenses: who has your Master Switch? Do you need other folks' permission to play/edit/export your own music? • DAWs • plugins • music app SDKs / libs • plugin formats / SDKs case study: VST2SDK (proprietary) • was killed by Steinberg around 2018 • only those who had the SDK beforehand can distribute VST2 plugins nowadays Homework: what are the implications on those SDKs and libraries?: VST3SDK, JUCE, CLAP, clap-wrapper, DPF
8. Audio Plugin API differences and problems
9. There are many audio plugin formats but they share a lot of functionalities: • scanning installed plugins • audio I/O and event I/O (MIDI alike) • parameters • states and presets • show and hide GUI inside a DAW Since they are similar, we have multi-format solutions like DPF, iPlug2, and JUCE Something like Linux LSB could help... maybe? Can't audio plugins be a "unified" solution there? LV2 CLAP VST3 AudioUnit
10. How are those plugin formats different? I can explain only some tip of those today... Preceding researches, discussion materials, etc. • GH: JeffMcClintock/GMPI - original dated in (2005) • GH: angushewlett/plugnpain "Questions you should ask of any proposed audio plug-in API" (ADC23, 2023/11) • YT: A Critique of Audio Plug-In Formats - VST, AU, AAX, JUCE and Beyond (ADC24, 2024/11) • the Ideal Audio Plugin Format (mine, Japanese) (技術書典17, 2024/10)
11. slow plugin scanning DAW needs to scan the installed plugins first. It can take like ~30 minutes󰷺 But in fact, good plugin formats can scan them quite fast. • VST < 3.7󰷹 and CLAP󰻶 need to load every plugin code to get plugin name etc. ◦ VST3.7 *with moduleinfo.json* can be scanned quickly 👍 • AudioUnit👍 and LV2👍 can just inspect metadata. ◦ Some plugin host code loads AU just like VST3 and CLAP == slow (e.g. JUCE)
12. sample-accurate events in processing audio • audio processing is done in tight time limitation like < 10 msec. • event inputs (note on/off, param changes, etc.) can't be applied immediately. • in almost all formats, events are timestamped👍 ("sample-accurate"), but traditionally they were not (e.g. VST2) • traditional cross-platform plugin SDKs like JUCE were based on VST2 and do not support timestamps󰻶 ◦ In JUCE vFuture (v9?) events will become sample-accurate.
13. should we "map" MIDI events in the plugin (format) ? • CLAP, AudioUnit, and LV2 accept MIDI 1.0 inputs, but VST3 does not. ◦ CLAP and AU accept MIDI 2.0 inputs too. • VST3: map MIDI events to VST3 events. But aren't MIDI per channel...? ◦ (128 + x) * 16ch. extra parameters (CC / program / ch-aftertouch / pitch bend) ◦ They could not generate outputs until VST3.6.12 ("Legacy" MIDI CC Out Event) ◦ Program changes were not regarded as implementable until 2019-ish. ◦ Steinberg actually worked on the solutions, but still complex compared to simple solution.
14. are states portable across machines and platforms? • Do they contain GUI states? ◦ VST3 and CLAP - it may contain UI states, hence requires the UI thread ◦ AU and LV2 are fine - can be saved and loaded on any thread • Can the state be portable across devices? ◦ GUI states => unlikely portable ◦ plugin-format-specific data? => not portable • Can you relativize file paths in the states? ◦ LV2 has decent support for making states loadable on different platforms
15. miscellaneous design consideration issues • are parameter IDs backward-compatible? (Apple) • are parameters applicable to channel, key, and note too? ◦ LV2 - not at all, MIDI 2.0 - channel and key, VST/AU/CLAP - all of them ◦ CLAP parameter defs are consistent for all, while they might differ on VST / AU / MIDI 2.0 • difference in plugin lifecycles - instantiation requires sample rate? (LV2) • can there be multiple event buses? (VST3, LV2) ◦ they can cause "undefined behavior" depending on queue processing order • are there flexible parameter types? care about RT safety? (LV2) • can DAWs pass automation data to plugins? ◦ AU can pass full data, but no curve type ◦ VST3 and CLAP can pass just automation read and/or write states, may be enough
16. Can we do any better?
17. build an "über" audio plugin format / SDK? Can we build "the only one" audio plugin format, and replace everything else? => Politically unlikely. (VST3 on Logic Pro, CLAP on Cubase, LV2 on Bitwig ?) Can we commonize those plugin format diffs and build "format-neutral" API? => Definitely, have a look at JUCE, iPlug2, DPF, etc. The actual question here should be "how much of those?" Also, many of those frameworks use MIDI 1.0 events which are not expressive.
18. Isn't new plugin format API / SDK considered harmful? Ignore "there is N+1 competing standards" problem, because it matters less. • case study: clap-wrapper ◦ write plugin code using CLAP API ◦ use crap-wrapper to build VST3 and AudioUnit ◦ same goes for plugin SDKs • using host as a plugin in a supported format ◦ wrap plugins (in unsupported formats) by DAW ◦ Ildaeil, Plugin Buddy ◦ yabridge, lv2vst, LinVst Those audio devs didn't complain when CLAP emerged.
19. Aren't there common music project file formats? • MusicXML? ◦ some instrument products use it. ◦ Not likely for general music (audio) file production. • Bitwig/dawproject? ◦ the Bitwig LV2 problem hits here too. No Linux DAW supports it. ◦ even file formats in OSS DAW engines e.g. Tracktion Engine work better because they have working reference implementations Maybe SMF2 container format would resolve most of the issues (except audio tracks)
20. Can MIDI 2.0 represent those audio plugin stuff? Not perfectly, but a lot: • parameters => assignable controllers, note expressions => per-note AC • parameter infos => MIDI-CI AllCtrlList / ChCtrlList properties • presets => program changes + MIDI-CI ProgramList properties • states => MIDI-CI Get and Set Device State What can we NOT do? • audio samples (but they are not really platform dependent) • song data package (SMF2 container format will be the savior?) • some details (note-id beyond key, 64-bit parameters) All those DAW vendors exporting/importing SMF2 is expect-able
21. idea: Authorized plugins for generative criteria • We can have an "authorized" list of plugins ◦ The ADC24 talk (linker earlier) suggests that some external entity can provide such a list • We could formalize "generative" plugin conformance levels, and tell plugin developers to follow the guidelines, or judge them by us plugin users. • Create a curated list of ("whitelisted") plugins that are safe to use for composition and sharing across platforms. • Something like GMPI, but can be implemented over existing plugin formats.
22. Thanks! Questions and feedbacks are welcome at • https://g0v.social/@atsushieno (Mastodon) • https://bsky.app/profile/atsushieno.bsky.social (ATproto) • https://github.com/atsushieno