DAW, because it's doable now ◦ There are usable open source engines (Tracktion Engine, Carla host) ◦ You can resort to AI ◦ there are many emerging DAW projects on GitHub I want to share some experiences on building next-gen. DAW engine
mobiles and Web - https://fl.studio/ for example (and they have FL Studio Mobile for years) Logic Pro for iPad (they have own goodness on continuous trackmaking on desktop and iPad)
◦ • • MCP integration: easy, just expose every feature via scripting host MIR integration (source separation etc.) - needs featureful OSS MIDI 2.0 native (VST3.8.0 etc.) document event model, for voice synthesizers (ARA2+ etc.) I need another 30+ min. slot to cover all 😅
one process (except for LV2 UI) Mobiles (AUv3, AAP) - host and plugin lives in different process Web Audio (WebCLAP) - AudioWorklet lives in its own process To cross process boundaries, we need IPC and shared memory. 🤖🍏 🐧🍎 🪟 🌐 Plugin Plugin Plugin DSP Plugin main Host Host Host DSP Host main
-> unlikely SDKs would resolve it for us: • • • AUv3: AudioToolbox and platform handle it WebCLAP: wclap-host-cpp loads DSP on AudioWorklet, and Web UI on main AAP: use AAP hosting API, JUCE API, or (pseudo) CLAP hosting API ◦ hosting library deals with Binder IPC, ASharedMemory, SurfaceControlViewHost (Native UI) and WebKit API (Web UI)
to run on audio thread == realtime-safe(!) ◦ ◦ • even if you don't care, AudioWorklet forces it (by silencing it) create and connec on main thread, swap graphs RT-safely graph nodes include: gain, stereo pan, analyzers Tracktion Engine also has a dedicated Tracktion Graph module
Their API requirement is similar ◦ • • desktop - easy (except for Linux...) AAP Native UI ◦ • • create UI, show UI, hide UI, resize UI, UI is resized (notification) ... create SurfaceView at host, use SurfaceControlViewHost to render plugin UI iOS: AudioUnit GUI WebCLAP, AAP Web UI ◦ create WebView, add JavaScript shim, and load plugin Web UI
For x-plat / multi-format architecture, we have to commonize features a lot. We also need some features flexible and product-neutral enough. As we add more features, code gets complicated ◦ especially AI slop quickly goes unmaintainable. Features affect data format and its backward compatibility.
playback engine clip positioning constraints ◦ • is there any standard for timeline representation? audio warps (time-stretch points) ◦ can this be standard? • DAG (directional acyclic graph) ◦ ◦ • Volume slider = Gain node ◦ • is it part of graph API? latency compensation ◦ • do we make it an enhanced feature? can there be only one graph model? it depends on graph track freezing ◦ ◦ tail length should be reliable should Gain change unfreeze it?
custom audio graph nodes ◦ ◦ What kind of nodes should exist? ▪ gain (volume), channel splitter / merger?, analyzer? ... How is it implemented / what to extend? ▪ graph node API: audio processing, add/remove, disable/enable ▪ data extensibility : save/load them on the project
track freezer functionality ◦ ◦ How freezing of a track should happen? ▪ We cannot use one instance for both playback and offline processing. How is it implemented / what to extend? ▪ we implement it as in "freeze track only when playback is stopped" approach. ▪ audio processor extension : skip audio processing with pre-processed buffer ▪ playback event listener : start freezing when stopped or paused
beyond SMF? • audio tracks ◦ ◦ • audio files: filename, or file stream if we bundle it, with file/mime type? positional time stretches (audio warps) instrument tracks ◦ ◦ the most common structure is still MIDI MIDI 2.0 clip makes better sense (32-bit parameters, modern features e.g. note expressions)
multi-platform DAW engines • • • • • • plugin formats on multiple platforms process isolation (host vs. plugin, audio vs. main) GUI model differences parameters consistency generalize things, yet keep open and extensible use common data format with additional extensions in compatible manner code: https://github.com/atsushieno/uapmd
clips, SMF2 Container-like project data) - - upcoming JUCE AudioProcessor v2 VST 3.8.0: IMidiMapping2, IMidiLearn2 SMF2 Container (music file format) to be released in 2026 Q3 (according to MIDI Association) which is a "true" common standard If you want to build a DAW at that level, you cannot use JUCE (AP v1) - no JUCE means no Tracktion Engine you can use atsushieno/uapmd 😎 seealso: my slides at COSCUP 2025
◦ • • create UI, show UI, hide UI, resize UI, UI is resized (notification) ... Windows, macOS (and iOS): easy Linux: each plugin format has to deal with event loop... ◦ ◦ X11 ▪ VST3, CLAP: you will have to deal with FD Wayland - not ready yet ▪ VST3.8.0: create surface but people don't like it... ▪ LV2: create subsurface instead? but that is ABI unstable #70 ▪ CLAP: in-between #474
of truth (SSOT) differs by each plugin format (namely VST3)... ◦ BUT host should always behave as the SSOT for the best consistency. Parameter list and definitions can dynamically change (Kontakt, BYOD, ysfx...) Default values are unreliable; a value might already change after initialization.
plugin: ◦ ◦ • format, id, states, potentially other properties (but you don't need it) MIDI mappings? graph: can we formalize? ◦ ◦ how are nodes connected? any graph nodes other than plugins? Gain? Pan? Are they serialized? Define them in separate resources so that we can standardize later.
host and externals (Scripts, MCP) They need some editable document model. - ARA is likely the closest concept (fairly good abstraction). track/clip added, clip range changed, track/clip deleted, note added etc. We should probably design something like Language Server Protocol here. (to not demand "implement ARA C API" on Web Audio DAWs)
atsushieno
.jpg){kind=avatar}
cargoodrich
mfonobong
dugsong
inesmontani
soysaucechin
kevinliebholz
mza
trallard
marcoroth
jakevdp
marktimemedia
lemiorhan
