Domain-Specificity, Normativity and Generality in the CSTs for Arts

Transcript

1. 2023/11/23 AIST Creative HCI Seminar #5 
 Tomoya Matsuura(me@matsuuratomoya.com) Domain-Specificity, Normativity and Generality in the CSTs for Arts through the development of programmable music software
 "otopoiesis"
2. দӜ஌໵ / Tomoya Matsuura • Art Media Center, Tokyo University of the Arts • Performance with self-made musical instrument "Exidiophone" that uses audio- feedback as a sound source • Design and Development of a programming language for music "mimium" • Research of an imaginary research fi eld 
 "Իָ౔໦޻ֶ/Civil Engineering of Music"
3. Agenda • DAW and the Programming Environment for Music • UGen: a Fundamental Model Remaining Unchanged • Alternative Abstractions of Time Functions • otopoiesis, a functional-style programmable Digital Audio Workstation
4. %"8
   BOE
   1SPHSBNNJOH
   &OWJSPOFNFOU
   GPS
   .VTJD
5. Digital Audio Workstation Avid Pro Tools from https://www.avid.com/ja/pro-tools
6. Cycling' 74 Max
7. https://plugdata.org/ plugdata, an alternative runtime of PureData, which is embeddable to DAW as a plugin https://www.ableton.com/ja/live/max-for-live/ Max for Live, Max as a Plugin in Ableton Live
8. Division of Labour in DAW and Sound Programming Note Level Manipulation (MIDI) Composition of Synth and Effect Development of Synth and Effect
9. 6(FO
   B
   'VOEBNFOUBM
   .PEFM
   3FNBJOJOH
   6ODIBOHFE
10. MUSIC and Unit Generator Concept (1963) by Max Mathews • Connecting basic units of signal processing • e.g. oscillators and fi lters • Drawing a diagram on the paper was a part of "Programming" as well as translating it to the text format and punch card • Most of major sound programming languages derives UGen concept • e.g. WebAudio API uses this concept too
11. UGen, Modular Synthesizer and Visual Thinking • UGen concept and a modular synthesizer are born mostly at the same era • Lately sound programming environment intentionally imported some concepts modular synth, like division of Control and Audio Signal (Vercoe 2012) • Demand for a realtime use in 1980s accelerated the use of UGen • Connecting a fi nite set of hardware UGen virtually on the computer, like SAMSON Box(1977) RCA Mark II Sound Synthesizer (CC0)
12. We are "directly" manipulating the visual metaphor of sound processing unit image from https://vcvrack.com/Rack
13. UGen-based programming can at best express only visually displayable complexity
14. Unit Generator cannot take a UGen itself as its data type (metaprogramming becomes imperative)
15. "Of Epistemic Tools: musical instruments as cognitive extensions", Thor Magnusson, Organized Sound 14-2, p173 (2009) "As opposed to the acoustic instrument maker, the designer of the composed digital instrument frames affordances through symbolic design, thereby creating a snapshot of musical theory, freezing musical culture in time." →DSLs for Arts, should not loose generality and extensibility
16. "MUFSOBUJWF
    "CTUSBDUJPOT
    
 PG
    5JNF
    'VODUJPOT
17. GROOVE(1970, Mathews and Moore) • Core Concept:"File System for Time Functions" • Rate: ~200Hz(Used for controlling modular synthesizer, not generating audible signal directly) →From Computational Model of Musical Instruments,
 Toward General Abstraction of Time Function
18. GROOVE(1970, Mathews and Moore) • "Many tasks now done by people are best described simply by one or more functions of time. To list only a few examples, the control of machine tools, the control of plants such as rolling mills or chemical processes, the control of the body itself, speaking, and playing music can all be characterized by a suitable set of time functions. If these tasks are to be automated, the basic job of the computer is to create the functions. Hence it seems worthwhile to develop a general program to produce time functions which are characteristic of human actions."
19. Chronic (2002, Brandt) Temporal Type Constructor • General data abstraction using 3 type constructor(function that generate new types) • event a (data with a timestamp) • vec a ( fi nite array of data) • ivec a (in fi nite array: generator which returns value of type a) • Implemented αs a library in OCaml (non- realtime)
20. Faust(2004~) delay and feedback with 1 sample can express an universal signal fl ow compose processor s with 5 primitive operator: parallel, sequential, split, merge, feedback
21. Lambda-Calculus based representation mimium(Matsuura and Jo, 2021)'s formal syntax 
 (work in progress for the new version, based on the work of W calculus(Arias et al. 2021) )
22. PUPQPJFTJT
    B
    GVODUJPOBM
    QSPHSBNNBCMF
    %"8
23. .BLF
    UIF
    EBUB
    TUSVDUVSF
    PG
    %"8
    B
    CJU
    NPSF
    BCTUSBDU
24. Track Project Contains Multiple Tracks and Mixer Project Track Track Mixer
25. Region Track Track Contains Multiple Regions Region Region Region
26. Region Region has start time, duration and and audio stream start duration 0
27. Vector of Notes(MIDI) → Audio Stream InstrumentʢSynthesizerʣ Audio Stream → Audio Stream Audio E ff ect These conversions are function Track(a) -> Track(b)
28. • Project is a function that generates sound stream from multiple tracks • Track is a function that translates audio stream from another track or multiple regions into data (sometimes non-audio) stream • Region is a function that generate some data stream, with a timestamp and duration • Track Presets (set of audio e ff ects and synths with speci fi ed parameter) are also functions • Project Templates are also functions that generate project • Freeze/Bake of Regions and Tracks is a Partial Evaluation/ Constant Folding of the Project • Maybe: Project can be embedded as a Region recursively then...
29. &YBNQMF
    "QQMZJOH
    'BEF
    UP
    3FHJPO
30. Raw Abstract Syntax Tree
 in JSON Format
31. let sinewave =|freq,amp,phase|{ ... } let apply_fadeinout = |start,dur,time_in,time_out,content|{ ... } let FadeInOut = |time_in,time_out,origin|{ let time_in = Param("time_in",0.01,0.0..inf); let time_out = Param("time_out",0.01,0.0..inf); Region({ start: origin.start dur : origin.dur content: apply_fadeinout(start,dur,time_in,time_out,origin.content) }) } let project = |sample_rate|{ let tracks = Track([ FadeInOut( 0.1, 0.1, region: { start:0.0, dur: 1.0, content: || sinewave{ Param("freq",440.0,20.0..20000.0) Param("amp",1.0,0.0..1.0) Param("phase",0.0,0.0..1.0) } } ) ]); tracks.map(|t| t.render()).sum() } • FadeInOut function takes a region, return new Region, with modi fi cation of the content (Code syntax is pseudo-code)
33. GUI for Parameters are automatically grouped by lexical scope of functions
34. Source Code (Project Generator) Playback Con fi g { Sample Rate:48000, ...} Output Sound Project Shared Memory GUI static analysis (extract parameter) Compile when source code changed that need to re-generate GUI (e.g Adding/Removing Track) Host App Sound Program
35. Audio Stream Considerable Example: Double Reverse of Region Composition of Non-linear E ff ects as a Function Application 1.Reversing Order 2.Render as Audio Audio Stream 3. Reverse Audio
36. Space for Improvement • Bidirectional programming with an inverse evaluation • Static typing for e ff i cient execution • Multi-stage computation would be nice for the purpose • Metadata/attributes for the special UI for speci fi c function • Switching an o ffl ine rendering and a realtime rendering depending on the situation • Data fl ow model between project and GUI
37. import("stdfaust.lib"); freqS = vslider("h:Oscillators/h:[0]Sawtooth/[0]freq",440,50,1000,0.1); gainS = vslider("h:Oscillators/h:[0]Sawtooth/[1]gain",0,0,1,0.01); freqT = vslider("h:Oscillators/h:[1]Triangle/[0]freq",440,50,1000,0.1); gainT = vslider("h:Oscillators/h:[1]Triangle/[1]gain",0,0,1,0.01); process = os.sawtooth(freqS)*gainS + os.triangle(freqT)*gainT; Faust can group UIs but it has di ff erent semantics on the label name
38. Wrap up • Data abstraction and its manipulation is a key of Domain-Speci fi c Languages • However, DSLs for Creativity Support Tools (esp. for Arts) de fi nes a kernel of expressions, which may causes a negative feedback loop and normalizes a diversity of expression • Challenge: keeping a space for modi fi cation by users: meta-ness of the tool, without loosing a low-barrier and an intuitive use for the beginner of programming(≒artists and musicians)? • DAW project as a composed function
39. Thank you for Listening! https://github.com/tomoyanonymous/mimium-rs Development repository of the new version of mimium compiler(written in Rust) https://github.com/mimium-org/mimium mimium's Original Repository(not actively maintained, written in C++) https://github.com/tomoyanonymous/otopoiesis otopoiesis development repository email: me@matsuuratomoya.com mastodon: social.matsuuratomoya.com/@tomoya