Slide 1

Slide 1 text

2023/11/23 AIST Creative HCI Seminar #5 
 Tomoya Matsuura([email protected]) Domain-Specificity, Normativity and Generality in the CSTs for Arts through the development of programmable music software
 "otopoiesis"

Slide 2

Slide 2 text

দӜ஌໵ / Tomoya Matsuura • Art Media Center, Tokyo University of the Arts • Performance with self-made musical instrument "Exidiophone" that uses audio- feedback as a sound source • Design and Development of a programming language for music "mimium" • Research of an imaginary research fi eld 
 "Իָ౔໦޻ֶ/Civil Engineering of Music"

Slide 3

Slide 3 text

Agenda • DAW and the Programming Environment for Music • UGen: a Fundamental Model Remaining Unchanged • Alternative Abstractions of Time Functions • otopoiesis, a functional-style programmable Digital Audio Workstation

Slide 4

Slide 4 text

%"8BOE1SPHSBNNJOH &OWJSPOFNFOUGPS.VTJD

Slide 5

Slide 5 text

Digital Audio Workstation Avid Pro Tools from https://www.avid.com/ja/pro-tools

Slide 6

Slide 6 text

Cycling' 74 Max

Slide 7

Slide 7 text

https://plugdata.org/ plugdata, an alternative runtime of PureData, which is embeddable to DAW as a plugin https://www.ableton.com/ja/live/max-for-live/ Max for Live, Max as a Plugin in Ableton Live

Slide 8

Slide 8 text

Division of Labour in DAW and Sound Programming Note Level Manipulation (MIDI) Composition of Synth and Effect Development of Synth and Effect

Slide 9

Slide 9 text

6(FOB'VOEBNFOUBM.PEFM 3FNBJOJOH6ODIBOHFE

Slide 10

Slide 10 text

MUSIC and Unit Generator Concept (1963) by Max Mathews • Connecting basic units of signal processing • e.g. oscillators and fi lters • Drawing a diagram on the paper was a part of "Programming" as well as translating it to the text format and punch card • Most of major sound programming languages derives UGen concept • e.g. WebAudio API uses this concept too

Slide 11

Slide 11 text

UGen, Modular Synthesizer and Visual Thinking • UGen concept and a modular synthesizer are born mostly at the same era • Lately sound programming environment intentionally imported some concepts modular synth, like division of Control and Audio Signal (Vercoe 2012) • Demand for a realtime use in 1980s accelerated the use of UGen • Connecting a fi nite set of hardware UGen virtually on the computer, like SAMSON Box(1977) RCA Mark II Sound Synthesizer (CC0)

Slide 12

Slide 12 text

We are "directly" manipulating the visual metaphor of sound processing unit image from https://vcvrack.com/Rack

Slide 13

Slide 13 text

UGen-based programming can at best express only visually displayable complexity

Slide 14

Slide 14 text

Unit Generator cannot take a UGen itself as its data type (metaprogramming becomes imperative)

Slide 15

Slide 15 text

"Of Epistemic Tools: musical instruments as cognitive extensions", Thor Magnusson, Organized Sound 14-2, p173 (2009) "As opposed to the acoustic instrument maker, the designer of the composed digital instrument frames affordances through symbolic design, thereby creating a snapshot of musical theory, freezing musical culture in time." →DSLs for Arts, should not loose generality and extensibility

Slide 16

Slide 16 text

"MUFSOBUJWF"CTUSBDUJPOT 
 PG5JNF'VODUJPOT

Slide 17

Slide 17 text

GROOVE(1970, Mathews and Moore) • Core Concept:"File System for Time Functions" • Rate: ~200Hz(Used for controlling modular synthesizer, not generating audible signal directly) →From Computational Model of Musical Instruments,
 Toward General Abstraction of Time Function

Slide 18

Slide 18 text

GROOVE(1970, Mathews and Moore) • "Many tasks now done by people are best described simply by one or more functions of time. To list only a few examples, the control of machine tools, the control of plants such as rolling mills or chemical processes, the control of the body itself, speaking, and playing music can all be characterized by a suitable set of time functions. If these tasks are to be automated, the basic job of the computer is to create the functions. Hence it seems worthwhile to develop a general program to produce time functions which are characteristic of human actions."

Slide 19

Slide 19 text

Chronic (2002, Brandt) Temporal Type Constructor • General data abstraction using 3 type constructor(function that generate new types) • event a (data with a timestamp) • vec a ( fi nite array of data) • ivec a (in fi nite array: generator which returns value of type a) • Implemented αs a library in OCaml (non- realtime)

Slide 20

Slide 20 text

Faust(2004~) delay and feedback with 1 sample can express an universal signal fl ow compose processor s with 5 primitive operator: parallel, sequential, split, merge, feedback

Slide 21

Slide 21 text

Lambda-Calculus based representation mimium(Matsuura and Jo, 2021)'s formal syntax 
 (work in progress for the new version, based on the work of W calculus(Arias et al. 2021) )

Slide 22

Slide 22 text

PUPQPJFTJTBGVODUJPOBM QSPHSBNNBCMF%"8

Slide 23

Slide 23 text

.BLFUIFEBUBTUSVDUVSFPG%"8BCJUNPSFBCTUSBDU

Slide 24

Slide 24 text

Track Project Contains Multiple Tracks and Mixer Project Track Track Mixer

Slide 25

Slide 25 text

Region Track Track Contains Multiple Regions Region Region Region

Slide 26

Slide 26 text

Region Region has start time, duration and and audio stream start duration 0

Slide 27

Slide 27 text

Vector of Notes(MIDI) → Audio Stream InstrumentʢSynthesizerʣ Audio Stream → Audio Stream Audio E ff ect These conversions are function Track(a) -> Track(b)

Slide 28

Slide 28 text

• Project is a function that generates sound stream from multiple tracks • Track is a function that translates audio stream from another track or multiple regions into data (sometimes non-audio) stream • Region is a function that generate some data stream, with a timestamp and duration • Track Presets (set of audio e ff ects and synths with speci fi ed parameter) are also functions • Project Templates are also functions that generate project • Freeze/Bake of Regions and Tracks is a Partial Evaluation/ Constant Folding of the Project • Maybe: Project can be embedded as a Region recursively then...

Slide 29

Slide 29 text

&YBNQMF"QQMZJOH'BEFUP3FHJPO

Slide 30

Slide 30 text

Raw Abstract Syntax Tree
 in JSON Format

Slide 31

Slide 31 text

let sinewave =|freq,amp,phase|{ ... } let apply_fadeinout = |start,dur,time_in,time_out,content|{ ... } let FadeInOut = |time_in,time_out,origin|{ let time_in = Param("time_in",0.01,0.0..inf); let time_out = Param("time_out",0.01,0.0..inf); Region({ start: origin.start dur : origin.dur content: apply_fadeinout(start,dur,time_in,time_out,origin.content) }) } let project = |sample_rate|{ let tracks = Track([ FadeInOut( 0.1, 0.1, region: { start:0.0, dur: 1.0, content: || sinewave{ Param("freq",440.0,20.0..20000.0) Param("amp",1.0,0.0..1.0) Param("phase",0.0,0.0..1.0) } } ) ]); tracks.map(|t| t.render()).sum() } • FadeInOut function takes a region, return new Region, with modi fi cation of the content (Code syntax is pseudo-code)

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

GUI for Parameters are automatically grouped by lexical scope of functions

Slide 34

Slide 34 text

Source Code (Project Generator) Playback Con fi g { Sample Rate:48000, ...} Output Sound Project Shared Memory GUI static analysis (extract parameter) Compile when source code changed that need to re-generate GUI (e.g Adding/Removing Track) Host App Sound Program

Slide 35

Slide 35 text

Audio Stream Considerable Example: Double Reverse of Region Composition of Non-linear E ff ects as a Function Application 1.Reversing Order 2.Render as Audio Audio Stream 3. Reverse Audio

Slide 36

Slide 36 text

Space for Improvement • Bidirectional programming with an inverse evaluation • Static typing for e ff i cient execution • Multi-stage computation would be nice for the purpose • Metadata/attributes for the special UI for speci fi c function • Switching an o ffl ine rendering and a realtime rendering depending on the situation • Data fl ow model between project and GUI

Slide 37

Slide 37 text

import("stdfaust.lib"); freqS = vslider("h:Oscillators/h:[0]Sawtooth/[0]freq",440,50,1000,0.1); gainS = vslider("h:Oscillators/h:[0]Sawtooth/[1]gain",0,0,1,0.01); freqT = vslider("h:Oscillators/h:[1]Triangle/[0]freq",440,50,1000,0.1); gainT = vslider("h:Oscillators/h:[1]Triangle/[1]gain",0,0,1,0.01); process = os.sawtooth(freqS)*gainS + os.triangle(freqT)*gainT; Faust can group UIs but it has di ff erent semantics on the label name

Slide 38

Slide 38 text

Wrap up • Data abstraction and its manipulation is a key of Domain-Speci fi c Languages • However, DSLs for Creativity Support Tools (esp. for Arts) de fi nes a kernel of expressions, which may causes a negative feedback loop and normalizes a diversity of expression • Challenge: keeping a space for modi fi cation by users: meta-ness of the tool, without loosing a low-barrier and an intuitive use for the beginner of programming(≒artists and musicians)? • DAW project as a composed function

Slide 39

Slide 39 text

Thank you for Listening! https://github.com/tomoyanonymous/mimium-rs Development repository of the new version of mimium compiler(written in Rust) https://github.com/mimium-org/mimium mimium's Original Repository(not actively maintained, written in C++) https://github.com/tomoyanonymous/otopoiesis otopoiesis development repository email: [email protected] mastodon: social.matsuuratomoya.com/@tomoya