Share
Embed
Domain-Specificity, Normativity and Generality in the CSTs for Arts
Tomoya Matsuura
November 23, 2023
Transcript
39 slides
-
2023/11/23 AIST Creative HCI Seminar #5 Tomoya Matsuura([email protected]) Domain-Specificity,
Normativity and Generality in the CSTs for Arts through the development of programmable music software "otopoiesis" -
দӜ / Tomoya Matsuura • Art Media Center, Tokyo University
of the Arts • Performance with self-made musical instrument "Exidiophone" that uses audio- feedback as a sound source • Design and Development of a programming language for music "mimium" • Research of an imaginary research fi eld "Իֶָ/Civil Engineering of Music" -
Agenda • DAW and the Programming Environment for Music •
UGen: a Fundamental Model Remaining Unchanged • Alternative Abstractions of Time Functions • otopoiesis, a functional-style programmable Digital Audio Workstation -
https://plugdata.org/ plugdata, an alternative runtime of PureData, which is embeddable
to DAW as a plugin https://www.ableton.com/ja/live/max-for-live/ Max for Live, Max as a Plugin in Ableton Live -
Division of Labour in DAW and Sound Programming Note Level
Manipulation (MIDI) Composition of Synth and Effect Development of Synth and Effect -
MUSIC and Unit Generator Concept (1963) by Max Mathews •
Connecting basic units of signal processing • e.g. oscillators and fi lters • Drawing a diagram on the paper was a part of "Programming" as well as translating it to the text format and punch card • Most of major sound programming languages derives UGen concept • e.g. WebAudio API uses this concept too -
UGen, Modular Synthesizer and Visual Thinking • UGen concept and
a modular synthesizer are born mostly at the same era • Lately sound programming environment intentionally imported some concepts modular synth, like division of Control and Audio Signal (Vercoe 2012) • Demand for a realtime use in 1980s accelerated the use of UGen • Connecting a fi nite set of hardware UGen virtually on the computer, like SAMSON Box(1977) RCA Mark II Sound Synthesizer (CC0) -
We are "directly" manipulating the visual metaphor of sound processing
unit image from https://vcvrack.com/Rack -
Unit Generator cannot take a UGen itself as its data
type (metaprogramming becomes imperative) -
"Of Epistemic Tools: musical instruments as cognitive extensions", Thor Magnusson,
Organized Sound 14-2, p173 (2009) "As opposed to the acoustic instrument maker, the designer of the composed digital instrument frames affordances through symbolic design, thereby creating a snapshot of musical theory, freezing musical culture in time." →DSLs for Arts, should not loose generality and extensibility -
GROOVE(1970, Mathews and Moore) • Core Concept:"File System for Time
Functions" • Rate: ~200Hz(Used for controlling modular synthesizer, not generating audible signal directly) →From Computational Model of Musical Instruments, Toward General Abstraction of Time Function -
GROOVE(1970, Mathews and Moore) • "Many tasks now done by
people are best described simply by one or more functions of time. To list only a few examples, the control of machine tools, the control of plants such as rolling mills or chemical processes, the control of the body itself, speaking, and playing music can all be characterized by a suitable set of time functions. If these tasks are to be automated, the basic job of the computer is to create the functions. Hence it seems worthwhile to develop a general program to produce time functions which are characteristic of human actions." -
Chronic (2002, Brandt) Temporal Type Constructor • General data abstraction
using 3 type constructor(function that generate new types) • event a (data with a timestamp) • vec a ( fi nite array of data) • ivec a (in fi nite array: generator which returns value of type a) • Implemented αs a library in OCaml (non- realtime) -
Faust(2004~) delay and feedback with 1 sample can express an
universal signal fl ow compose processor s with 5 primitive operator: parallel, sequential, split, merge, feedback -
Lambda-Calculus based representation mimium(Matsuura and Jo, 2021)'s formal syntax
(work in progress for the new version, based on the work of W calculus(Arias et al. 2021) ) -
Region Region has start time, duration and and audio stream
start duration 0 -
Vector of Notes(MIDI) → Audio Stream InstrumentʢSynthesizerʣ Audio Stream →
Audio Stream Audio E ff ect These conversions are function Track(a) -> Track(b) -
• Project is a function that generates sound stream from
multiple tracks • Track is a function that translates audio stream from another track or multiple regions into data (sometimes non-audio) stream • Region is a function that generate some data stream, with a timestamp and duration • Track Presets (set of audio e ff ects and synths with speci fi ed parameter) are also functions • Project Templates are also functions that generate project • Freeze/Bake of Regions and Tracks is a Partial Evaluation/ Constant Folding of the Project • Maybe: Project can be embedded as a Region recursively then... -
let sinewave =|freq,amp,phase|{ ... } let apply_fadeinout = |start,dur,time_in,time_out,content|{ ...
} let FadeInOut = |time_in,time_out,origin|{ let time_in = Param("time_in",0.01,0.0..inf); let time_out = Param("time_out",0.01,0.0..inf); Region({ start: origin.start dur : origin.dur content: apply_fadeinout(start,dur,time_in,time_out,origin.content) }) } let project = |sample_rate|{ let tracks = Track([ FadeInOut( 0.1, 0.1, region: { start:0.0, dur: 1.0, content: || sinewave{ Param("freq",440.0,20.0..20000.0) Param("amp",1.0,0.0..1.0) Param("phase",0.0,0.0..1.0) } } ) ]); tracks.map(|t| t.render()).sum() } • FadeInOut function takes a region, return new Region, with modi fi cation of the content (Code syntax is pseudo-code) -
Source Code (Project Generator) Playback Con fi g { Sample
Rate:48000, ...} Output Sound Project Shared Memory GUI static analysis (extract parameter) Compile when source code changed that need to re-generate GUI (e.g Adding/Removing Track) Host App Sound Program -
Audio Stream Considerable Example: Double Reverse of Region Composition of
Non-linear E ff ects as a Function Application 1.Reversing Order 2.Render as Audio Audio Stream 3. Reverse Audio -
Space for Improvement • Bidirectional programming with an inverse evaluation
• Static typing for e ff i cient execution • Multi-stage computation would be nice for the purpose • Metadata/attributes for the special UI for speci fi c function • Switching an o ffl ine rendering and a realtime rendering depending on the situation • Data fl ow model between project and GUI -
import("stdfaust.lib"); freqS = vslider("h:Oscillators/h:[0]Sawtooth/[0]freq",440,50,1000,0.1); gainS = vslider("h:Oscillators/h:[0]Sawtooth/[1]gain",0,0,1,0.01); freqT = vslider("h:Oscillators/h:[1]Triangle/[0]freq",440,50,1000,0.1);
gainT = vslider("h:Oscillators/h:[1]Triangle/[1]gain",0,0,1,0.01); process = os.sawtooth(freqS)*gainS + os.triangle(freqT)*gainT; Faust can group UIs but it has di ff erent semantics on the label name -
Wrap up • Data abstraction and its manipulation is a
key of Domain-Speci fi c Languages • However, DSLs for Creativity Support Tools (esp. for Arts) de fi nes a kernel of expressions, which may causes a negative feedback loop and normalizes a diversity of expression • Challenge: keeping a space for modi fi cation by users: meta-ness of the tool, without loosing a low-barrier and an intuitive use for the beginner of programming(≒artists and musicians)? • DAW project as a composed function -
Thank you for Listening! https://github.com/tomoyanonymous/mimium-rs Development repository of the new
version of mimium compiler(written in Rust) https://github.com/mimium-org/mimium mimium's Original Repository(not actively maintained, written in C++) https://github.com/tomoyanonymous/otopoiesis otopoiesis development repository email: [email protected] mastodon: social.matsuuratomoya.com/@tomoya