Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Domain-Specificity, Normativity and Generality in the CSTs for Arts

Domain-Specificity, Normativity and Generality in the CSTs for Arts

Tomoya Matsuura

November 23, 2023

More Decks by Tomoya Matsuura

Other Decks in Research


  1. 2023/11/23 AIST Creative HCI Seminar #5 
 Tomoya Matsuura([email protected]) Domain-Specificity,

    Normativity and Generality in the CSTs for Arts through the development of programmable music software
  2. দӜ஌໵ / Tomoya Matsuura • Art Media Center, Tokyo University

    of the Arts • Performance with self-made musical instrument "Exidiophone" that uses audio- feedback as a sound source • Design and Development of a programming language for music "mimium" • Research of an imaginary research fi eld 
 "Իָ౔໦޻ֶ/Civil Engineering of Music"
  3. Agenda • DAW and the Programming Environment for Music •

    UGen: a Fundamental Model Remaining Unchanged • Alternative Abstractions of Time Functions • otopoiesis, a functional-style programmable Digital Audio Workstation
  4. https://plugdata.org/ plugdata, an alternative runtime of PureData, which is embeddable

    to DAW as a plugin https://www.ableton.com/ja/live/max-for-live/ Max for Live, Max as a Plugin in Ableton Live
  5. Division of Labour in DAW and Sound Programming Note Level

    Manipulation (MIDI) Composition of Synth and Effect Development of Synth and Effect
  6. MUSIC and Unit Generator Concept (1963) by Max Mathews •

    Connecting basic units of signal processing • e.g. oscillators and fi lters • Drawing a diagram on the paper was a part of "Programming" as well as translating it to the text format and punch card • Most of major sound programming languages derives UGen concept • e.g. WebAudio API uses this concept too
  7. UGen, Modular Synthesizer and Visual Thinking • UGen concept and

    a modular synthesizer are born mostly at the same era • Lately sound programming environment intentionally imported some concepts modular synth, like division of Control and Audio Signal (Vercoe 2012) • Demand for a realtime use in 1980s accelerated the use of UGen • Connecting a fi nite set of hardware UGen virtually on the computer, like SAMSON Box(1977) RCA Mark II Sound Synthesizer (CC0)
  8. Unit Generator cannot take a UGen itself as its data

    type (metaprogramming becomes imperative)
  9. "Of Epistemic Tools: musical instruments as cognitive extensions", Thor Magnusson,

    Organized Sound 14-2, p173 (2009) "As opposed to the acoustic instrument maker, the designer of the composed digital instrument frames affordances through symbolic design, thereby creating a snapshot of musical theory, freezing musical culture in time." →DSLs for Arts, should not loose generality and extensibility
  10. GROOVE(1970, Mathews and Moore) • Core Concept:"File System for Time

    Functions" • Rate: ~200Hz(Used for controlling modular synthesizer, not generating audible signal directly) →From Computational Model of Musical Instruments,
 Toward General Abstraction of Time Function
  11. GROOVE(1970, Mathews and Moore) • "Many tasks now done by

    people are best described simply by one or more functions of time. To list only a few examples, the control of machine tools, the control of plants such as rolling mills or chemical processes, the control of the body itself, speaking, and playing music can all be characterized by a suitable set of time functions. If these tasks are to be automated, the basic job of the computer is to create the functions. Hence it seems worthwhile to develop a general program to produce time functions which are characteristic of human actions."
  12. Chronic (2002, Brandt) Temporal Type Constructor • General data abstraction

    using 3 type constructor(function that generate new types) • event a (data with a timestamp) • vec a ( fi nite array of data) • ivec a (in fi nite array: generator which returns value of type a) • Implemented αs a library in OCaml (non- realtime)
  13. Faust(2004~) delay and feedback with 1 sample can express an

    universal signal fl ow compose processor s with 5 primitive operator: parallel, sequential, split, merge, feedback
  14. Lambda-Calculus based representation mimium(Matsuura and Jo, 2021)'s formal syntax 

    (work in progress for the new version, based on the work of W calculus(Arias et al. 2021) )
  15. Vector of Notes(MIDI) → Audio Stream InstrumentʢSynthesizerʣ Audio Stream →

    Audio Stream Audio E ff ect These conversions are function Track(a) -> Track(b)
  16. • Project is a function that generates sound stream from

    multiple tracks • Track is a function that translates audio stream from another track or multiple regions into data (sometimes non-audio) stream • Region is a function that generate some data stream, with a timestamp and duration • Track Presets (set of audio e ff ects and synths with speci fi ed parameter) are also functions • Project Templates are also functions that generate project • Freeze/Bake of Regions and Tracks is a Partial Evaluation/ Constant Folding of the Project • Maybe: Project can be embedded as a Region recursively then...
  17. let sinewave =|freq,amp,phase|{ ... } let apply_fadeinout = |start,dur,time_in,time_out,content|{ ...

    } let FadeInOut = |time_in,time_out,origin|{ let time_in = Param("time_in",0.01,0.0..inf); let time_out = Param("time_out",0.01,0.0..inf); Region({ start: origin.start dur : origin.dur content: apply_fadeinout(start,dur,time_in,time_out,origin.content) }) } let project = |sample_rate|{ let tracks = Track([ FadeInOut( 0.1, 0.1, region: { start:0.0, dur: 1.0, content: || sinewave{ Param("freq",440.0,20.0..20000.0) Param("amp",1.0,0.0..1.0) Param("phase",0.0,0.0..1.0) } } ) ]); tracks.map(|t| t.render()).sum() } • FadeInOut function takes a region, return new Region, with modi fi cation of the content (Code syntax is pseudo-code)
  18. Source Code (Project Generator) Playback Con fi g { Sample

    Rate:48000, ...} Output Sound Project Shared Memory GUI static analysis (extract parameter) Compile when source code changed that need to re-generate GUI (e.g Adding/Removing Track) Host App Sound Program
  19. Audio Stream Considerable Example: Double Reverse of Region Composition of

    Non-linear E ff ects as a Function Application 1.Reversing Order 2.Render as Audio Audio Stream 3. Reverse Audio
  20. Space for Improvement • Bidirectional programming with an inverse evaluation

    • Static typing for e ff i cient execution • Multi-stage computation would be nice for the purpose • Metadata/attributes for the special UI for speci fi c function • Switching an o ffl ine rendering and a realtime rendering depending on the situation • Data fl ow model between project and GUI
  21. import("stdfaust.lib"); freqS = vslider("h:Oscillators/h:[0]Sawtooth/[0]freq",440,50,1000,0.1); gainS = vslider("h:Oscillators/h:[0]Sawtooth/[1]gain",0,0,1,0.01); freqT = vslider("h:Oscillators/h:[1]Triangle/[0]freq",440,50,1000,0.1);

    gainT = vslider("h:Oscillators/h:[1]Triangle/[1]gain",0,0,1,0.01); process = os.sawtooth(freqS)*gainS + os.triangle(freqT)*gainT; Faust can group UIs but it has di ff erent semantics on the label name
  22. Wrap up • Data abstraction and its manipulation is a

    key of Domain-Speci fi c Languages • However, DSLs for Creativity Support Tools (esp. for Arts) de fi nes a kernel of expressions, which may causes a negative feedback loop and normalizes a diversity of expression • Challenge: keeping a space for modi fi cation by users: meta-ness of the tool, without loosing a low-barrier and an intuitive use for the beginner of programming(≒artists and musicians)? • DAW project as a composed function
  23. Thank you for Listening! https://github.com/tomoyanonymous/mimium-rs Development repository of the new

    version of mimium compiler(written in Rust) https://github.com/mimium-org/mimium mimium's Original Repository(not actively maintained, written in C++) https://github.com/tomoyanonymous/otopoiesis otopoiesis development repository email: [email protected] mastodon: social.matsuuratomoya.com/@tomoya