Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings Jaime Arias, Myriam Desainte-Catherine and Shlomo Dubnov Université de Bordeaux, LaBRI, UMR 5800 University of California, San Diego, CREL Inria - Bordeaux Sud-Ouest Workshop On Dynamics Of Creative Improvisation July 8, 2016 

Motivation Dynamic Models of Creativity Figure: Human-(Musical) Robot Control: Information Exchange for Meta-Creation Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 1/20 1/20 

Motivation Umberto Eco: The Poetics of the Open Work Figure: An incomplete knowledge of the system is an essential feature Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 2/20 2/20 

Motivation Stockhausen: Klavierst￿cke XI Figure: Mobile structure and graphic layout in 19 fragments Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 3/20 3/20 

Introduction Overview of the system • PyOracle: https://github.com/himito/PyOracle_I-score • i-score: https://github.com/himito/i-score • VMO-Score: https://himito.github.io/vmo_i-score_generator Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 4/20 4/20 

Introduction Overview of the system Pre-Recorded Audio Variable Markov Oracle (VMO) Audio Segmentation Audio Oracle SNAKES Petri Net Model input output output input output Configuration File output (a) Generation of the structure Petri Net Model Offline Improviser output input input Audio Synthesizer Audio File output input Audio Oracle input Petri Net Parameters Actions Oracle Parameters Performer Controls Configuration File Oracle Regions Sequence Audio Buffer input (b) Offline improvisation Figure: Overview of the system Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 5/20 5/20 

Introduction Overview of the system Pre-Recorded Audio Variable Markov Oracle (VMO) Audio Segmentation Audio Oracle SNAKES Petri Net Model input output output input output Configuration File output (a) Generation of the structure Petri Net Model Audio Oracle i-score Petri Net Parameters Configuration File PyOracle input input input Real-time Performer Controls input input inout Real-time Audio Output Output (b) Real-time improvisation Figure: Overview of the system Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 6/20 6/20 

Composition 

Composition Audio Recording Figure: Audio recording: Philou. Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 7/20 7/20 

Composition PyOracle Improviser Figure: PyOracle Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 8/20 8/20 

Composition Segmentation Figure: Segmentation Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 9/20 9/20 

Composition Petri Net G D C t (a) The marking before firing the enabled transition t. G D C t (b) The marking after firing transition t, where t is disabled. Figure: Illustration of a firing rule in a Petri Net. Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 10/20 10/20 

Composition Petri Net Figure: Segmentation end init t1 t0 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 Figure: Time Petri Net Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 11/20 11/20 

Composition Parameters of the Petri Net 1 # file: configuration.yml 2 3 conditions: 4 - transition : 't0' 5 time-min : 0.0 6 time-max : 3.0 7 condition : '/device/key == 10' 8 9 - transition : 't1' 10 ... init t0 [0.0, 3.0] /device/key == 10 Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 12/20 12/20 

Offline Improvisation 

Offline Improvisation Specification of actions 1 # file: configuration.yml 2 3 actions: 4 - address : '/volume/sensor/pos_x' 5 value : 10 6 time : 250 7 8 - address : ... Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 13/20 13/20 

Offline Improvisation Adding environment init t0 /device/key == 10 [0.0, 3.0] Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 14/20 14/20 

Offline Improvisation Adding environment (a) Human-robot interaction init env t0 (a,v,t) /device/key == 10 [0.0, 3.0] (b) Petri net Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 15/20 15/20 

Offline Improvisation Adding environment init env t0 (a,v,t) (a == /device/key) ∧ (v == 10) ∧ (t == get_time()) [0.0, 3.0] Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 16/20 16/20 

Offline Improvisation Adding environment init env t0 (a,v,t) te (a,v,t) get_time() > t (a == /device/key) ∧ (v == 10) ∧ (t == get_time()) [0.0, 3.0] Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 17/20 17/20 

Real-time Improvisation 

Real-time Improvisation The inter-media sequencer i-score score / Initial Transition T10 T6 T5 T8 T4 T2 T1 T9 T7 T11 Popolari61oreweed40 Loop pattern snozzle24ambier35 Loop pattern pronger64inurbane46 Loop pattern spouty57carthame37 Loop pattern T5 T6 T10 T1 T2 T4 T8 T11 T7 T9 T3 scissel86thermo23 Loop pattern T3 (c) i-score end init t1 t0 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 (d) Petri net Figure: i-score representation. Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 18/20 18/20 

Real-time Improvisation The inter-media sequencer i-score score / Initial Transition T10 T6 T5 T8 T4 T2 T1 Popolari61oreweed40 Loop pattern snozzle24ambier35 Loop pattern pronger64inurbane46 Loop pattern T5 T6 T10 T1 T2 T4 T8 (a) i-score end init t1 t0 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 (b) Petri net Figure: i-score representation. Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 19/20 19/20 

Concluding Remarks 

Concluding Remarks • Music as a set of combinatorial processes with concurrency control • Improvisation as a stylistic combinatorial system • Open form establishes a set of conditions and actions for synchronization • Generalizes and automates existing DAW tools (session view - clip, follow...) Combinatorial systems date back to ars inventendi (Leibniz) and are the core of ars magna (great art) of the Franciscan monk Ramon Llull (1232-1316), as well as the ecstatic Kabbalah of Abraham Abulafia (1240-1291), and Renaissance philosophers such as Giordano Bruno (1548-1600). “One should not pay attention to the properties of terms but only to the fact that they define an order, a texture, an architecture” U. Eco, The Search for the Perfect Language, Wiley-Blackwell 1997 Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 20/20 20/20 

Questions 

Thank you for your attention! 

Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings Jaime Arias, Myriam Desainte-Catherine and Shlomo Dubnov Université de Bordeaux, LaBRI, UMR 5800 University of California, San Diego, CREL Inria - Bordeaux Sud-Ouest Workshop On Dynamics Of Creative Improvisation July 8, 2016