Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings

Transcript

1. Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings

   Jaime Arias, Myriam Desainte-Catherine and Shlomo Dubnov Université de Bordeaux, LaBRI, UMR 5800 Inria - Bordeaux Sud-Ouest University of California, San Diego, CREL MuMe Workshop International Conference on Computational Creativity June 27, 2016

2. Motivation Dynamic Models of Creativity Figure: Human-(Musical) Robot Control: Information

   Exchange for Meta-Creation Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 1/20 1/20

3. Motivation Umberto Eco: The Poetics of the Open Work Figure:

   An incomplete knowledge of the system is an essential feature Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 2/20 2/20

4. Motivation Stockhausen: Klavierst￿cke XI Figure: Mobile structure and graphic layout

   in 19 fragments Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 3/20 3/20

5. Introduction Overview of the system • PyOracle: https://gitlab.com/himito/PyOracle_I-score • i-score:

   https://github.com/himito/i-score • VMO-Score: https://himito.github.io/vmo_i-score_generator Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 4/20 4/20

6. Introduction Overview of the system Pre-Recorded Audio Variable Markov Oracle

   (VMO) Audio Segmentation Audio Oracle SNAKES Petri Net Model input output output input output Configuration File output (a) Generation of the structure Petri Net Model Offline Improviser output input input Audio Synthesizer Audio File output input Audio Oracle input Petri Net Parameters Actions Oracle Parameters Performer Controls Configuration File Oracle Regions Sequence Audio Buffer input (b) Offline improvisation Figure: Improvisation Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 5/20 5/20

7. Introduction Overview of the system Pre-Recorded Audio Variable Markov Oracle

   (VMO) Audio Segmentation Audio Oracle SNAKES Petri Net Model input output output input output Configuration File output (a) Generation of the structure Petri Net Model Audio Oracle i-score Petri Net Parameters Configuration File PyOracle input input input Real-time Performer Controls input input inout Real-time Audio Output Output (b) Real-time improvisation Figure: Composition Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 6/20 6/20

8. Composition

9. Composition Audio Recording Figure: Audio recording: Philou. Shlomo Dubnov -

   University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 7/20 7/20

10. Composition PyOracle Improviser Figure: Segmentation Shlomo Dubnov - University of

    California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 8/20 8/20

11. Composition Segmentation Figure: Segmentation Shlomo Dubnov - University of California,

    San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 9/20 9/20

12. Composition Petri Net G D C t (a) The marking

    before firing the enabled transition t. G D C t (b) The marking after firing transition t, where t is disabled. Figure: Illustration of a firing rule in a Petri Net. Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 10/20 10/20

13. Composition Petri Net Figure: Segmentation end init t1 t0 t2

    t3 t4 t5 t6 t7 t8 t9 t10 t11 Figure: Timed Petri Net Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 11/20 11/20

14. Composition Parameters of the Petri Net 1 # file: configuration.yml

    2 3 conditions: 4 - transition : 't0' 5 time-min : 0.0 6 time-max : 3.0 7 condition : '/device/key == 10' 8 9 - transition : 't1' 10 ... init t0 [0.0, 3.0] /device/key == 10 Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 12/20 12/20

15. Offline Improvisation

16. Offline Improvisation Specification of actions 1 # file: configuration.yml 2

    3 actions: 4 - address : '/volume/sensor/pos_x' 5 value : 10 6 time : 250 7 8 - address : ... Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 13/20 13/20

17. Offline Improvisation Adding environment init t0 /device/key == 10 [0.0,

    3.0] Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 14/20 14/20

18. Offline Improvisation Adding environment (a) human-robot interaction init env t0

    (a,v,t) /device/key == 10 [0.0, 3.0] (b) i-score Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 15/20 15/20

19. Offline Improvisation Adding environment init env t0 (a,v,t) (a ==

    /device/key) ∧ (v == 10) ∧ (t == get_time()) [0.0, 3.0] Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 16/20 16/20

20. Offline Improvisation Adding environment init env t0 (a,v,t) te (a,v,t)

    get_time() > t (a == /device/key) ∧ (v == 10) ∧ (t == get_time()) [0.0, 3.0] Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 17/20 17/20

21. Real-time Improvisation

22. Real-time Improvisation The inter-media sequencer i-score score / Initial Transition

    T10 T6 T5 T8 T4 T2 T1 T9 T7 T11 Popolari61oreweed40 Loop pattern snozzle24ambier35 Loop pattern pronger64inurbane46 Loop pattern spouty57carthame37 Loop pattern T5 T6 T10 T1 T2 T4 T8 T11 T7 T9 T3 scissel86thermo23 Loop pattern T3 (c) i-score end init t1 t0 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 (d) Petri net Figure: i-score representation. Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 18/20 18/20

23. Real-time Improvisation The inter-media sequencer i-score score / Initial Transition

    T10 T6 T5 T8 T4 T2 T1 Popolari61oreweed40 Loop pattern snozzle24ambier35 Loop pattern pronger64inurbane46 Loop pattern T5 T6 T10 T1 T2 T4 T8 (a) i-score end init t1 t0 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 (b) Petri net Figure: i-score representation. Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 19/20 19/20

24. Concluding Remarks

25. Concluding Remarks • Music as a set of combinatorial processes

    with concurrency control • Improvisation as a stylistic combinatorial system • Open form establishes a set of conditions and actions for synchronization • Generalizes and automates existing DAW tools (session view - clip, follow...) Combinatorial systems date back to ars inventendi (Leibniz) and are the core of ars magna (great art) of the Franciscan monk Ramon Llull (1232-1316), as well as the ecstatic Kabbalah of Abraham Abulafia (1240-1291), and Renaissance philosophers such as Giordano Bruno (1548-1600). ’́One should not pay attention to the properties of terms but only to the fact that they define an order, a texture, an architecture‘̀ U. Eco, The Search for the Perfect Language, Wiley-Blackwell 1997 Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 20/20 20/20

26. Questions

27. Thank you for your attention!

28. Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings

    Jaime Arias, Myriam Desainte-Catherine and Shlomo Dubnov Université de Bordeaux, LaBRI, UMR 5800 Inria - Bordeaux Sud-Ouest University of California, San Diego, CREL MuMe Workshop International Conference on Computational Creativity June 27, 2016