Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings

Transcript

1. Automatic Construction of Interactive Machine
   Improvisation Scenarios from Audio Recordings
   Jaime Arias, Myriam Desainte-Catherine and Shlomo Dubnov
   Université de Bordeaux, LaBRI, UMR 5800
   Inria - Bordeaux Sud-Ouest
   University of California, San Diego, CREL
   MuMe Workshop
   International Conference on Computational Creativity
   June 27, 2016
   

   View Slide

2. Motivation
   Dynamic Models of Creativity
   Figure: Human-(Musical) Robot Control: Information Exchange for Meta-Creation
   Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 1/20
   1/20
   

   View Slide

3. Motivation
   Umberto Eco: The Poetics of the Open Work
   Figure: An incomplete knowledge of the system is an essential feature
   Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 2/20
   2/20
   

   View Slide

4. Motivation
   Stockhausen: Klavierstcke XI
   Figure: Mobile structure and graphic layout in 19 fragments
   Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 3/20
   3/20
   

   View Slide

5. Introduction
   Overview of the system
   • PyOracle: https://gitlab.com/himito/PyOracle_I-score
   • i-score: https://github.com/himito/i-score
   • VMO-Score: https://himito.github.io/vmo_i-score_generator
   Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 4/20
   4/20
   

   View Slide

6. Introduction
   Overview of the system
   Pre-Recorded
   Audio
   Variable Markov
   Oracle (VMO)
   Audio
   Segmentation
   Audio
   Oracle
   SNAKES
   Petri Net Model
   input
   output
   output
   input
   output
   Configuration
   File
   output
   (a) Generation of the structure
   Petri Net Model
   Offline
   Improviser
   output
   input
   input
   Audio
   Synthesizer
   Audio File
   output
   input
   Audio Oracle
   input
   Petri Net
   Parameters
   Actions
   Oracle
   Parameters
   Performer Controls
   Configuration File
   Oracle Regions
   Sequence
   Audio
   Buffer
   input
   (b) Offline improvisation
   Figure: Improvisation
   Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 5/20
   5/20
   

   View Slide

7. Introduction
   Overview of the system
   Pre-Recorded
   Audio
   Variable Markov
   Oracle (VMO)
   Audio
   Segmentation
   Audio
   Oracle
   SNAKES
   Petri Net Model
   input
   output
   output
   input
   output
   Configuration
   File
   output
   (a) Generation of the structure
   Petri Net
   Model
   Audio
   Oracle
   i-score
   Petri Net
   Parameters
   Configuration File
   PyOracle
   input
   input
   input
   Real-time
   Performer Controls
   input
   input
   inout
   Real-time Audio
   Output
   Output
   (b) Real-time improvisation
   Figure: Composition
   Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 6/20
   6/20
   

   View Slide

8. Composition
   

   View Slide

9. Composition
   Audio Recording
   Figure: Audio recording: Philou.
   Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 7/20
   7/20
   

   View Slide

10. Composition
    PyOracle Improviser
    Figure: Segmentation
    Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 8/20
    8/20
    

    View Slide

11. Composition
    Segmentation
    Figure: Segmentation
    Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 9/20
    9/20
    

    View Slide

12. Composition
    Petri Net
    G
    D
    C
    t
    (a) The marking before firing the enabled
    transition t.
    G
    D
    C
    t
    (b) The marking after firing transition t,
    where t is disabled.
    Figure: Illustration of a firing rule in a Petri Net.
    Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 10/20
    10/20
    

    View Slide

13. Composition
    Petri Net
    Figure: Segmentation
    end
    init
    t1
    t0
    t2
    t3
    t4 t5
    t6
    t7
    t8
    t9
    t10
    t11
    Figure: Timed Petri Net
    Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 11/20
    11/20
    

    View Slide

14. Composition
    Parameters of the Petri Net
    1
    # file: configuration.yml
    2
    3
    conditions:
    4
    - transition : 't0'
    5
    time-min : 0.0
    6
    time-max : 3.0
    7
    condition : '/device/key == 10'
    8
    9
    - transition : 't1'
    10
    ...
    init t0
    [0.0, 3.0]
    /device/key == 10
    Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 12/20
    12/20
    

    View Slide

15. Offline Improvisation
    

    View Slide

16. Offline Improvisation
    Specification of actions
    1
    # file: configuration.yml
    2
    3
    actions:
    4
    - address : '/volume/sensor/pos_x'
    5
    value : 10
    6
    time : 250
    7
    8
    - address : ...
    Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 13/20
    13/20
    

    View Slide

17. Offline Improvisation
    Adding environment
    init t0
    /device/key == 10
    [0.0, 3.0]
    Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 14/20
    14/20
    

    View Slide

18. Offline Improvisation
    Adding environment
    (a) human-robot interaction
    init
    env
    t0
    (a,v,t)
    /device/key == 10
    [0.0, 3.0]
    (b) i-score
    Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 15/20
    15/20
    

    View Slide

19. Offline Improvisation
    Adding environment
    init
    env
    t0
    (a,v,t)
    (a == /device/key) ∧ (v == 10) ∧ (t == get_time())
    [0.0, 3.0]
    Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 16/20
    16/20
    

    View Slide

20. Offline Improvisation
    Adding environment
    init
    env
    t0
    (a,v,t)
    te
    (a,v,t)
    get_time() > t
    (a == /device/key) ∧ (v == 10) ∧ (t == get_time())
    [0.0, 3.0]
    Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 17/20
    17/20
    

    View Slide

21. Real-time Improvisation
    

    View Slide

22. Real-time Improvisation
    The inter-media sequencer i-score
    score /
    Initial Transition
    T10
    T6
    T5
    T8
    T4
    T2
    T1
    T9
    T7
    T11
    Popolari61oreweed40
    Loop pattern
    snozzle24ambier35
    Loop pattern
    pronger64inurbane46
    Loop pattern
    spouty57carthame37
    Loop pattern
    T5
    T6
    T10
    T1
    T2
    T4
    T8
    T11
    T7
    T9
    T3
    scissel86thermo23
    Loop pattern
    T3
    (c) i-score
    end
    init
    t1
    t0
    t2
    t3
    t4 t5
    t6
    t7
    t8
    t9
    t10
    t11
    (d) Petri net
    Figure: i-score representation.
    Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 18/20
    18/20
    

    View Slide

23. Real-time Improvisation
    The inter-media sequencer i-score
    score /
    Initial Transition
    T10
    T6
    T5
    T8
    T4
    T2
    T1
    Popolari61oreweed40
    Loop pattern
    snozzle24ambier35
    Loop pattern
    pronger64inurbane46
    Loop pattern
    T5
    T6
    T10
    T1
    T2
    T4
    T8
    (a) i-score
    end
    init
    t1
    t0
    t2
    t3
    t4 t5
    t6
    t7
    t8
    t9
    t10
    t11
    (b) Petri net
    Figure: i-score representation.
    Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 19/20
    19/20
    

    View Slide

24. Concluding Remarks
    

    View Slide

25. Concluding Remarks
    • Music as a set of combinatorial processes with concurrency control
    • Improvisation as a stylistic combinatorial system
    • Open form establishes a set of conditions and actions for synchronization
    • Generalizes and automates existing DAW tools (session view - clip, follow...)
    Combinatorial systems date back to ars inventendi (Leibniz) and are the core of ars
    magna (great art) of the Franciscan monk Ramon Llull (1232-1316), as well as the
    ecstatic Kabbalah of Abraham Abulafia (1240-1291), and Renaissance philosophers
    such as Giordano Bruno (1548-1600).
    ’́One should not pay attention to the properties of terms but only to the fact
    that they define an order, a texture, an architecture‘̀ U. Eco, The Search for the
    Perfect Language, Wiley-Blackwell 1997
    Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 20/20
    20/20
    

    View Slide

26. Questions
    

    View Slide

27. Thank you for your attention!
    

    View Slide

28. Automatic Construction of Interactive Machine
    Improvisation Scenarios from Audio Recordings
    Jaime Arias, Myriam Desainte-Catherine and Shlomo Dubnov
    Université de Bordeaux, LaBRI, UMR 5800
    Inria - Bordeaux Sud-Ouest
    University of California, San Diego, CREL
    MuMe Workshop
    International Conference on Computational Creativity
    June 27, 2016
    

    View Slide