Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings

Transcript

1. Automatic Construction of Interactive Machine
   Improvisation Scenarios from Audio Recordings
   Jaime Arias, Myriam Desainte-Catherine and Shlomo Dubnov
   Université de Bordeaux, LaBRI, UMR 5800
   Inria - Bordeaux Sud-Ouest
   Les Cycles Arts et Sciences du SCRIME
   June 24, 2016
   

   View Slide

2. Motivation
   • PyOracle: https://gitlab.com/himito/PyOracle_I-score
   • i-score: https://github.com/himito/i-score
   • VMO-Score: https://himito.github.io/vmo_i-score_generator
   Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 1/16
   1/16
   

   View Slide

3. Motivation
   Overview of the system
   Pre-Recorded
   Audio
   Variable Markov
   Oracle (VMO)
   Audio
   Segmentation
   Audio
   Oracle
   SNAKES
   Petri Net Model
   input
   output
   output
   input
   output
   Configuration
   File
   output
   (a) Generation of the structure
   Petri Net Model
   Offline
   Improviser
   output
   input
   input
   Audio
   Synthesizer
   Audio File
   output
   input
   Audio Oracle
   input
   Petri Net
   Parameters
   Actions
   Oracle
   Parameters
   Performer Controls
   Configuration File
   Oracle Regions
   Sequence
   Audio
   Buffer
   input
   (b) Offline improvisation
   Figure: Overview
   Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 2/16
   2/16
   

   View Slide

4. Motivation
   Overview of the system
   Pre-Recorded
   Audio
   Variable Markov
   Oracle (VMO)
   Audio
   Segmentation
   Audio
   Oracle
   SNAKES
   Petri Net Model
   input
   output
   output
   input
   output
   Configuration
   File
   output
   (a) Generation of the structure
   Petri Net
   Model
   Audio
   Oracle
   i-score
   Petri Net
   Parameters
   Configuration File
   PyOracle
   input
   input
   input
   Real-time
   Performer Controls
   input
   input
   inout
   Real-time Audio
   Output
   Output
   (b) Real-time improvisation
   Figure: Overview
   Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 3/16
   3/16
   

   View Slide

5. Composition
   

   View Slide

6. Composition
   Audio Recording
   Figure: Audio recording: Philou.
   Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 4/16
   4/16
   

   View Slide

7. Composition
   Segmentation
   Figure: Segmentation
   Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 5/16
   5/16
   

   View Slide

8. Composition
   Petri Net
   G
   D
   C
   t
   (a) The marking before firing the enabled
   transition t.
   G
   D
   C
   t
   (b) The marking after firing transition t,
   where t is disabled.
   Figure: Illustration of a firing rule in a Petri Net.
   Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 6/16
   6/16
   

   View Slide

9. Composition
   Petri Net
   Figure: Segmentation
   end
   init
   t1
   t0
   t2
   t3
   t4 t5
   t6
   t7
   t8
   t9
   t10
   t11
   Figure: Timed Petri Net
   Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 7/16
   7/16
   

   View Slide

10. Composition
    Parameters of the Petri Net
    1
    # file: configuration.yml
    2
    3
    conditions:
    4
    - transition : 't0'
    5
    time-min : 0.0
    6
    time-max : 3.0
    7
    condition : '/device/key == 10'
    8
    9
    - transition : 't1'
    10
    ...
    init t0
    [0.0, 3.0]
    /device/key == 10
    Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 8/16
    8/16
    

    View Slide

11. Offline Improvisation
    

    View Slide

12. Offline Improvisation
    Specification of actions
    1
    # file: configuration.yml
    2
    3
    actions:
    4
    - address : '/volume/sensor/pos_x'
    5
    value : 10
    6
    time : 250
    7
    8
    - address : ...
    Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 9/16
    9/16
    

    View Slide

13. Offline Improvisation
    Adding environment
    init t0
    /device/key == 10
    [0.0, 3.0]
    Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 10/16
    10/16
    

    View Slide

14. Offline Improvisation
    Adding environment
    init
    env
    t0
    (a,v,t)
    /device/key == 10
    [0.0, 3.0]
    Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 11/16
    11/16
    

    View Slide

15. Offline Improvisation
    Adding environment
    init
    env
    t0
    (a,v,t)
    (a == /device/key) ∧ (v == 10) ∧ (t == get_time())
    [0.0, 3.0]
    Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 12/16
    12/16
    

    View Slide

16. Offline Improvisation
    Adding environment
    init
    env
    t0
    (a,v,t)
    te
    (a,v,t)
    get_time() > t
    (a == /device/key) ∧ (v == 10) ∧ (t == get_time())
    [0.0, 3.0]
    Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 13/16
    13/16
    

    View Slide

17. Real-time Improvisation
    

    View Slide

18. Real-time Improvisation
    The inter-media sequencer i-score
    score /
    Initial Transition
    T10
    T6
    T5
    T8
    T4
    T2
    T1
    T9
    T7
    T11
    Popolari61oreweed40
    Loop pattern
    snozzle24ambier35
    Loop pattern
    pronger64inurbane46
    Loop pattern
    spouty57carthame37
    Loop pattern
    T5
    T6
    T10
    T1
    T2
    T4
    T8
    T11
    T7
    T9
    T3
    scissel86thermo23
    Loop pattern
    T3
    (a) i-score
    end
    init
    t1
    t0
    t2
    t3
    t4 t5
    t6
    t7
    t8
    t9
    t10
    t11
    (b) Petri net
    Figure: i-score representation.
    Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 14/16
    14/16
    

    View Slide

19. Real-time Improvisation
    The inter-media sequencer i-score
    score /
    Initial Transition
    T10
    T6
    T5
    T8
    T4
    T2
    T1
    Popolari61oreweed40
    Loop pattern
    snozzle24ambier35
    Loop pattern
    pronger64inurbane46
    Loop pattern
    T5
    T6
    T10
    T1
    T2
    T4
    T8
    (a) i-score
    end
    init
    t1
    t0
    t2
    t3
    t4 t5
    t6
    t7
    t8
    t9
    t10
    t11
    (b) Petri net
    Figure: i-score representation.
    Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 15/16
    15/16
    

    View Slide

20. Concluding Remarks
    

    View Slide

21. Concluding Remarks
    Shlomo Dubnov - University of California, San Diego - CREL (2016) Automatic Construction of Interactive Machine Improvisation Scenarios from Audio Recordings 16/16
    16/16
    

    View Slide

22. Thank you for your attention!
    

    View Slide

23. Automatic Construction of Interactive Machine
    Improvisation Scenarios from Audio Recordings
    Jaime Arias, Myriam Desainte-Catherine and Shlomo Dubnov
    Université de Bordeaux, LaBRI, UMR 5800
    Inria - Bordeaux Sud-Ouest
    Les Cycles Arts et Sciences du SCRIME
    June 24, 2016
    

    View Slide