Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Execution of Interactive Scores Using ReactiveML

Execution of Interactive Scores Using ReactiveML

Jaime Arias Almeida

June 26, 2013
Tweet

More Decks by Jaime Arias Almeida

Other Decks in Research

Transcript

  1. Execution of Interactive Scores using ReactiveML1
    Jaime E. ARIAS ALMEIDA
    GT Image et Son / SCRIME - LaBRI - Université Bordeaux 1
    Supervisors: Myriam DESAINTE-CATHERINE and Camilo RUEDA
    June 26, 2013
    1OSSIA ANR Project.

    View full-size slide

  2. .
    Outline
    Interactive Scores
    ReactiveML
    Interactive Scores in ReactiveML
    Graphical Interface in INScore
    Demonstration
    Work to be done
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 1/26
    .
    .
    .
    1/26

    View full-size slide

  3. .
    Outline
    Interactive Scores
    ReactiveML
    Interactive Scores in ReactiveML
    Graphical Interface in INScore
    Demonstration
    Work to be done
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 2/26
    .
    .
    .
    2/26

    View full-size slide

  4. .
    Interactive Scores2
    Introduction
    • An interactive score is represented by:
    ◦ Temporal objects: are sound, video, lights, etc.
    ◦ Temporal relations: provide a partial order for the execution of the
    temporal objects.
    ◦ Interactive objects: allow to control the start and the end of temporal
    objects.
    • They are equipped with hierarchy.
    2M. Toro-Bermudez, “Structured Interactive Scores: From a Simple Structural
    Description of a Multimedia Scenario to a Real-Time Capable Implementation with
    Formal Semantics,” Université de Bordeaux 1, Bordeaux, 2012.
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 3/26
    .
    .
    .
    3/26

    View full-size slide

  5. .
    Interactive Scores
    Example
    B
    C
    A
    F
    S(f)
    E(f)
    S(a) E(a)
    S(c) E(c) S(b) E(b)
    event1
    event2
    Delta(f)
    Delta(a)
    Delta(c) Delta(b)
    Delta(r0)
    Delta(r1)
    Delta(sa)
    Delta(sc)
    Delta(eb)
    D
    S(d) E(d)
    Delta(d)
    Delta(r2)
    Delta(ed)
    Figure : Example of an Interactive Score
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 4/26
    .
    .
    .
    4/26

    View full-size slide

  6. .
    Interactive Scores
    Petri Nets
    S(f) E(f)
    S(a) E(a)
    S(c) E(c)
    S(b) E(b)
    event1
    event2
    Delta(f)
    Delta(sa)
    Delta(sc)
    Delta(a)
    Delta(c)
    Delta(r0)
    Delta(r1)
    Delta(b) Delta(eb)
    S(d) E(d)
    Delta(d)
    Delta(r2)
    Delta(ed)
    Figure : Petri Net of an Interactive Score
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 5/26
    .
    .
    .
    5/26

    View full-size slide

  7. .
    Outline
    Interactive Scores
    ReactiveML
    Interactive Scores in ReactiveML
    Graphical Interface in INScore
    Demonstration
    Work to be done
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 6/26
    .
    .
    .
    6/26

    View full-size slide

  8. .
    ReactiveML4
    Introduction
    • ReactiveML is a synchronous reactive extension of the functional
    language Ocaml (without objects, labels and functors).
    • The language is based on the reactive model introduced by
    Boussinot3.
    ◦ It is based on the synchronous model.
    • It is founded on the ideal zero delay.
    • They forbidden important features like recursion.
    ◦ The reaction to absence of signals is delayer (a signal cannot be
    presented and absent during an instant).
    ◦ Programs are causal by construction.
    ◦ Dynamic creation of processes since the system cannot react
    instantaneously to the absence of an event.
    3Boussinot F., de Simone R., « The SL Synchronous Language », Software
    Engineering, vol. 22, n. 4, p. 256-266, 1996.
    4http://rml.lri.fr
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 7/26
    .
    .
    .
    7/26

    View full-size slide

  9. .
    ReactiveML
    Introduction
    • Time is view as a sequence of logical instants.
    ◦ Ocaml functions are considered to be instantaneous.
    ◦ Processes may last over several logical instants.
    • Behavioral semantics à la Esterel. Defines what is a valid reaction no
    matter how this reaction is actually computed.
    • Communication between parallel processes is made by broadcasting
    signals.
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 8/26
    .
    .
    .
    8/26

    View full-size slide

  10. .
    ReactiveML
    Introduction
    • Applications:
    ◦ Video games
    ◦ Graphical interfaces
    ◦ Simulators
    • Recently the Antescofo language was embedded in ReactiveML.
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 9/26
    .
    .
    .
    9/26

    View full-size slide

  11. .
    ReactiveML
    Example
    1 let process hello_world =
    2 signal s in
    3 (await s; print_string "World!")
    4 ||
    5 (print_string "Hello "; emit s)
    6 ;;
    1 #run hello_world;;
    Program code 1: Hello world
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 10/26
    .
    .
    .
    10/26

    View full-size slide

  12. .
    ReactiveML
    Example
    1 let process emit_tic period tic =
    2 let next = ref (Unix.gettimeofday() +. period) in
    3 loop
    4 let current = Unix.gettimeofday() in
    5 if current >= !next then begin
    6 emit tic;
    7 next := !next +. period
    8 end;
    9 pause
    10 end
    11 ;;
    Program code 2: Clock Generator
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 11/26
    .
    .
    .
    11/26

    View full-size slide

  13. .
    Outline
    Interactive Scores
    ReactiveML
    Interactive Scores in ReactiveML
    Graphical Interface in INScore
    Demonstration
    Work to be done
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 12/26
    .
    .
    .
    12/26

    View full-size slide

  14. .
    Interactive Scores in ReactiveML
    Description of a score
    • Very intuitive !.
    ◦ OSC Message: (path, data)
    type osc_message = string * RLO.message_data list ;;
    ◦ Interactive Point:
    type interactive_point = No_IP | IP of osc_message;;
    ◦ Relation:
    type relation = {id_r : identifier; dur_r : time};;
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 13/26
    .
    .
    .
    13/26

    View full-size slide

  15. .
    Interactive Scores in ReactiveML
    Description of a score
    • Very intuitive !.
    ◦ Box:
    type box =
    {
    id : identifier; start_t : time; dur : time;
    start_m : osc_message; end_m : osc_message;
    r_pred : predecessors; r_suce : sucessors;
    ip_start : interactive_point;
    ip_end : interactive_point;
    } ;;
    ◦ Score:
    type score = {
    boxes : box list ; relations : relation array};;
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 14/26
    .
    .
    .
    14/26

    View full-size slide

  16. .
    Interactive Scores in ReactiveML
    Behavior of a Box
    start of
    the box
    wait for
    the start
    time of the
    box
    no prede
    wait for the
    predecessors
    of the box
    wait for
    the
    interactive
    point
    prede
    ip
    ip
    Start the
    external
    process
    no ip
    no ip
    ip
    Wait the
    duration of
    the box
    no ip
    Stop the
    external
    process
    is start
    is end
    Schedule
    the
    successors
    boxes
    Behavior of a box
    Figure : A simple representation with finite automata
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 15/26
    .
    .
    .
    15/26

    View full-size slide

  17. .
    Interactive Scores in ReactiveML
    Components
    process: emit_tic
    generate a clock signal
    period: float tic: signal
    process: clock
    translate a clock signal
    into time
    period: float time: signal(float)
    tic: signal
    Figure : Clock
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 16/26
    .
    .
    .
    16/26

    View full-size slide

  18. .
    Interactive Scores in ReactiveML
    Components
    make_scheduler
    process: schedule
    scheduler
    process: plan
    add a signal 's' to the
    scheduler
    date: signal(float)
    add_queue:
    signal(float,signal)
    dur: float
    s: signal
    process: wait
    generate a delay
    period: float
    tic: signal
    duration: float
    process: send
    send a signal scheduled for
    the current time
    date: signal(float)
    queue_pop:
    function
    process: update
    schedule a signal
    add_queue: signal
    queue_push:
    function
    date: signal(float)
    add_queue: signal
    process: plan
    date: signal(float)
    add_queue
    dur
    s
    process: schedule
    Figure : Scheduler
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 17/26
    .
    .
    .
    17/26

    View full-size slide

  19. .
    Interactive Scores in ReactiveML
    Components
    process: osc_client
    OSC client
    host: string
    client_port: int
    osc_messages: signal
    process: osc_server
    OSC server (async)
    server_port: int
    listen: signal
    process:
    mem_to_sig
    Send messages of a thread
    in a signal
    process: read_message
    (rec)
    Execute the server in a thread
    m: share_mem s: signal
    server: RLO.server
    Figure : OSC Client/Server
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 18/26
    .
    .
    .
    18/26

    View full-size slide

  20. .
    Interactive Scores in ReactiveML
    Components
    make_player
    process: wait_pred
    (rec)
    wait for the predecessors
    boxes
    pred: list (predecessors)
    process: successors_end
    (rec)
    schedule the successors boxes
    succe: list (succesors)
    process: exec
    (rec)
    execute a score
    score: list (boxes)
    process: exec_box_begin
    execute the start of a box
    (predecessors, no predecessors)
    box: box
    process: exec_ip
    wait for interactive events or
    execute a process 'proc'
    ip: list(interactive_point)
    proc: process
    process: exec_box
    execute the body of a box
    box: box
    process: end_box
    execute the end of a box
    (successors, no successors)
    box: box
    iscore: score
    plan: function
    listen: signal
    osc_messages: signal
    relations: signal array
    events: signal
    tic: signal
    period: float
    Figure : Score Player
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 19/26
    .
    .
    .
    19/26

    View full-size slide

  21. .
    Outline
    Interactive Scores
    ReactiveML
    Interactive Scores in ReactiveML
    Graphical Interface in INScore
    Demonstration
    Work to be done
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 20/26
    .
    .
    .
    20/26

    View full-size slide

  22. .
    Graphical Interface in INScore5
    • INScore is an environment for the design of Interactive Augmented
    Music Scores.
    • Synchronize ReactiveML with INScore via OSC messages !.
    • INScore observes the execution of the score.
    • Build the scenario dynamically.
    Figure : Graphical Interface in INScore
    5http://inscore.sourceforge.net
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 21/26
    .
    .
    .
    21/26

    View full-size slide

  23. .
    Outline
    Interactive Scores
    ReactiveML
    Interactive Scores in ReactiveML
    Graphical Interface in INScore
    Demonstration
    Work to be done
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 22/26
    .
    .
    .
    22/26

    View full-size slide

  24. .
    Demonstration
    Let’s try!
    Box 2
    Box 1
    Box 0
    /end 1
    /start 0
    5.0
    3.0 5.0
    r0 = 2.0
    r1 = 5.0
    3.0 Box 3
    3.0
    r2 = 6.0
    2.0
    10.0
    14.0
    Figure : Example
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 23/26
    .
    .
    .
    23/26

    View full-size slide

  25. .
    Outline
    Interactive Scores
    ReactiveML
    Interactive Scores in ReactiveML
    Graphical Interface in INScore
    Demonstration
    Work to be done
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 24/26
    .
    .
    .
    24/26

    View full-size slide

  26. .
    Work to be done
    • Parser: XML format → Score structure.
    • Hierarchy.
    • Conditionals.
    • Loops.
    • Improve the graphical interface.
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 25/26
    .
    .
    .
    25/26

    View full-size slide

  27. .
    Thank you for your attention.
    Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 26/26
    .
    .
    .
    26/26

    View full-size slide