Execution of Interactive Scores using ReactiveML1 Jaime E. ARIAS ALMEIDA GT Image et Son / SCRIME - LaBRI - Université Bordeaux 1 Supervisors: Myriam DESAINTE-CATHERINE and Camilo RUEDA June 26, 2013 1OSSIA ANR Project.

. Outline Interactive Scores ReactiveML Interactive Scores in ReactiveML Graphical Interface in INScore Demonstration Work to be done Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 1/26 . . . 1/26

. Outline Interactive Scores ReactiveML Interactive Scores in ReactiveML Graphical Interface in INScore Demonstration Work to be done Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 2/26 . . . 2/26

. Interactive Scores2 Introduction • An interactive score is represented by: ◦ Temporal objects: are sound, video, lights, etc. ◦ Temporal relations: provide a partial order for the execution of the temporal objects. ◦ Interactive objects: allow to control the start and the end of temporal objects. • They are equipped with hierarchy. 2M. Toro-Bermudez, “Structured Interactive Scores: From a Simple Structural Description of a Multimedia Scenario to a Real-Time Capable Implementation with Formal Semantics,” Université de Bordeaux 1, Bordeaux, 2012. Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 3/26 . . . 3/26

. Interactive Scores Example B C A F S(f) E(f) S(a) E(a) S(c) E(c) S(b) E(b) event1 event2 Delta(f) Delta(a) Delta(c) Delta(b) Delta(r0) Delta(r1) Delta(sa) Delta(sc) Delta(eb) D S(d) E(d) Delta(d) Delta(r2) Delta(ed) Figure : Example of an Interactive Score Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 4/26 . . . 4/26

. Interactive Scores Petri Nets S(f) E(f) S(a) E(a) S(c) E(c) S(b) E(b) event1 event2 Delta(f) Delta(sa) Delta(sc) Delta(a) Delta(c) Delta(r0) Delta(r1) Delta(b) Delta(eb) S(d) E(d) Delta(d) Delta(r2) Delta(ed) Figure : Petri Net of an Interactive Score Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 5/26 . . . 5/26

. Outline Interactive Scores ReactiveML Interactive Scores in ReactiveML Graphical Interface in INScore Demonstration Work to be done Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 6/26 . . . 6/26

. ReactiveML4 Introduction • ReactiveML is a synchronous reactive extension of the functional language Ocaml (without objects, labels and functors). • The language is based on the reactive model introduced by Boussinot3. ◦ It is based on the synchronous model. • It is founded on the ideal zero delay. • They forbidden important features like recursion. ◦ The reaction to absence of signals is delayer (a signal cannot be presented and absent during an instant). ◦ Programs are causal by construction. ◦ Dynamic creation of processes since the system cannot react instantaneously to the absence of an event. 3Boussinot F., de Simone R., « The SL Synchronous Language », Software Engineering, vol. 22, n. 4, p. 256-266, 1996. 4http://rml.lri.fr Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 7/26 . . . 7/26

. ReactiveML Introduction • Time is view as a sequence of logical instants. ◦ Ocaml functions are considered to be instantaneous. ◦ Processes may last over several logical instants. • Behavioral semantics à la Esterel. Defines what is a valid reaction no matter how this reaction is actually computed. • Communication between parallel processes is made by broadcasting signals. Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 8/26 . . . 8/26

. ReactiveML Introduction • Applications: ◦ Video games ◦ Graphical interfaces ◦ Simulators • Recently the Antescofo language was embedded in ReactiveML. Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 9/26 . . . 9/26

. ReactiveML Example 1 let process hello_world = 2 signal s in 3 (await s; print_string "World!") 4 || 5 (print_string "Hello "; emit s) 6 ;; 1 #run hello_world;; Program code 1: Hello world Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 10/26 . . . 10/26

. ReactiveML Example 1 let process emit_tic period tic = 2 let next = ref (Unix.gettimeofday() +. period) in 3 loop 4 let current = Unix.gettimeofday() in 5 if current >= !next then begin 6 emit tic; 7 next := !next +. period 8 end; 9 pause 10 end 11 ;; Program code 2: Clock Generator Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 11/26 . . . 11/26

. Outline Interactive Scores ReactiveML Interactive Scores in ReactiveML Graphical Interface in INScore Demonstration Work to be done Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 12/26 . . . 12/26

. Interactive Scores in ReactiveML Description of a score • Very intuitive !. ◦ OSC Message: (path, data) type osc_message = string * RLO.message_data list ;; ◦ Interactive Point: type interactive_point = No_IP | IP of osc_message;; ◦ Relation: type relation = {id_r : identifier; dur_r : time};; Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 13/26 . . . 13/26

. Interactive Scores in ReactiveML Description of a score • Very intuitive !. ◦ Box: type box = { id : identifier; start_t : time; dur : time; start_m : osc_message; end_m : osc_message; r_pred : predecessors; r_suce : sucessors; ip_start : interactive_point; ip_end : interactive_point; } ;; ◦ Score: type score = { boxes : box list ; relations : relation array};; Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 14/26 . . . 14/26

. Interactive Scores in ReactiveML Behavior of a Box start of the box wait for the start time of the box no prede wait for the predecessors of the box wait for the interactive point prede ip ip Start the external process no ip no ip ip Wait the duration of the box no ip Stop the external process is start is end Schedule the successors boxes Behavior of a box Figure : A simple representation with finite automata Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 15/26 . . . 15/26

. Interactive Scores in ReactiveML Components process: emit_tic generate a clock signal period: float tic: signal process: clock translate a clock signal into time period: float time: signal(float) tic: signal Figure : Clock Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 16/26 . . . 16/26

. Interactive Scores in ReactiveML Components make_scheduler process: schedule scheduler process: plan add a signal 's' to the scheduler date: signal(float) add_queue: signal(float,signal) dur: float s: signal process: wait generate a delay period: float tic: signal duration: float process: send send a signal scheduled for the current time date: signal(float) queue_pop: function process: update schedule a signal add_queue: signal queue_push: function date: signal(float) add_queue: signal process: plan date: signal(float) add_queue dur s process: schedule Figure : Scheduler Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 17/26 . . . 17/26

. Interactive Scores in ReactiveML Components process: osc_client OSC client host: string client_port: int osc_messages: signal process: osc_server OSC server (async) server_port: int listen: signal process: mem_to_sig Send messages of a thread in a signal process: read_message (rec) Execute the server in a thread m: share_mem s: signal server: RLO.server Figure : OSC Client/Server Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 18/26 . . . 18/26

. Interactive Scores in ReactiveML Components make_player process: wait_pred (rec) wait for the predecessors boxes pred: list (predecessors) process: successors_end (rec) schedule the successors boxes succe: list (succesors) process: exec (rec) execute a score score: list (boxes) process: exec_box_begin execute the start of a box (predecessors, no predecessors) box: box process: exec_ip wait for interactive events or execute a process 'proc' ip: list(interactive_point) proc: process process: exec_box execute the body of a box box: box process: end_box execute the end of a box (successors, no successors) box: box iscore: score plan: function listen: signal osc_messages: signal relations: signal array events: signal tic: signal period: float Figure : Score Player Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 19/26 . . . 19/26

. Outline Interactive Scores ReactiveML Interactive Scores in ReactiveML Graphical Interface in INScore Demonstration Work to be done Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 20/26 . . . 20/26

. Graphical Interface in INScore5 • INScore is an environment for the design of Interactive Augmented Music Scores. • Synchronize ReactiveML with INScore via OSC messages !. • INScore observes the execution of the score. • Build the scenario dynamically. Figure : Graphical Interface in INScore 5http://inscore.sourceforge.net Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 21/26 . . . 21/26

. Outline Interactive Scores ReactiveML Interactive Scores in ReactiveML Graphical Interface in INScore Demonstration Work to be done Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 22/26 . . . 22/26

. Demonstration Let’s try! Box 2 Box 1 Box 0 /end 1 /start 0 5.0 3.0 5.0 r0 = 2.0 r1 = 5.0 3.0 Box 3 3.0 r2 = 6.0 2.0 10.0 14.0 Figure : Example Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 23/26 . . . 23/26

. Outline Interactive Scores ReactiveML Interactive Scores in ReactiveML Graphical Interface in INScore Demonstration Work to be done Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 24/26 . . . 24/26

. Work to be done • Parser: XML format → Score structure. • Hierarchy. • Conditionals. • Loops. • Improve the graphical interface. Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 25/26 . . . 25/26

. Thank you for your attention. Jaime E. ARIAS ALMEIDA Execution of Interactive Scores using ReactiveML 26/26 . . . 26/26