Executing Hierarchical Interactive Scores in ReactiveML Jaime Arias Joint work with Myriam Desainte-Catherine, Sylvain Salvati and Camilo Rueda Laboratoire Bordelais de Recherche en Informatique (LaBRI) Université de Bordeaux Journées d’Informatique Musicale Bourges, May 2014 1 
 

Motivation • Interactive scores [10] is a formalism for writing and executing interactive scenarios. • A wide range of application such as video games and museum installations. ◦ Non-linear behaviour ⇝ conditionals, loops . • I-score1. The execution model is based on Hierarchical Time Stream Petri Nets [19]. ◦ Very static. ◦ New features would require a complete redesign of the network. • There are some problems: ◦ Hierarchical aspects. ◦ Feedback during execution. ◦ Complexity of the network. 1http://i-score.org Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 1/18 1/18 

Motivation • ReactiveML is a programming language for implementing interactive systems [16]. • Synchronous reactive model of Boussinot [8]. • It provides a discrete model of time, clear semantics, synchronous, deterministic and dynamic features. • It has been previously used in music applications [4] showing to be ◦ Very expressive. ◦ Efficient. ◦ Capable of interacting with the environment during the performance of complex scores. ◦ Well suited for building prototypes easily. Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 2/18 2/18 

This talk is about … We explore a new way to define and implement interactive scores, aiming at a more dynamic and simple model. Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 3/18 3/18 

Outline What is essential to know? Interactive Scores ReactiveML INScore A synchronous model for IS Improving the visualization Demo Summary Future work Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 4/18 4/18 

What is essential to know? Interactive Scores 2 4 3 5 1 6 0 r2 r1 r3 r4 r5 r6 r7 Figure: Example of an interactive score. skip Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 5/18 5/18 

What is essential to know? Interactive Scores Figure: Example of an interactive score designed in i-score. Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 6/18 6/18 

What is essential to know? ReactiveML 1 let process emit_tic period tic = 2 let start = Unix.gettimeofday () in 3 let next = ref (start +. period) in 4 loop 5 let current = Unix.gettimeofday() in 6 if (current >= !next) then begin 7 emit tic (); 8 next := !next +. period 9 end; 10 pause 11 end Figure: Example of ReactiveML language.2 2Based on the work of Baudart et al. [5] Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 7/18 7/18 

What is essential to know? ReactiveML logical time seconds signal tic; #run emit_tic 1.0 tic Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 8/18 8/18 

What is essential to know? ReactiveML logical time seconds signal tic; #run emit_tic 1.0 tic start = Unix.gettimeofday () next = ref (start +. period) current = Unix.gettimeofday() current >= next ? Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 8/18 8/18 

What is essential to know? ReactiveML logical time seconds signal tic; #run emit_tic 1.0 tic start = Unix.gettimeofday () next = ref (start +. period) current = Unix.gettimeofday() current >= next ? no Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 8/18 8/18 

What is essential to know? ReactiveML logical time seconds signal tic; #run emit_tic 1.0 tic start = Unix.gettimeofday () next = ref (start +. period) current = Unix.gettimeofday() current >= next ? no pause Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 8/18 8/18 

What is essential to know? ReactiveML logical time seconds signal tic; #run emit_tic 1.0 tic start = Unix.gettimeofday () next = ref (start +. period) current = Unix.gettimeofday() current >= next ? Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 8/18 8/18 

What is essential to know? ReactiveML logical time seconds signal tic; #run emit_tic 1.0 tic start = Unix.gettimeofday () next = ref (start +. period) current = Unix.gettimeofday() current >= next ? no Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 8/18 8/18 

What is essential to know? ReactiveML logical time seconds signal tic; #run emit_tic 1.0 tic start = Unix.gettimeofday () next = ref (start +. period) current = Unix.gettimeofday() current >= next ? no pause Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 8/18 8/18 

What is essential to know? ReactiveML logical time seconds signal tic; #run emit_tic 1.0 tic start = Unix.gettimeofday () next = ref (start +. period) current = Unix.gettimeofday() current >= next ? Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 8/18 8/18 

What is essential to know? ReactiveML logical time seconds signal tic; #run emit_tic 1.0 tic start = Unix.gettimeofday () next = ref (start +. period) current = Unix.gettimeofday() current >= next ? yes Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 8/18 8/18 

What is essential to know? ReactiveML logical time seconds signal tic; #run emit_tic 1.0 tic 1 start = Unix.gettimeofday () next = ref (start +. period) current = Unix.gettimeofday() current >= next ? yes emit tic () Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 8/18 8/18 

What is essential to know? ReactiveML logical time seconds signal tic; #run emit_tic 1.0 tic 1 start = Unix.gettimeofday () next = ref (start +. period) current = Unix.gettimeofday() current >= next ? yes emit tic () pause Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 8/18 8/18 

What is essential to know? ReactiveML logical time seconds signal tic; #run emit_tic 1.0 tic 1 2 Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 8/18 8/18 

What is essential to know? INScore Figure: Example of a score in INScore. Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 9/18 9/18 

A synchronous model for IS Temporal relations time fixed interval interactive interval duration minimum duration maximum duration interaction point enabled Figure: Representation of intervals. Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 10/18 10/18 

A synchronous model for IS Boxes 1 2 r1 r2 (a) Visual 1 signal pb1_s, hb1_s, hb2_s, r1_s, r2_s; 2 let ip = ("/stop",[Int32 3]); 3 let start_m = ("/box/1",[String "start "]); 4 let stop_m = ("/box/1",[String "stop"]); 5 let r1 = Fixed (Finite 2, r1_s); 6 let r2 = Fixed (Finite 0, r2_s); 7 let p_box = Process (1, 8 Fixed (Finite 2, pb1_s), 9 start_m, stop_m); 10 let h_box = Hierarchical (2, 11 [p_box], 12 [(2,1,r1);(1,2,r2)], 13 Interactive ((Finite 2, hb1_s), 14 (Infinite, hb2_s), ip)) (b) Text Figure: Representation of boxes. Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 11/18 11/18 

A synchronous model for IS Merging intervals 2 3 1 r1 r2 Figure: Example of a box with two preceding temporal relations. Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 12/18 12/18 

A synchronous model for IS Merging intervals time fixed interval 1 fixed interval 2 merge fixed (1,2) (a) Merging fixed intervals. interactive interval 1 interactive interval 2 merge interactive (1,2) min max min max min max interaction point enabled time (b) Merging interactive intervals. Figure: Merging intervals. Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 13/18 13/18 

Improving the visualization Interpreter Sync. Observer INScore signals Pure Data OSC OSC OSC Figure: Diagram of the visualization system. Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 14/18 14/18 

Demo Let’s try it! see scenario Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 15/18 15/18 

Summary • We presented a synchronous interpreter of multimedia interactive scores. • We showed that the model is simple. • Contrary to the Petri Net model, our approach allows to model precisely the hierarchical behaviour of boxes. • We could prototype new features easily and execute living code using the toplevel of ReactiveML. • We explored the use of INScore to develop a graphical interface that provides a real-time visualization of the execution of the score. • Our approach would allow to execute dynamic processes unlike Petri Nets. Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 16/18 16/18 

Future work • Formalize the operational semantics (generic model). • Take advantage of ReactiveML properties for prototyping new features. ◦ Add conditionals and loops for describing non-linear behaviour. • Analyze data-flow programming languages like Lucid Synchrone for handling streams. • Carry out a performance testing. • Implement a compiler i-score ⇝ Rml-score • Verification of properties. • Improve the visual environment in INScore. Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 17/18 17/18 

Thank you for your attention. Jaime Arias (LaBRI) Executing Hierarchical Interactive Scores in ReactiveML 18/18 18/18 

Executing Hierarchical Interactive Scores in ReactiveML Jaime Arias Joint work with Myriam Desainte-Catherine, Sylvain Salvati and Camilo Rueda Laboratoire Bordelais de Recherche en Informatique (LaBRI) Université de Bordeaux Journées d’Informatique Musicale Bourges, May 2014 1 
 

References Allen, J., ‹Maintaining knowledge about temporal intervals›, Communications of the ACM, ACM, vol. 26 (11), New York, NY, USA, 1983, p. 832–843. Allombert, A., ‹Aspects temporels d’un système de partitions musicales interactives pour la composition et l’exécution›, Ph.D. Thesis, Bordeaux, France, 2009. Allombert, A., Desainte-Catherine, M., and Assayag, G., ‹Iscore: A system for writing interaction›, Proceedings of the Third International Conference on Digital Interactive Media in Entertainment and Arts, ACM, New York, NY, USA, 2008, p. 360–367. Baudart, G., Jacquemard, F., Mandel, L., and Pouzet, M., ‹A synchronous embedding of Antescofo, a domain-specific language for interactive mixed music›, Proceedings of the Thirteen International Conference on Embedded Software, Montreal, Canada, 2013. 

References Baudart, G., Mandel, L., and Pouzet, M., ‹Programming mixed music in ReactiveML›, Proceedings of the First ACM SIGPLAN Workshop on Functional Art, Music, Modeling, Boston, USA, 2013. Benveniste, A., Caspi, P., Edwards, S., Halbwachs, N., Le Guernic, P., and De Simone, R., ‹The synchronous languages 12 years later›, Proceedings of the IEEE, IEEE, vol. 91 (1), 2003, p. 64–83. Berry, G., and Gonthier, G., ‹The Esterel synchronous programming language, design, semantics, implementation›, Science of Computer Programming, Elsevier, vol. 19 (2), Amsterdam, The Netherlands, 1992, p. 87–152. Boussinot, F., and De Simone, R., ‹The SL synchronous language›, IEEE Transactions on Software Engineering, IEEE Press, vol. 22 (4), Piscataway, USA, 1996, p. 256–266. 

References Cont, A., ‹ANTESCOFO: Anticipatory synchronization and control of interactive parameters in computer music›, Proceedings of International Computer Music Conference, Belfast, Ireland, 2008. Desainte-Catherine, M., Allombert, A., and Assayag, G., ‹Towards a hybrid temporal paradigm for musical composition and performance: The case of musical interpretation›, Computer Music Journal, MIT Press, vol. 37 (2), Cambridge, MA, USA, 2013, p. 61–72. Fober, D., Orlarey, Y., and Letz, S., ‹An environment for the design of live music scores›, Proceedings of the Linux Audio Conference, CCRMA, Stanford University, California, US, 2012, p. 47–54. Halbwachs, N., Caspi, P., Raymond, P., and Pilaud, D., ‹The synchronous dataflow programming language lustre›, Proceedings of the IEEE, IEEE, vol. 79 (9), 1991, p. 1305–-1320. 

References Halbwachs, N., Lagnier, F., and Raymond, P., ‹Synchronous observers and the verification of reactive systems›, Proceedings of the Third International Conference on Methodology and Software Technology, Springer-Verlag, London, UK, 1994, p. 83–96. Halbwachs, N., and Raymond, P., ‹Validation of synchronous reactive systems: From formal verification to automatic testing›, Proceedings of the Fifth Asian Computing Science Conference on Advances in Computing Science, Springer-Verlag, 1999, p. 1–12. Mandel, L., and Plateau, F., ‹Interactive programming of reactive systems›, Electronic Notes in Theoretical Computer Science, Elsevier, vol. 238 (1), Amsterdam, The Netherlands, 2009, p. 21–36. 

References Mandel, L., and Pouzet, M., ‹ReactiveML, a reactive extension to ML›, Proceedings of the Seventh ACM SIGPLAN International Symposium on Principles and Practice of Declarative Programming, Lisbon, Portugal, 2005. Marczak, R., Desainte-Catherine, M., and Allombert, A., ‹Real-time temporal control of musical processes›, The Third International Conferences on Advances in Multimedia, Budapest, Hungary, 2011, p. 12–17. Pouzet, M., ‹Lucid Synchrone, version 3. Tutorial and reference manual›, http://www.di.ens.fr/~pouzet/lucid-synchrone/ lucid-synchrone-3.0-manual.pdf. 

References Sénac, P., De Saqui-Sannes, P., and Willrich, R., ‹Hierarchical Time Stream Petri Net: A model for hypermedia systems›, Proceedings of the Sixteenth International Conference on Application and Theory of Petri Nets, Springer, Turin, Italy, 1995, p. 451–470.