Modelling Data Processing for Interactive Scores Using Coloured Petri Nets Jaime Arias Joint work with Myriam Desainte-Catherine and Camilo Rueda Laboratoire Bordelais de Recherche en Informatique (LaBRI) Université de Bordeaux 14th International Conference on Application of Concurrency to System Design Tunis, June 2014 1 
 

Motivation Interactive Scores • Formalism for writing and executing interactive multimedia scenarios [Desainte-Catherine et al., 2013] (e.g., art installations). • An interactive score is composed of ◦ Multimedia objects. ◦ Hierarchical boxes. ◦ Temporal relations (Allen’s relations [Allen]). • Rigid relation • Flexible relation ◦ Interaction points. • i-score1 [Marczak et al., 2011] is a software that implements the above formalism. ◦ Authoring side : Constraint Satisfaction Problem (Gecode). ◦ Performance side : Hierarchical Time Stream Petri Net [Sénac et al., 1995] (HTSPN). 1http://i-score.org Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 1/22 1/22 

Motivation Interactive Scores • Formalism for writing and executing interactive multimedia scenarios [Desainte-Catherine et al., 2013] (e.g., art installations). • An interactive score is composed of ◦ Multimedia objects. ◦ Hierarchical boxes. ◦ Temporal relations (Allen’s relations [Allen]). • Rigid relation • Flexible relation ◦ Interaction points. • i-score1 [Marczak et al., 2011] is a software that implements the above formalism. ◦ Authoring side : Constraint Satisfaction Problem (Gecode). ◦ Performance side : Hierarchical Time Stream Petri Net [Sénac et al., 1995] (HTSPN). 1http://i-score.org Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 1/22 1/22 

Motivation Interactive Scores • Formalism for writing and executing interactive multimedia scenarios [Desainte-Catherine et al., 2013] (e.g., art installations). • An interactive score is composed of ◦ Multimedia objects. ◦ Hierarchical boxes. ◦ Temporal relations (Allen’s relations [Allen]). • Rigid relation • Flexible relation ◦ Interaction points. • i-score1 [Marczak et al., 2011] is a software that implements the above formalism. ◦ Authoring side : Constraint Satisfaction Problem (Gecode). ◦ Performance side : Hierarchical Time Stream Petri Net [Sénac et al., 1995] (HTSPN). 1http://i-score.org Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 1/22 1/22 

Motivation Interactive Scores Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 2/22 2/22 

Motivation Interactive Scores temporal relation interaction point hierarchical box process box Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 2/22 2/22 

Motivation Interactive Scores • Temporal relations are preserved during the authoring and performance sides ; a scenario can be interpreted in a limited set of possibilities. • The execution of multimedia processes is carried out by external applications (e.g., Pure Data, Max/MSP). • A wide range of applications such as the games and museum installations. ◦ Linear execution (i.e., without branching and loops). • Composers have increasingly needed to represent and manipulate complex data in their multimedia scenarios. ◦ i-score does not handle complex data (e.g., audio streams). Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 3/22 3/22 

Motivation Interactive Scores • Temporal relations are preserved during the authoring and performance sides ; a scenario can be interpreted in a limited set of possibilities. • The execution of multimedia processes is carried out by external applications (e.g., Pure Data, Max/MSP). • A wide range of applications such as the games and museum installations. ◦ Linear execution (i.e., without branching and loops). • Composers have increasingly needed to represent and manipulate complex data in their multimedia scenarios. ◦ i-score does not handle complex data (e.g., audio streams). Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 3/22 3/22 

Motivation Interactive Scores • Temporal relations are preserved during the authoring and performance sides ; a scenario can be interpreted in a limited set of possibilities. • The execution of multimedia processes is carried out by external applications (e.g., Pure Data, Max/MSP). • A wide range of applications such as the games and museum installations. ◦ Linear execution (i.e., without branching and loops). • Composers have increasingly needed to represent and manipulate complex data in their multimedia scenarios. ◦ i-score does not handle complex data (e.g., audio streams). Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 3/22 3/22 

Motivation Interactive Scores • Temporal relations are preserved during the authoring and performance sides ; a scenario can be interpreted in a limited set of possibilities. • The execution of multimedia processes is carried out by external applications (e.g., Pure Data, Max/MSP). • A wide range of applications such as the games and museum installations. ◦ Linear execution (i.e., without branching and loops). • Composers have increasingly needed to represent and manipulate complex data in their multimedia scenarios. ◦ i-score does not handle complex data (e.g., audio streams). Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 3/22 3/22 

This talk is about … Extension of the execution model of i-score using Coloured Petri Nets (CPNs) in order to handle data streams. Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 4/22 4/22 

Outline What is essential to know? The behaviour of interactive scores Execution model of interactive scores Extending the model for handling data Summary Future work Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 5/22 5/22 

What is essential to know? The behaviour of interactive scores 2 4 3 5 1 6 0 r2 r1 r3 r4 r5 r6 r7 time 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 r1 r2 Box 1 Box 2 Box 6 min max r3 min max max min r5 r6 Box 5 Box 4 min max Box 3 min r7 24 25 26 27 28 r4 merge (r3,r4) Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 6/22 6/22 

What is essential to know? The behaviour of interactive scores 2 4 3 5 1 6 0 r2 r1 r3 r4 r5 r6 r7 time 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 r1 r2 Box 1 Box 2 Box 6 min max r3 min max max min r5 r6 Box 5 Box 4 min max Box 3 min r7 24 25 26 27 28 r4 merge (r3,r4) Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 6/22 6/22 

What is essential to know? The behaviour of interactive scores 2 4 3 5 1 6 0 r2 r1 r3 r4 r5 r6 r7 time 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 r1 r2 Box 1 Box 2 Box 6 min max r3 min max max min r5 r6 Box 5 Box 4 min max Box 3 min r7 24 25 26 27 28 r4 merge (r3,r4) Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 6/22 6/22 

What is essential to know? The behaviour of interactive scores 2 4 3 5 1 6 0 r2 r1 r3 r4 r5 r6 r7 time 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 r1 r2 Box 1 Box 2 Box 6 min max r3 min max max min r5 r6 Box 5 Box 4 min max Box 3 min r7 24 25 26 27 28 r4 merge (r3,r4) Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 6/22 6/22 

What is essential to know? The behaviour of interactive scores 2 4 3 5 1 6 0 r2 r1 r3 r4 r5 r6 r7 time 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 r1 r2 Box 1 Box 2 Box 6 min max r3 min max max min r5 r6 Box 5 Box 4 min max Box 3 min r7 24 25 26 27 28 r4 merge (r3,r4) Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 6/22 6/22 

What is essential to know? The behaviour of interactive scores 2 4 3 5 1 6 0 r2 r1 r3 r4 r5 r6 r7 time 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 r1 r2 Box 1 Box 2 Box 6 min max r3 min max max min r5 r6 Box 5 Box 4 min max Box 3 min r7 24 25 26 27 28 r4 merge (r3,r4) Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 6/22 6/22 

What is essential to know? The behaviour of interactive scores 2 4 3 5 1 6 0 r2 r1 r3 r4 r5 r6 r7 time 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 r1 r2 Box 1 Box 2 Box 6 min max r3 min max max min r5 r6 Box 5 Box 4 min max Box 3 min r7 24 25 26 27 28 r4 merge (r3,r4) Boxes behave like rela- tions with an attached process! Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 6/22 6/22 

Execution model of interactive scores Modelling a rigid interval time fixed interval interactive interval duration minimum duration maximum duration interaction point enabled Declarations colset TIME = time; colset UNIT = unit; colset UT = UNIT timed; var dur : TIME; start duration TIME ()@+dur dur wait UT end () Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 7/22 7/22 

Execution model of interactive scores Modelling a static box start box start process end box stop process dur dur duration TIME start process start interval end interval fixed interval duration TIME end process Declarations colset TIME = time; colset UNIT = unit; var dur : TIME; Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 8/22 8/22 

Execution model of interactive scores Modelling a flexible interval Declarations colset TIME = time; colset UNIT = unit; var dmin : TIME; var dmax : TIME; start interval end interval 1 duration 1 TIME start interval 2 end max. interval duration 2 TIME dmin dmax-dmin dmin dmax end min. interval min. duration TIME max. duration TIME fixed interval fixed interval Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 9/22 9/22 

Execution model of interactive scores Listening events Declarations colset TIME = time; colset UNIT = unit; colset UT = UNIT timed; var ip_t : TIME; start stop end interact. point enabled ip_t ip_t TIME UT t3 ()@+1 time() t1 [ip_t < time()] t2 [ip_t = time()] Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 10/22 10/22 

Execution model of interactive scores Handling interaction points time fixed interval interactive interval duration minimum duration maximum duration interaction point enabled Declarations colset TIME = time; colset UNIT = unit; colset BOOL = bool; var e : BOOL; interact. point start get stop get end interval get IP ss se flexible interval start interval TIME max. duration TIME min. duration end max. interval end min. interval enable e false BOOL true if (e=true) then 1`() else empty Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 11/22 11/22 

Execution model of interactive scores Modelling an interactive box Declarations colset TIME = time; colset UNIT = unit; var dmin : TIME; var dmax : TIME; start box start process end box dmin min. duration TIME start process end process stop process TIME max. duration start interval TIME min. duration interact. point end interval interactive interval max. duration TIME dmax dmin dmax Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 12/22 12/22 

Execution model of interactive scores Synchronization of fixed intervals time fixed interval 1 fixed interval 2 merge fixed (1,2) end interval r5 fixed interval end interval r6 fixed interval TIME duration box D start box D fixed box Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 13/22 13/22 

Execution model of interactive scores Synchronization of interactive intervals interactive interval 1 interactive interval 2 merge interactive (1,2) min max min max min max interaction point enabled time TIME duration box C start box C fixed box end max. interval r3 end min. interval r3 flexible interval end max. interval r4 end min. interval r4 flexible interval get IP interact. point C start get stop get end get ss se enable e false BOOL true if (e=true) then 1`() else empty Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 14/22 14/22 

Extending the model for handling data Main idea • Multimedia streams are often cut into temporal frames to be carried from one process to another ; we model frames as coloured tokens. • Asynchronous functional composition (i.e., buffering the output of processes). • Modules for reading, appending, reversing audio files. • Implementation and simulation in CPN Tools2. 2http://cpntools.org Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 15/22 15/22 

Extending the model for handling data Main idea • Multimedia streams are often cut into temporal frames to be carried from one process to another ; we model frames as coloured tokens. • Asynchronous functional composition (i.e., buffering the output of processes). • Modules for reading, appending, reversing audio files. • Implementation and simulation in CPN Tools2. 2http://cpntools.org Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 15/22 15/22 

Extending the model for handling data Main idea • Multimedia streams are often cut into temporal frames to be carried from one process to another ; we model frames as coloured tokens. • Asynchronous functional composition (i.e., buffering the output of processes). • Modules for reading, appending, reversing audio files. • Implementation and simulation in CPN Tools2. 2http://cpntools.org Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 15/22 15/22 

Extending the model for handling data Main idea • Multimedia streams are often cut into temporal frames to be carried from one process to another ; we model frames as coloured tokens. • Asynchronous functional composition (i.e., buffering the output of processes). • Modules for reading, appending, reversing audio files. • Implementation and simulation in CPN Tools2. 2http://cpntools.org Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 15/22 15/22 

Extending the model for handling data Reading an audio file Declarations colset TIME = time; colset UNIT = unit; colset INT = int; colset BOOL = bool; colset DATA = INT; colset DURATION = TIME timed; colset FILE = product INT*DATA; var f_dur : TIME; var e : BOOL; var n_max : INT; var n : INT; var f : DATA; TIME frame duration start INT max number frames f_dur n_max f_dur get frame DURATION f_dur f_dur@+f_dur max num INT n_max n_max n_max file FILE (n,f) continue stop next frame INT n+1 n receive frame FILE [n <= n_max] (n,f) (n,f) n_max n_max 1 if (n = n_max) then 1`() else empty (n,f) end EOF wait sync next frame f_dur R E A D limit if (n = n_max) then 1`() else empty if (e = true) then 1`() else empty 2`() n frames read n-1 INT stop enabled BOOL output FILE e 1`false e 1`false true Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 16/22 16/22 

Extending the model for handling data Appending two audio files Declarations colset TIME = time; colset UNIT = unit; colset INT = int; colset DATA = INT; colset FILE = product INT*DATA; var n : INT; var n1 : INT; var n2 : INT; var i : INT; var f : DATA; TIME frame duration file 1 start file 1 INT max number frames file 1 stop file 1 file 1 output file 1 FILE end file 1 EOF file 1 next frame file 1 TIME frame duration file 2 start file 2 INT max number frames file 2 stop file 2 file 2 output file 2 FILE end file 2 EOF file 2 next frame file 2 FILE FILE output FILE index (n,f) 1 INT i i+1 i i+1 (i,f) (n,f) (i,f) next 1 INT n n+1 next 2 1 INT n n+1 frames read 1 INT frames read 2 INT read file read file total frames read INT n2 n1 n1+n2 stop append Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 17/22 17/22 

Extending the model for handling data Reversing an audio file Declarations colset TIME = time; colset UNIT = unit; colset INT = int; colset DATA = INT; colset FILE = product INT*DATA; var n : INT; var n_max : INT; var i : INT; var f : DATA; TIME frame duration start reverse INT max frames stop file temp output FILE end EOF next frame TIME frame duration start INT max number stop reverse file reverse output reverse FILE end reverse EOF reverse next frame reverse FILE FILE (n,f) index INT i i-1 next 1 INT n n+1 0 INT max number frames [i=0] (i,f) [i>0] i n_max R E V n_max n_max n_max frames read INT frames reversed INT read file read file Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 18/22 18/22 

Extending the model for handling data Example Append Files Read File 1 Read File 2 time (ms) max min min max Reverse File r1 (values) (values) (values) (param) (param) r2 r3 r4 r5 time (ms) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 f1 f2 f3 f4 f5 f1 f2 f3 f4 f5 28 29 30 31 32 min max f1 f2 f3 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 f4 f5 f6 f7 f8 54 55 56 57 58 f1 f2 f3 min max Read File 1 (process) Read File 1 (box) Read File 2 (process) Read File 2 (box) r3 Reverse File (process) Reverse File (box) r5 Append Files (process) Append Files (box) r4 Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 19/22 19/22 

Summary • Modular model in CPNs for executing interactive multimedia scores with the capability to handle complex data (i.e., data audio streams). • Representation of audio frames as coloured tokens. • Asynchronous functional composition of processes by adding inputs/outputs to boxes and passing values through intervals. • Modules for processing audio files such as reading, appending and reversing files. • Implementation and simulation of our model in CPN Tools. Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 20/22 20/22 

Summary • Modular model in CPNs for executing interactive multimedia scores with the capability to handle complex data (i.e., data audio streams). • Representation of audio frames as coloured tokens. • Asynchronous functional composition of processes by adding inputs/outputs to boxes and passing values through intervals. • Modules for processing audio files such as reading, appending and reversing files. • Implementation and simulation of our model in CPN Tools. Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 20/22 20/22 

Summary • Modular model in CPNs for executing interactive multimedia scores with the capability to handle complex data (i.e., data audio streams). • Representation of audio frames as coloured tokens. • Asynchronous functional composition of processes by adding inputs/outputs to boxes and passing values through intervals. • Modules for processing audio files such as reading, appending and reversing files. • Implementation and simulation of our model in CPN Tools. Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 20/22 20/22 

Summary • Modular model in CPNs for executing interactive multimedia scores with the capability to handle complex data (i.e., data audio streams). • Representation of audio frames as coloured tokens. • Asynchronous functional composition of processes by adding inputs/outputs to boxes and passing values through intervals. • Modules for processing audio files such as reading, appending and reversing files. • Implementation and simulation of our model in CPN Tools. Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 20/22 20/22 

Summary • Modular model in CPNs for executing interactive multimedia scores with the capability to handle complex data (i.e., data audio streams). • Representation of audio frames as coloured tokens. • Asynchronous functional composition of processes by adding inputs/outputs to boxes and passing values through intervals. • Modules for processing audio files such as reading, appending and reversing files. • Implementation and simulation of our model in CPN Tools. Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 20/22 20/22 

Future work • We are working on extending our model with conditionals and loops. • We plan to model new modules for audio processing (e.g., change the speed for reading audio files). • We plan to explore the verification of properties in scenarios (e.g., maximum number of processes that can be executed at the same time). Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 21/22 21/22 

Future work • We are working on extending our model with conditionals and loops. • We plan to model new modules for audio processing (e.g., change the speed for reading audio files). • We plan to explore the verification of properties in scenarios (e.g., maximum number of processes that can be executed at the same time). Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 21/22 21/22 

Future work • We are working on extending our model with conditionals and loops. • We plan to model new modules for audio processing (e.g., change the speed for reading audio files). • We plan to explore the verification of properties in scenarios (e.g., maximum number of processes that can be executed at the same time). Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 21/22 21/22 

Thank you for your attention E-mail: [email protected] Jaime Arias et al. (LaBRI) Modelling Data Processing for Interactive Scores Using Coloured Petri Nets 22/22 22/22 

Modelling Data Processing for Interactive Scores Using Coloured Petri Nets Jaime Arias Joint work with Myriam Desainte-Catherine and Camilo Rueda Laboratoire Bordelais de Recherche en Informatique (LaBRI) Université de Bordeaux 14th International Conference on Application of Concurrency to System Design Tunis, June 2014 1 
 

References Myriam Desainte-Catherine, Antoine Allombert, and Gérard Assayag. Towards a hybrid temporal paradigm for musical composition and performance: The case of musical interpretation. Computer Music Journal, 37(2):61–72, 2013. James F. Allen. Maintaining knowledge about temporal intervals. 26: 832–843. ISSN 0001-0782. doi: 10.1145/182.358434. Raphaël Marczak, Myriam Desainte-Catherine, and Antoine Allombert. Real-time temporal control of musical processes. In The Third International Conferences on Advances in Multimedia, MMEDIA 2011, pages 12–17, 2011. ISBN 978-1-61208-129-8. 

References Patrick Sénac, Pierre de Saqui-Sannes, and Roberto Willrich. Hierarchical time stream petri net: A model for hypermedia systems. In Giorgio De Michelis and Michel Diaz, editors, Application and Theory of Petri Nets 1995, volume 935 of Lecture Notes in Computer Science, pages 451–470. Springer Berlin Heidelberg, 1995. ISBN 978-3-540-60029-9. doi: 10.1007/3-540-60029-9_54. URL http://dx.doi.org/10.1007/3-540-60029-9_54.