Formal Models to Compose and Execute Multimedia Interactive Scores

Transcript

1. Formal Models to Compose and Execute Multimedia Interactive Scores Jaime Arias, Myriam Desainte-Catherine and Camilo Rueda Laboratoire Bordelais de Recherche en Informatique (LaBRI) Université de Bordeaux Research Seminar - DECC - Javeriana, Cali October, 2014 1 
   
2. Motivation Interactive scores Composers Multimedia Interaction Interpretation Interactive Multimedia Score A B Temporal Relations Hierarchy Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 1/1 1/1
3. Motivation Interactive scores Texture Structure Rigid Interval Semi-Flexible Interval Interaction Point OSC Messages Multimedia Processes Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 2/1 2/1
4. Motivation Interactive scores A wide range of applications Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 3/1 3/1
5. Motivation Interactive scores Conditionals Loops Data Stream Some Limitations Visual Representation Automatic Verification Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 4/1 4/1
6. This talk is about … We present some approaches developed to overcome the limitations of the current model of interactive scores. Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 5/1 5/1
7. Outline Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 6/1 6/1
8. Outline Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 7/1 7/1
9. Coloured Petri Nets Introduction Declarations colset TIME = time; colset UNIT = unit; colset INT = int; colset DATA = int; colset DURATION = TIME timed; colset FILE = product INT*DATA; var f_dur : TIME; var n_max : INT; var n : INT; var f : DATA; TIME frame duration start f_dur f_dur get frame DURATION f_dur f_dur@+f_dur max num INT n_max n_max file FILE (n,f) next frame INT n+1 n receive frame FILE [n <= n_max] (n,f) n_max n_max 1 if (n = n_max) then 1`() else empty end EOF R E A D if (n = n_max) then 1`() else empty output FILE 5 1,5 2,9 3,6 4,1 (n,f) (n,f) 4 TIME = 0 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 8/1 8/1
10. Coloured Petri Nets Introduction Declarations colset TIME = time; colset UNIT = unit; colset INT = int; colset DATA = int; colset DURATION = TIME timed; colset FILE = product INT*DATA; var f_dur : TIME; var n_max : INT; var n : INT; var f : DATA; TIME frame duration start f_dur f_dur get frame DURATION f_dur f_dur@+f_dur max num INT n_max n_max file FILE (n,f) next frame INT n+1 n receive frame FILE [n <= n_max] (n,f) n_max n_max 1 if (n = n_max) then 1`() else empty end EOF R E A D if (n = n_max) then 1`() else empty output FILE 5 1,5 2,9 3,6 4,1 (n,f) (n,f) 4 TIME = 0 start TIME frame durat ion max num INT outp ut FILE end EOF file FILE Hierarchy Time Data SML Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 8/1 8/1
11. Coloured Petri Nets Introduction Declarations colset TIME = time; colset UNIT = unit; colset INT = int; colset DATA = int; colset DURATION = TIME timed; colset FILE = product INT*DATA; var f_dur : TIME; var n_max : INT; var n : INT; var f : DATA; TIME frame duration start f_dur f_dur get frame DURATION f_dur f_dur@+f_dur max num INT n_max n_max file FILE (n,f) next frame INT n+1 n receive frame FILE [n <= n_max] (n,f) n_max n_max 1 if (n = n_max) then 1`() else empty end EOF R E A D if (n = n_max) then 1`() else empty output FILE 5 1,5 2,9 3,6 4,1 (n,f) (n,f) 4 TIME = 0 start TIME frame durat ion max num INT outp ut FILE end EOF file FILE Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 8/1 8/1
12. Coloured Petri Nets Introduction Declarations colset TIME = time; colset UNIT = unit; colset INT = int; colset DATA = int; colset DURATION = TIME timed; colset FILE = product INT*DATA; var f_dur : TIME; var n_max : INT; var n : INT; var f : DATA; TIME frame duration start f_dur f_dur get frame DURATION f_dur f_dur@+f_dur max num INT n_max n_max file FILE (n,f) next frame INT n+1 n receive frame FILE [n <= n_max] (n,f) n_max n_max 2 if (n = n_max) then 1`() else empty end EOF R E A D if (n = n_max) then 1`() else empty output FILE 5@5 1,5 2,9 3,6 4,1 (n,f) (n,f) 4 TIME = 0 start TIME frame durat ion max num INT outp ut FILE end EOF file FILE Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 8/1 8/1
13. Coloured Petri Nets Introduction Declarations colset TIME = time; colset UNIT = unit; colset INT = int; colset DATA = int; colset DURATION = TIME timed; colset FILE = product INT*DATA; var f_dur : TIME; var n_max : INT; var n : INT; var f : DATA; TIME frame duration start f_dur f_dur get frame DURATION f_dur f_dur@+f_dur max num INT n_max n_max file FILE (n,f) next frame INT n+1 n receive frame FILE [n <= n_max] (n,f) n_max n_max 2 if (n = n_max) then 1`() else empty end EOF R E A D if (n = n_max) then 1`() else empty output FILE 5@5 1,5 2,9 3,6 4,1 (n,f) (n,f) 4 TIME = 0 start TIME frame durat ion max num INT outp ut FILE end EOF file FILE Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 8/1 8/1
14. Coloured Petri Nets Introduction Declarations colset TIME = time; colset UNIT = unit; colset INT = int; colset DATA = int; colset DURATION = TIME timed; colset FILE = product INT*DATA; var f_dur : TIME; var n_max : INT; var n : INT; var f : DATA; TIME frame duration start f_dur f_dur get frame DURATION f_dur f_dur@+f_dur max num INT n_max n_max file FILE (n,f) next frame INT n+1 n receive frame FILE [n <= n_max] (n,f) n_max n_max 3 if (n = n_max) then 1`() else empty end EOF R E A D if (n = n_max) then 1`() else empty output FILE 5@ 10 1,5 2,9 3,6 4,1 (n,f) (n,f) 4 TIME = 5 start TIME frame durat ion max num INT outp ut FILE end EOF file FILE Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 8/1 8/1
15. Coloured Petri Nets Introduction Declarations colset TIME = time; colset UNIT = unit; colset INT = int; colset DATA = int; colset DURATION = TIME timed; colset FILE = product INT*DATA; var f_dur : TIME; var n_max : INT; var n : INT; var f : DATA; TIME frame duration start f_dur f_dur get frame DURATION f_dur f_dur@+f_dur max num INT n_max n_max file FILE (n,f) next frame INT n+1 n receive frame FILE [n <= n_max] (n,f) n_max n_max 3 if (n = n_max) then 1`() else empty end EOF R E A D if (n = n_max) then 1`() else empty output FILE 5@ 10 1,5 2,9 3,6 4,1 (n,f) (n,f) 4 TIME = 5 start TIME frame durat ion max num INT outp ut FILE end EOF file FILE Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 8/1 8/1
16. Coloured Petri Nets Summary • It allows to represent and handle complex data in interactive scores (coloured tokens). • Modules for processing audio files such as reading, appending and reversing files were created. • The model can be easily extended. • CPN tool was used to specify and simulate our model. • The notion of asynchronous functional composition was introduced (buffering). • It is possible to model conditionals and loops, but the resulting network would be very large and complex. • CPN tool is only available for Windows and its interface is very unfriendly. Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 9/1 9/1
17. Outline Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 10/1 10/1
18. Synchronous Programming Language Introduction The Synchrony Hypothesis Signal Broadcasting Concurrency … … … Logical Time Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 11/1 11/1
19. Synchronous Programming Language Introduction Process A Process B t = 10 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 12/1 12/1
20. Synchronous Programming Language Introduction Process A Process B Hey!, your mom is waiting for you! t = 18 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 12/1 12/1
21. Synchronous Programming Language Introduction Process A Process B Hey!, your mom is waiting for you! t = 18 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 12/1 12/1
22. Reactive Programming Language • Model introduced by Frédéric Boussinot and Robert de Simone1. • Synchronous model - immediate reaction to signal absences = free of causality problems. • ReactiveML enables dynamic creation of processes. A B 1Boussinot, F.; de Simone, R., ”The SL synchronous language,” Software Engineering, IEEE Transactions on , vol.22, no.4, pp.256,266, Apr 1996 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 13/1 13/1
23. Reactive Programming Language Our Approach Interpreter RML Observer RML Pure Data / OSC Messages INScore Events OSC Messages OSC Client OSC Server OSC Messages Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 14/1 14/1
24. Reactive Programming Language Our Approach Interpreter RML Observer RML Pure Data / MAX/MSP OSC Messages INScore Events OSC Messages OSC Client OSC Server OSC Messages Box 1 Box 2 Rel 1 Rel2 … … Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 14/1 14/1
25. Reactive Programming Language Summary • The synchronous model made easy to describe the behaviour of interactive scores. • The notion of hierarchy was modelled. • The implementation was simple and small. • The visual feedback of execution was improved using INScore. • Conditionals and loop can be easily implemented. • ReactiveML allows the dynamic creation of processes, but it does not support the real-time execution of programs. Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 15/1 15/1
26. Outline Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 16/1 16/1
27. Timed Automata Introduction Finite-state machine Multiple Clocks Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 17/1 17/1
28. Timed Automata Introduction Finite-state machine Multiple Clocks Process 1 Process 2 c1 c1 c2 c2 Communication Channels (CCS) Network of TA x = 10 x = 30 t = 1 Shared Variables Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 18/1 18/1
29. Timed Automata Introduction Conditional Point idle wait C stop C vol < 10 , ⌧, ; vol 10 , start ! , ; true, event?, ; true , stop ? , ; urg = true,start!,; true ,stop? ,; true, launch?, ; urg = false,skip!,; urg = true Event wait stop x  4 vol = 0 x = 4 , event ! , { vol = 12} x = 0 Flexible Interval idle min max stop t = 0, ⌧, {t = 0} t  3 t = 3, launch!, ; t  5 t = 5 , stop ! , { t = 0} true, start?, {t = 0} t = 0 t  0 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 19/1 19/1
30. Timed Automata Introduction Conditional Point idle wait C stop C vol < 10 , ⌧, ; vol 10 , start ! , ; true, event?, ; true , stop ? , ; urg = true,start!,; true ,stop? ,; true, launch?, ; urg = false,skip!,; urg = true Event wait stop x  4 vol = 0 x = 4 , event ! , { vol = 12} x = 0 Flexible Interval idle min max stop t = 0, ⌧, {t = 0} t  3 t = 3, launch!, ; t  5 t = 5 , stop ! , { t = 0} true, start?, {t = 0} t = 0 t  0 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 19/1 19/1
31. Timed Automata Introduction Conditional Point idle wait C stop C vol < 10 , ⌧, ; vol 10 , start ! , ; true, event?, ; true , stop ? , ; urg = true,start!,; true ,stop? ,; true, launch?, ; urg = false,skip!,; urg = true Event wait stop x  4 vol = 0 x = 4 , event ! , { vol = 12} x = 3 Flexible Interval idle min max stop t = 0, ⌧, {t = 0} t  3 t = 3, launch!, ; t  5 t = 5 , stop ! , { t = 0} true, start?, {t = 0} t = 3 t  0 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 19/1 19/1
32. Timed Automata Introduction Conditional Point idle wait C stop C vol < 10 , ⌧, ; vol 10 , start ! , ; true, event?, ; true , stop ? , ; urg = true,start!,; true ,stop? ,; true, launch?, ; urg = false,skip!,; urg = true Event wait stop x  4 vol = 0 x = 4 , event ! , { vol = 12} x = 3 Flexible Interval idle min max stop t = 0, ⌧, {t = 0} t  3 t = 3, launch!, ; t  5 t = 5 , stop ! , { t = 0} true, start?, {t = 0} t = 3 t  0 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 19/1 19/1
33. Timed Automata Introduction Conditional Point idle wait C stop C vol < 10 , ⌧, ; vol 10 , start ! , ; true, event?, ; true , stop ? , ; urg = true,start!,; true ,stop? ,; true, launch?, ; urg = false,skip!,; urg = true Event wait stop x  4 vol = 0 x = 4 , event ! , { vol = 12} x = 4 Flexible Interval idle min max stop t = 0, ⌧, {t = 0} t  3 t = 3, launch!, ; t  5 t = 5 , stop ! , { t = 0} true, start?, {t = 0} t = 4 t  0 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 19/1 19/1
34. Timed Automata Introduction Conditional Point idle wait C stop C vol < 10 , ⌧, ; vol 10 , start ! , ; true, event?, ; true , stop ? , ; urg = true,start!,; true ,stop? ,; true, launch?, ; urg = false,skip!,; urg = true Event wait stop x  4 vol = 12 x = 4 , event ! , { vol = 12} x = 4 Flexible Interval idle min max stop t = 0, ⌧, {t = 0} t  3 t = 3, launch!, ; t  5 t = 5 , stop ! , { t = 0} true, start?, {t = 0} t = 4 t  0 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 19/1 19/1
35. Timed Automata Introduction Conditional Point idle wait C stop C vol < 10 , ⌧, ; vol 10 , start ! , ; true, event?, ; true , stop ? , ; urg = true,start!,; true ,stop? ,; true, launch?, ; urg = false,skip!,; urg = true Event wait stop x  4 vol = 12 x = 4 , event ! , { vol = 12} x = 4 Flexible Interval idle min max stop t = 0, ⌧, {t = 0} t  3 t = 3, launch!, ; t  5 t = 5 , stop ! , { t = 0} true, start?, {t = 0} t = 4 t  0 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 19/1 19/1
36. Timed Automata Our Approach Composition Verification Interpretation Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 20/1 20/1
37. Timed Automata Our Approach Composition Verification Interpretation Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 20/1 20/1
38. Timed Automata Our Approach Composition IS2UPP Verification Property Checking Interpretation Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 20/1 20/1
39. Timed Automata Our Approach Composition IS2UPP Verification Property Checking C/C++ Code UPP2C Interpretation Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 20/1 20/1
40. Timed Automata Our Approach Composition IS2UPP Verification Property Checking C/C++ Code VHDL/System Verilog Code UPP2C UPP2HDL Interpretation Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 20/1 20/1
41. Timed Automata Summary • Timed automata is well-suited to model interactive scores. • Unlike CPN tools, UPPAAL is cross-platform and maintained. • Asynchronous and synchronous communication improve the complexity of the model and the modularity. • The model checking algorithm is efficient. Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 21/1 21/1
42. Outline Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 22/1 22/1
43. Domain-Specific Programming Language ReactiveIS Program State hS, !, i Temporal Transition hS, !, i Instantaneous Transition The Synchrony Hypothesis Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 23/1 23/1
44. Domain-Specific Programming Language ReactiveIS
          
                     
      !    
      !
    
          
            " " #$   %&$'%"   %&$'%" (( 
          
            " " #$   %&$'%   %&$'% (( 
          
           
     " " #$   % %"   % %" (( 
          
     "   ) %       
       #$   % %   % % ((  Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 24/1 24/1
45. Domain-Specific Programming Language ReactiveIS Dates start: 0 stop: ! Dates start: 3 stop: ! B Dates start: 1 stop: 3 A Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 25/1 25/1
46. Domain-Specific Programming Language ReactiveIS Dates start: 0 stop: ! Dates start: 3 stop: ! B Dates start: 1 stop: 3 A Dates start: 0 stop: ! Dates start: 3 stop: 5 B Dates start: 1 stop: 3 A stop(S,B,5) Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 25/1 25/1
47. Domain-Specific Programming Language ReactiveIS Dates start: 0 stop: ! Dates start: 3 stop: ! B Dates start: 1 stop: 3 A Dates start: 0 stop: ! Dates start: 3 stop: 5 B Dates start: 1 stop: 3 A stop(S,B,5) Dates start: 0 stop: ! Dates start: 8 stop: ! C Dates start: 3 stop: 5 B Dates start: 1 stop: 3 A start(S,C,8) Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 25/1 25/1
48. Domain-Specific Programming Language ReactiveIS hP, S, I, O, ti Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 26/1 26/1
49. Domain-Specific Programming Language ReactiveIS hP, S, I, O, ti Program Tree Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 26/1 26/1
50. Domain-Specific Programming Language ReactiveIS hP, S, I, O, ti Program Tree Current State Tree Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 26/1 26/1
51. Domain-Specific Programming Language ReactiveIS hP, S, I, O, ti Program Tree Current State Tree Current Input Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 26/1 26/1
52. Domain-Specific Programming Language ReactiveIS hP, S, I, O, ti Program Tree Current State Tree Current Output Current Input Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 26/1 26/1
53. Domain-Specific Programming Language ReactiveIS hP, S, I, O, ti Program Tree Current State Tree Current Output Current Input Current Time Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 26/1 26/1
54. Domain-Specific Programming Language RSTART p ∈ canStart(S, P) ⟨P, S, I, O, t⟩ |= cs (target(p)) ⟨P, S, I, O, t⟩ −→ ⟨P, start(S, p, t), I, O ∪ {ms (target(p))}, t⟩ RSTOP p ∈ canStop(S) ⟨P, S, I, O, t⟩ |= ce(target(p)) ⟨P, S, I, O, t⟩ −→ ⟨P, stop(S, p, t), I, O ∪ {me(target(p))}, t⟩ RTIME ⟨P, S, I, ∅, t⟩ −→∗ ⟨P, S′, I, O, t⟩ ̸ −→ ⟨P, S, t⟩ I,O =⇒ ⟨P, S′, t + 1⟩ Table: Rules for the internal reduction −→ and the observable reduction =⇒. Notation γ ̸ −→ means that there is no γ′ such that γ −→ γ′. Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 27/1 27/1
55. Domain-Specific Programming Language Summary • ReactiveIS is a programming language for the specification and execution of interactive scores. • Tree structures are used to represent programs and execution states. • Trees provide a clear and intuitive representation of scores and the evolution of their execution. • Conditions defined as formulas of first-order logic allow to define more complex temporal relations in scores. • The operational semantics can be easily implemented in a programming language. Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 28/1 28/1
56. Outline Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 29/1 29/1
57. Concluding Remarks • Different formalisms can be used to model interactive scores. Thanks to their mathematical basis, we can reason about the models and the composed scenarios. • Each model has advantages and disadvantages over the others. For example, real-time, automatic verification, expressiveness, …. • It is possible to develop a framework to compose, verify, simulate and interpret interactive scores. • The modelled scenarios can be translated into executable code (only on paper). Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 30/1 30/1
58. Concluding Remarks Future Work • Add conditionals and loops to the models (timed automata). • Verify some properties of scenarios (e.g., playability). • Develop verification methods and tools for ReactiveIS. • Implement the synchronous approach in Lustre in order to handle streams in real-time. • Implement the compilers that are necessary for the proposed framework. • Improve the graphical interface in INScore. Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 31/1 31/1
59. Thank you!
60. Formal Models to Compose and Execute Multimedia Interactive Scores Jaime Arias, Myriam Desainte-Catherine and Camilo Rueda Laboratoire Bordelais de Recherche en Informatique (LaBRI) Université de Bordeaux Research Seminar - DECC - Javeriana, Cali October, 2014 1