Share
Embed
Formal Models to Compose and Execute Multimedia Interactive Scores
Jaime Arias Almeida
October 02, 2014
Transcript
60 slides
-
Formal Models to Compose and Execute Multimedia Interactive Scores Jaime
Arias, Myriam Desainte-Catherine and Camilo Rueda Laboratoire Bordelais de Recherche en Informatique (LaBRI) Université de Bordeaux Research Seminar - DECC - Javeriana, Cali October, 2014 1 -
Motivation Interactive scores Composers Multimedia Interaction Interpretation Interactive Multimedia Score
A B Temporal Relations Hierarchy Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 1/1 1/1 -
Motivation Interactive scores Texture Structure Rigid Interval Semi-Flexible Interval Interaction
Point OSC Messages Multimedia Processes Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 2/1 2/1 -
Motivation Interactive scores A wide range of applications Arias, Desainte-Catherine
and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 3/1 3/1 -
Motivation Interactive scores Conditionals Loops Data Stream Some Limitations Visual
Representation Automatic Verification Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 4/1 4/1 -
This talk is about … We present some approaches developed
to overcome the limitations of the current model of interactive scores. Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 5/1 5/1 -
Outline Arias, Desainte-Catherine and Rueda Formal Models to Compose and
Execute Multimedia Interactive Scores 6/1 6/1 -
Outline Arias, Desainte-Catherine and Rueda Formal Models to Compose and
Execute Multimedia Interactive Scores 7/1 7/1 -
Coloured Petri Nets Introduction Declarations colset TIME = time; colset
UNIT = unit; colset INT = int; colset DATA = int; colset DURATION = TIME timed; colset FILE = product INT*DATA; var f_dur : TIME; var n_max : INT; var n : INT; var f : DATA; TIME frame duration start f_dur f_dur get frame DURATION f_dur f_dur@+f_dur max num INT n_max n_max file FILE (n,f) next frame INT n+1 n receive frame FILE [n <= n_max] (n,f) n_max n_max 1 if (n = n_max) then 1`() else empty end EOF R E A D if (n = n_max) then 1`() else empty output FILE 5 1,5 2,9 3,6 4,1 (n,f) (n,f) 4 TIME = 0 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 8/1 8/1 -
Coloured Petri Nets Introduction Declarations colset TIME = time; colset
UNIT = unit; colset INT = int; colset DATA = int; colset DURATION = TIME timed; colset FILE = product INT*DATA; var f_dur : TIME; var n_max : INT; var n : INT; var f : DATA; TIME frame duration start f_dur f_dur get frame DURATION f_dur f_dur@+f_dur max num INT n_max n_max file FILE (n,f) next frame INT n+1 n receive frame FILE [n <= n_max] (n,f) n_max n_max 1 if (n = n_max) then 1`() else empty end EOF R E A D if (n = n_max) then 1`() else empty output FILE 5 1,5 2,9 3,6 4,1 (n,f) (n,f) 4 TIME = 0 start TIME frame durat ion max num INT outp ut FILE end EOF file FILE Hierarchy Time Data SML Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 8/1 8/1 -
Coloured Petri Nets Introduction Declarations colset TIME = time; colset
UNIT = unit; colset INT = int; colset DATA = int; colset DURATION = TIME timed; colset FILE = product INT*DATA; var f_dur : TIME; var n_max : INT; var n : INT; var f : DATA; TIME frame duration start f_dur f_dur get frame DURATION f_dur f_dur@+f_dur max num INT n_max n_max file FILE (n,f) next frame INT n+1 n receive frame FILE [n <= n_max] (n,f) n_max n_max 1 if (n = n_max) then 1`() else empty end EOF R E A D if (n = n_max) then 1`() else empty output FILE 5 1,5 2,9 3,6 4,1 (n,f) (n,f) 4 TIME = 0 start TIME frame durat ion max num INT outp ut FILE end EOF file FILE Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 8/1 8/1 -
Coloured Petri Nets Introduction Declarations colset TIME = time; colset
UNIT = unit; colset INT = int; colset DATA = int; colset DURATION = TIME timed; colset FILE = product INT*DATA; var f_dur : TIME; var n_max : INT; var n : INT; var f : DATA; TIME frame duration start f_dur f_dur get frame DURATION f_dur f_dur@+f_dur max num INT n_max n_max file FILE (n,f) next frame INT n+1 n receive frame FILE [n <= n_max] (n,f) n_max n_max 2 if (n = n_max) then 1`() else empty end EOF R E A D if (n = n_max) then 1`() else empty output FILE 5@5 1,5 2,9 3,6 4,1 (n,f) (n,f) 4 TIME = 0 start TIME frame durat ion max num INT outp ut FILE end EOF file FILE Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 8/1 8/1 -
Coloured Petri Nets Introduction Declarations colset TIME = time; colset
UNIT = unit; colset INT = int; colset DATA = int; colset DURATION = TIME timed; colset FILE = product INT*DATA; var f_dur : TIME; var n_max : INT; var n : INT; var f : DATA; TIME frame duration start f_dur f_dur get frame DURATION f_dur f_dur@+f_dur max num INT n_max n_max file FILE (n,f) next frame INT n+1 n receive frame FILE [n <= n_max] (n,f) n_max n_max 2 if (n = n_max) then 1`() else empty end EOF R E A D if (n = n_max) then 1`() else empty output FILE 5@5 1,5 2,9 3,6 4,1 (n,f) (n,f) 4 TIME = 0 start TIME frame durat ion max num INT outp ut FILE end EOF file FILE Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 8/1 8/1 -
Coloured Petri Nets Introduction Declarations colset TIME = time; colset
UNIT = unit; colset INT = int; colset DATA = int; colset DURATION = TIME timed; colset FILE = product INT*DATA; var f_dur : TIME; var n_max : INT; var n : INT; var f : DATA; TIME frame duration start f_dur f_dur get frame DURATION f_dur f_dur@+f_dur max num INT n_max n_max file FILE (n,f) next frame INT n+1 n receive frame FILE [n <= n_max] (n,f) n_max n_max 3 if (n = n_max) then 1`() else empty end EOF R E A D if (n = n_max) then 1`() else empty output FILE 5@ 10 1,5 2,9 3,6 4,1 (n,f) (n,f) 4 TIME = 5 start TIME frame durat ion max num INT outp ut FILE end EOF file FILE Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 8/1 8/1 -
Coloured Petri Nets Introduction Declarations colset TIME = time; colset
UNIT = unit; colset INT = int; colset DATA = int; colset DURATION = TIME timed; colset FILE = product INT*DATA; var f_dur : TIME; var n_max : INT; var n : INT; var f : DATA; TIME frame duration start f_dur f_dur get frame DURATION f_dur f_dur@+f_dur max num INT n_max n_max file FILE (n,f) next frame INT n+1 n receive frame FILE [n <= n_max] (n,f) n_max n_max 3 if (n = n_max) then 1`() else empty end EOF R E A D if (n = n_max) then 1`() else empty output FILE 5@ 10 1,5 2,9 3,6 4,1 (n,f) (n,f) 4 TIME = 5 start TIME frame durat ion max num INT outp ut FILE end EOF file FILE Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 8/1 8/1 -
Coloured Petri Nets Summary • It allows to represent and
handle complex data in interactive scores (coloured tokens). • Modules for processing audio files such as reading, appending and reversing files were created. • The model can be easily extended. • CPN tool was used to specify and simulate our model. • The notion of asynchronous functional composition was introduced (buffering). • It is possible to model conditionals and loops, but the resulting network would be very large and complex. • CPN tool is only available for Windows and its interface is very unfriendly. Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 9/1 9/1 -
Outline Arias, Desainte-Catherine and Rueda Formal Models to Compose and
Execute Multimedia Interactive Scores 10/1 10/1 -
Synchronous Programming Language Introduction The Synchrony Hypothesis Signal Broadcasting Concurrency
… … … Logical Time Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 11/1 11/1 -
Synchronous Programming Language Introduction Process A Process B t =
10 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 12/1 12/1 -
Synchronous Programming Language Introduction Process A Process B Hey!, your
mom is waiting for you! t = 18 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 12/1 12/1 -
Synchronous Programming Language Introduction Process A Process B Hey!, your
mom is waiting for you! t = 18 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 12/1 12/1 -
Reactive Programming Language • Model introduced by Frédéric Boussinot and
Robert de Simone1. • Synchronous model - immediate reaction to signal absences = free of causality problems. • ReactiveML enables dynamic creation of processes. A B 1Boussinot, F.; de Simone, R., ”The SL synchronous language,” Software Engineering, IEEE Transactions on , vol.22, no.4, pp.256,266, Apr 1996 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 13/1 13/1 -
Reactive Programming Language Our Approach Interpreter RML Observer RML Pure
Data / OSC Messages INScore Events OSC Messages OSC Client OSC Server OSC Messages Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 14/1 14/1 -
Reactive Programming Language Our Approach Interpreter RML Observer RML Pure
Data / MAX/MSP OSC Messages INScore Events OSC Messages OSC Client OSC Server OSC Messages Box 1 Box 2 Rel 1 Rel2 … … Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 14/1 14/1 -
Reactive Programming Language Summary • The synchronous model made easy
to describe the behaviour of interactive scores. • The notion of hierarchy was modelled. • The implementation was simple and small. • The visual feedback of execution was improved using INScore. • Conditionals and loop can be easily implemented. • ReactiveML allows the dynamic creation of processes, but it does not support the real-time execution of programs. Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 15/1 15/1 -
Outline Arias, Desainte-Catherine and Rueda Formal Models to Compose and
Execute Multimedia Interactive Scores 16/1 16/1 -
Timed Automata Introduction Finite-state machine Multiple Clocks Arias, Desainte-Catherine and
Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 17/1 17/1 -
Timed Automata Introduction Finite-state machine Multiple Clocks Process 1 Process
2 c1 c1 c2 c2 Communication Channels (CCS) Network of TA x = 10 x = 30 t = 1 Shared Variables Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 18/1 18/1 -
Timed Automata Introduction Conditional Point idle wait C stop C
vol < 10 , ⌧, ; vol 10 , start ! , ; true, event?, ; true , stop ? , ; urg = true,start!,; true ,stop? ,; true, launch?, ; urg = false,skip!,; urg = true Event wait stop x 4 vol = 0 x = 4 , event ! , { vol = 12} x = 0 Flexible Interval idle min max stop t = 0, ⌧, {t = 0} t 3 t = 3, launch!, ; t 5 t = 5 , stop ! , { t = 0} true, start?, {t = 0} t = 0 t 0 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 19/1 19/1 -
Timed Automata Introduction Conditional Point idle wait C stop C
vol < 10 , ⌧, ; vol 10 , start ! , ; true, event?, ; true , stop ? , ; urg = true,start!,; true ,stop? ,; true, launch?, ; urg = false,skip!,; urg = true Event wait stop x 4 vol = 0 x = 4 , event ! , { vol = 12} x = 0 Flexible Interval idle min max stop t = 0, ⌧, {t = 0} t 3 t = 3, launch!, ; t 5 t = 5 , stop ! , { t = 0} true, start?, {t = 0} t = 0 t 0 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 19/1 19/1 -
Timed Automata Introduction Conditional Point idle wait C stop C
vol < 10 , ⌧, ; vol 10 , start ! , ; true, event?, ; true , stop ? , ; urg = true,start!,; true ,stop? ,; true, launch?, ; urg = false,skip!,; urg = true Event wait stop x 4 vol = 0 x = 4 , event ! , { vol = 12} x = 3 Flexible Interval idle min max stop t = 0, ⌧, {t = 0} t 3 t = 3, launch!, ; t 5 t = 5 , stop ! , { t = 0} true, start?, {t = 0} t = 3 t 0 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 19/1 19/1 -
Timed Automata Introduction Conditional Point idle wait C stop C
vol < 10 , ⌧, ; vol 10 , start ! , ; true, event?, ; true , stop ? , ; urg = true,start!,; true ,stop? ,; true, launch?, ; urg = false,skip!,; urg = true Event wait stop x 4 vol = 0 x = 4 , event ! , { vol = 12} x = 3 Flexible Interval idle min max stop t = 0, ⌧, {t = 0} t 3 t = 3, launch!, ; t 5 t = 5 , stop ! , { t = 0} true, start?, {t = 0} t = 3 t 0 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 19/1 19/1 -
Timed Automata Introduction Conditional Point idle wait C stop C
vol < 10 , ⌧, ; vol 10 , start ! , ; true, event?, ; true , stop ? , ; urg = true,start!,; true ,stop? ,; true, launch?, ; urg = false,skip!,; urg = true Event wait stop x 4 vol = 0 x = 4 , event ! , { vol = 12} x = 4 Flexible Interval idle min max stop t = 0, ⌧, {t = 0} t 3 t = 3, launch!, ; t 5 t = 5 , stop ! , { t = 0} true, start?, {t = 0} t = 4 t 0 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 19/1 19/1 -
Timed Automata Introduction Conditional Point idle wait C stop C
vol < 10 , ⌧, ; vol 10 , start ! , ; true, event?, ; true , stop ? , ; urg = true,start!,; true ,stop? ,; true, launch?, ; urg = false,skip!,; urg = true Event wait stop x 4 vol = 12 x = 4 , event ! , { vol = 12} x = 4 Flexible Interval idle min max stop t = 0, ⌧, {t = 0} t 3 t = 3, launch!, ; t 5 t = 5 , stop ! , { t = 0} true, start?, {t = 0} t = 4 t 0 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 19/1 19/1 -
Timed Automata Introduction Conditional Point idle wait C stop C
vol < 10 , ⌧, ; vol 10 , start ! , ; true, event?, ; true , stop ? , ; urg = true,start!,; true ,stop? ,; true, launch?, ; urg = false,skip!,; urg = true Event wait stop x 4 vol = 12 x = 4 , event ! , { vol = 12} x = 4 Flexible Interval idle min max stop t = 0, ⌧, {t = 0} t 3 t = 3, launch!, ; t 5 t = 5 , stop ! , { t = 0} true, start?, {t = 0} t = 4 t 0 Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 19/1 19/1 -
Timed Automata Our Approach Composition Verification Interpretation Arias, Desainte-Catherine and
Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 20/1 20/1 -
Timed Automata Our Approach Composition Verification Interpretation Arias, Desainte-Catherine and
Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 20/1 20/1 -
Timed Automata Our Approach Composition IS2UPP Verification Property Checking Interpretation
Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 20/1 20/1 -
Timed Automata Our Approach Composition IS2UPP Verification Property Checking C/C++
Code UPP2C Interpretation Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 20/1 20/1 -
Timed Automata Our Approach Composition IS2UPP Verification Property Checking C/C++
Code VHDL/System Verilog Code UPP2C UPP2HDL Interpretation Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 20/1 20/1 -
Timed Automata Summary • Timed automata is well-suited to model
interactive scores. • Unlike CPN tools, UPPAAL is cross-platform and maintained. • Asynchronous and synchronous communication improve the complexity of the model and the modularity. • The model checking algorithm is efficient. Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 21/1 21/1 -
Outline Arias, Desainte-Catherine and Rueda Formal Models to Compose and
Execute Multimedia Interactive Scores 22/1 22/1 -
Domain-Specific Programming Language ReactiveIS Program State hS, !, i Temporal
Transition hS, !, i Instantaneous Transition The Synchrony Hypothesis Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 23/1 23/1 -
Domain-Specific Programming Language ReactiveIS
! ! " " #$ %&$'%" %&$'%" (( " " #$ %&$'% %&$'% (( " " #$ % %" % %" (( " ) % #$ % % % % (( Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 24/1 24/1 -
Domain-Specific Programming Language ReactiveIS Dates start: 0 stop: ! Dates
start: 3 stop: ! B Dates start: 1 stop: 3 A Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 25/1 25/1 -
Domain-Specific Programming Language ReactiveIS Dates start: 0 stop: ! Dates
start: 3 stop: ! B Dates start: 1 stop: 3 A Dates start: 0 stop: ! Dates start: 3 stop: 5 B Dates start: 1 stop: 3 A stop(S,B,5) Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 25/1 25/1 -
Domain-Specific Programming Language ReactiveIS Dates start: 0 stop: ! Dates
start: 3 stop: ! B Dates start: 1 stop: 3 A Dates start: 0 stop: ! Dates start: 3 stop: 5 B Dates start: 1 stop: 3 A stop(S,B,5) Dates start: 0 stop: ! Dates start: 8 stop: ! C Dates start: 3 stop: 5 B Dates start: 1 stop: 3 A start(S,C,8) Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 25/1 25/1 -
Domain-Specific Programming Language ReactiveIS hP, S, I, O, ti Arias,
Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 26/1 26/1 -
Domain-Specific Programming Language ReactiveIS hP, S, I, O, ti Program
Tree Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 26/1 26/1 -
Domain-Specific Programming Language ReactiveIS hP, S, I, O, ti Program
Tree Current State Tree Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 26/1 26/1 -
Domain-Specific Programming Language ReactiveIS hP, S, I, O, ti Program
Tree Current State Tree Current Input Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 26/1 26/1 -
Domain-Specific Programming Language ReactiveIS hP, S, I, O, ti Program
Tree Current State Tree Current Output Current Input Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 26/1 26/1 -
Domain-Specific Programming Language ReactiveIS hP, S, I, O, ti Program
Tree Current State Tree Current Output Current Input Current Time Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 26/1 26/1 -
Domain-Specific Programming Language RSTART p ∈ canStart(S, P) ⟨P, S,
I, O, t⟩ |= cs (target(p)) ⟨P, S, I, O, t⟩ −→ ⟨P, start(S, p, t), I, O ∪ {ms (target(p))}, t⟩ RSTOP p ∈ canStop(S) ⟨P, S, I, O, t⟩ |= ce(target(p)) ⟨P, S, I, O, t⟩ −→ ⟨P, stop(S, p, t), I, O ∪ {me(target(p))}, t⟩ RTIME ⟨P, S, I, ∅, t⟩ −→∗ ⟨P, S′, I, O, t⟩ ̸ −→ ⟨P, S, t⟩ I,O =⇒ ⟨P, S′, t + 1⟩ Table: Rules for the internal reduction −→ and the observable reduction =⇒. Notation γ ̸ −→ means that there is no γ′ such that γ −→ γ′. Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 27/1 27/1 -
Domain-Specific Programming Language Summary • ReactiveIS is a programming language
for the specification and execution of interactive scores. • Tree structures are used to represent programs and execution states. • Trees provide a clear and intuitive representation of scores and the evolution of their execution. • Conditions defined as formulas of first-order logic allow to define more complex temporal relations in scores. • The operational semantics can be easily implemented in a programming language. Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 28/1 28/1 -
Outline Arias, Desainte-Catherine and Rueda Formal Models to Compose and
Execute Multimedia Interactive Scores 29/1 29/1 -
Concluding Remarks • Different formalisms can be used to model
interactive scores. Thanks to their mathematical basis, we can reason about the models and the composed scenarios. • Each model has advantages and disadvantages over the others. For example, real-time, automatic verification, expressiveness, …. • It is possible to develop a framework to compose, verify, simulate and interpret interactive scores. • The modelled scenarios can be translated into executable code (only on paper). Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 30/1 30/1 -
Concluding Remarks Future Work • Add conditionals and loops to
the models (timed automata). • Verify some properties of scenarios (e.g., playability). • Develop verification methods and tools for ReactiveIS. • Implement the synchronous approach in Lustre in order to handle streams in real-time. • Implement the compilers that are necessary for the proposed framework. • Improve the graphical interface in INScore. Arias, Desainte-Catherine and Rueda Formal Models to Compose and Execute Multimedia Interactive Scores 31/1 31/1 -
Formal Models to Compose and Execute Multimedia Interactive Scores Jaime
Arias, Myriam Desainte-Catherine and Camilo Rueda Laboratoire Bordelais de Recherche en Informatique (LaBRI) Université de Bordeaux Research Seminar - DECC - Javeriana, Cali October, 2014 1