Formal Semantcs and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice

Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with
Interactive Choice Jaime Arias Almeida Supervisors: Myriam Desainte-Catherine and Camilo Rueda Université de Bordeaux Laboratoire Bordelais de Recherche en Informatique (LaBRI) Doctoral Thesis Defense November 27, 2015 1

Motivation • Interactive multimedia = multimedia content + interaction with
the environment + spatial and temporal organization Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 1/44

Motivation • Interactive multimedia = multimedia content + interaction with
the environment + spatial and temporal organization (a) Live performances. (b) Museum installations. (c) Plastic art installations. Figure: Some examples of interactive multimedia systems. Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 1/44

Motivation • Interactive Scores1(IS) is a formalism for composing and
executing interactive multimedia scenarios. 1Antoine Allombert. “Aspects Temporels d’un Système de Partitions Musicales Interactives pour la Composition et l’Exécution”. PhD thesis. Université de Bordeaux, 2009 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 2/44

Motivation • Interactive Scores1(IS) is a formalism for composing and
executing interactive multimedia scenarios. • An interactive scenario is composed of: ◦ Textures → multimedia processes ◦ Structures → hierarchy ◦ Temporal relations ◦ Interaction points 1Antoine Allombert. “Aspects Temporels d’un Système de Partitions Musicales Interactives pour la Composition et l’Exécution”. PhD thesis. Université de Bordeaux, 2009 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 2/44

Motivation • i-score2 is a software for interactive scenarios based
on IS. ◦ Composition time → CSP problem ◦ Execution time → Timed Petri Net (TdPN) • IS mixes two temporal paradigms used in the current multimedia tools3: time-line and time-flow. • i-score has been used successfully during the last years4. 2Website: http://i-score.org/ 3Myriam Desainte-Catherine, Antoine Allombert, and Gérard Assayag. “Towards a Hybrid Temporal Paradigm for Musical Composition and Performance: The Case of Musical Interpretation”. Computer Music Journal 37.2 (2013), pp. 61–72. doi: 10.1162/COMJ_a_00179 4Jean-Michaël Celerier et al. “OSSIA: towards a unified interface for scoring time and interaction”. Proceedings of the First International Conference on Technologies for Music Notation and Representation - TENOR2015. Ed. by Marc Battier et al. Paris, France: Institut de Recherche en Musicologie, 2015, pp. 81–90. isbn: 978-2-9552905-0-7 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 3/44

Motivation • Nowadays, several applications (e.g., video games and interactive
museum installations) increasingly demand5: ◦ Flexible control structures: conditionals and loops. ◦ Mechanisms for the automatic verification of scenarios. 5Théo de la Hogue et al. “OSSIA: Open Scenario System for Interactive Applications”. Journées d’Informatique Musicale, JIM 2014, Bourges, France, May 21-23, 2014. 2014, pp. 78–84 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 4/44

museum installations) increasingly demand5: ◦ Flexible control structures: conditionals and loops. ◦ Mechanisms for the automatic verification of scenarios. • Several models have been proposed ◦ Olarte and Rueda. “A Declarative Language for Dynamic Multimedia Interaction Systems”. 2009 ◦ Toro, Desainte-Catherine, and Rueda. “Formal semantics for interactive music scores: a framework to design, specify properties and execute interactive scenarios”. 2014 5Théo de la Hogue et al. “OSSIA: Open Scenario System for Interactive Applications”. Journées d’Informatique Musicale, JIM 2014, Bourges, France, May 21-23, 2014. 2014, pp. 78–84 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 4/44

museum installations) increasingly demand5: ◦ Flexible control structures: conditionals and loops. ◦ Mechanisms for the automatic verification of scenarios. • Several models have been proposed ◦ Olarte and Rueda. “A Declarative Language for Dynamic Multimedia Interaction Systems”. 2009 ◦ Toro, Desainte-Catherine, and Rueda. “Formal semantics for interactive music scores: a framework to design, specify properties and execute interactive scenarios”. 2014 • Disadvantages of existing models: ◦ They do not satisfy the composers’ demands. ◦ Real-time execution is not provided. ◦ There is no practical solutions for automatic verification. 5Théo de la Hogue et al. “OSSIA: Open Scenario System for Interactive Applications”. Journées d’Informatique Musicale, JIM 2014, Bourges, France, May 21-23, 2014. 2014, pp. 78–84 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 4/44

Motivation The OSSIA6 project aimed to develop a formal model
and a set of tools for writing and verification of nonlinear and distributed scenarios. 6Open Scenario System for Interactive Application (OSSIA): http://www.agence-nationale-recherche.fr/?Projet=ANR-12-CORD-0024 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 5/44

Our Goal We aim at developing a formal model for
the specification and automatic verification of multimedia interactive scenarios with interactive choices. Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 6/44

Our Contributions ReactiveIS: • A domain-specific language for hierarchical interactive
scenarios. • Intuitive operational semantics based on tree-like structures. • Declarative interpretation of ReactiveIS programs as formulas in SELL7. • A new mechanism for reasoning about scenarios. 7Vivek Nigam, Carlos Olarte, and Elaine Pimentel. “A General Proof System for Modalities in Concurrent Constraint Programming”. CONCUR 2013 - Concurrency Theory - 24th International Conference, CONCUR 2013, Buenos Aires, Argentina, August 27-30, 2013. Proceedings. Vol. 8052. Lecture Notes in Computer Science. Springer, 2013, pp. 410–424. doi: 10.1007/978-3-642-40184-8_29. Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 7/44

Our Contributions Framework: • Specification of hierarchical interactive scenarios as
a network of Timed Automata8. • Extension of IS with interaction points guarded by conditions. • A mechanism for the automatic verification of scenarios. • A physical parallel implementation of interactive scenarios. 8Rajeev Alur and David L. Dill. “A Theory of Timed Automata”. Theor. Comput. Sci. 126.2 (1994), pp. 183–235. doi: 10.1016/0304-3975(94)90010-8. Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 8/44

Our Contributions Dynamic Scenarios: • Reactive interpreter of interactive scenarios.
• An environment for building prototypes easily and living code. • A novel graphical interface reflecting the dynamic behavior of scenarios using synchronous observers9. 9Nicolas Halbwachs, Fabienne Lagnier, and Pascal Raymond. “Synchronous Observers and the Verification of Reactive Systems”. Algebraic Methodology and Software Technology (AMAST ’93), Proceedings of the Third International Conference on Methodology and Software Technology, University of Twente, Enschede, The Netherlands, 21-25 June, 1993. Workshops in Computing. Springer, 1993, pp. 83–96. doi: 10.1007/978-1-4471-3227-1_8 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 9/44

Our Contributions Dynamic Scenarios: • Reactive interpreter of interactive scenarios.
• An environment for building prototypes easily and living code. • A novel graphical interface reflecting the dynamic behavior of scenarios using synchronous observers9. Streams: • Extension of IS with data streams using Colored Petri Nets10. • We provide the notion of asynchronous functional composition for IS. • We open the possibility of verifying properties at a low-level. 10Kurt Jensen and Lars Michael Kristensen. “Colored Petri nets: a graphical language for formal modeling and validation of concurrent systems”. Commun. ACM 58.6 (2015), pp. 61–70. doi: 10.1145/2663340 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 9/44

Outline 1. ReactiveIS 2. Timed Automata Framework 3. Reactive Synchronous
Interpreter 4. Handling Data Streams 5. Concluding Remarks Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 10/44

Outline 1. ReactiveIS 1.1. Program Representation 1.2. State Representation 1.3.
Operational Semantics 1.4. Logical Characterization 2. Timed Automata Framework 3. Reactive Synchronous Interpreter 4. Handling Data Streams 5. Concluding Remarks Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 11/44

ReactiveIS Overview11 • Domain-Specific Language (DSL) for specification, execution and
verification of interactive scenarios. ◦ Tree representation of programs. ◦ Tree-based operational semantics. ◦ Logical semantics based on SELL. • Trees are simple, intuitive, and formal structures. • This is joint work with Carlos Olarte (MUSICAL) and Sylvain Salvati (PoSET). 11Jaime Arias, Myriam Desainte-Catherine, Carlos Olarte, and Camilo Rueda. “Foundations for Reliable and Flexible Interactive Multimedia Scores”. 5th International Conference on Mathematics and Computation in Music, MCM 2015, London, UK, June 22-25, 2015. Ed. by Tom Collins, David Meredith, and Anja Volk. Vol. 9110. Lecture Notes in Computer Science. Springer, 2015, pp. 29–41. doi: 10.1007/978-3-319-20603-5_3. Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 12/44

ReactiveIS Program Representation F WS WH LB G Jaime Arias
Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 13/44

ReactiveIS Program Representation F WS WH LB G cs (ε)
= true ce (ε) = EndScenario Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 13/44

ReactiveIS Program Representation F WS WH LB G cs (G.WH)
= (WaitFromStart(G, 2, 5)∧ WaitEvent(mouse1)) ∨ WaitFromStart(G, 5, 5) ce (G.WH) = WaitFromStart(G.WH, 1, 1) ms (G.WH) = sound1_on me (G.WH) = sound1_oﬀ Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 13/44

ReactiveIS State Representation ts (ε) = 0 te (ε) =
⊥ Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 14/44

ReactiveIS State Representation F ts (ε) = 0 te (ε)
= ⊥ ts (F) = 1 te (F) = ⊥ Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 14/44

ReactiveIS State Representation F WS ts (WS) = 3 te
(WS) = ⊥ ts (F) = 1 te (F) = 3 ts (ε) = 0 te (ε) = ⊥ Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 14/44

ReactiveIS Operational Semantics – Rule RStart RSTART p ∈ canStart(S,
P) ⟨P, S, I, t⟩ ⊢ cs (n) ⟨St, O⟩I,t S −→P ⟨start(St, p, t), O ∪ {ms (n)}⟩I,t S where n = · P (p) WH LB G F WS 0 ⊥ 8 ⊥ 3 5 1 3 G F WS t = 9 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 15/44

P) ⟨P, S, I, t⟩ ⊢ cs (n) ⟨St, O⟩I,t S −→P ⟨start(St, p, t), O ∪ {ms (n)}⟩I,t S where n = · P (p) WH LB G F WS canStart(P, S) 0 ⊥ 8 ⊥ 3 5 1 3 G F WS t = 9 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 15/44

P) ⟨P, S, I, t⟩ ⊢ cs (n) ⟨St, O⟩I,t S −→P ⟨start(St, p, t), O ∪ {ms (n)}⟩I,t S where n = · P (p) WH LB G F WS 0 ⊥ 8 ⊥ 3 5 1 3 G F WS t = 9 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 15/44

P) ⟨P, S, I, t⟩ ⊢ cs (n) ⟨St, O⟩I,t S −→P ⟨start(St, p, t), O ∪ {ms (n)}⟩I,t S where n = · P (p) 0 ⊥ 8 ⊥ 3 5 1 3 G F WS t = 9 cs ( · P (G.LB)) = WaitFromStart(G, 1, 1) Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 15/44

P) ⟨P, S, I, t⟩ ⊢ cs (n) ⟨St, O⟩I,t S −→P ⟨ start(St, p, t) , O ∪ {ms (n)}⟩I,t S where n = · P (p) 0 ⊥ 8 ⊥ 3 5 1 3 G F WS t = 9 cs ( · P (G.LB)) = WaitFromStart(G, 1, 1) Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 15/44

P) ⟨P, S, I, t⟩ ⊢ cs (n) ⟨St, O⟩I,t S −→P ⟨ start(St, p, t) , O ∪ {ms (n)}⟩I,t S where n = · P (p) 0 ⊥ 8 ⊥ 9 ⊥ LB 3 5 1 3 G F WS t = 9 cs ( · P (G.LB)) = WaitFromStart(G, 1, 1) Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 15/44

ReactiveIS Operational Semantics – Rule RStop RSTOP p ∈ canStop(S)
⟨P, S, I, t⟩ ⊢ ce (n) ⟨St, O⟩I,t S −→P ⟨stop(St, p, t), O ∪ {me (n)}⟩I,t S where n = · P (p) 0 ⊥ 8 ⊥ 9 ⊥ LB 3 5 1 3 G F WS t = 12 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 16/44

⟨P, S, I, t⟩ ⊢ ce (n) ⟨St, O⟩I,t S −→P ⟨ stop(St, p, t) , O ∪ {me (n)}⟩I,t S where n = · P (p) 0 ⊥ 8 ⊥ 9 ⊥ LB 3 5 1 3 G F WS t = 12 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 16/44

⟨P, S, I, t⟩ ⊢ ce (n) ⟨St, O⟩I,t S −→P ⟨ stop(St, p, t) , O ∪ {me (n)}⟩I,t S where n = · P (p) 0 ⊥ 8 ⊥ 9 12 LB 3 5 1 3 G F WS t = 12 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 16/44

ReactiveIS Operational Semantics – Rule ROBS ROBS ⟨S, ∅⟩I,t S
−→∗ P ⟨S′, O⟩I,t S ̸ −→P ⟨S, t⟩ I,O =⇒P ⟨S′, t + 1⟩ Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 17/44

ReactiveIS Operational Semantics – Rule ROBS ROBS ⟨S, ∅⟩I,t S
−→∗ P ⟨S′, O⟩I,t S ̸ −→P ⟨S, t⟩ I,O =⇒P ⟨S′, t + 1⟩ Corollary (Determinism) For all state S and input I, if ⟨S, t⟩ I,O1 =⇒P ⟨S′ 1 , t′⟩ and ⟨S, t⟩ I,O2 =⇒P ⟨S′ 2 , t′⟩ then O1 = O2 and S′ 1 = S′ 2 . Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 17/44

ReactiveIS Subexponentials in Linear Logic (SELL) Girard’s Linear Logic12: Formulas
are seen as resources, e.g, c ⊗ c ̸−→ c. Subexponentials: meanings for !aF • “F holds in location a” (e.g., in a given structure) • “F holds with certain modality a” (e.g., in a time-unit t). 12Jean-Yves Girard. “Light Linear Logic”. Inf. Comput. 143.2 (1998), pp. 175–204. doi: 10.1006/inco.1998.2700. Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 18/44

ReactiveIS Declarative Meaning of ReactiveIS programs We defined an encoding
from ReactiveIS programs into SELL formulas. Adequacy P reaches a state s iff the sequent [[P]] −→ [[s]] is provable in SELL. This opens the possibility of reasoning about IS by using well established techniques in proof theory! We can verify properties such as: • Is it possible for the structure A and B to be played concurrently ? • Is the execution of A always preceded by an execution of B ? • Will all the structures be eventually played ? Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 19/44

Timed Automata Framework Overview i-score scenario IS2UPPAAL UPPAAL model Property
Checking UPP2C UPP2HDL … C/C++ Code VHDL/System Verilog Code … Composition Verification Interpretation i-score scenario Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 21/44

Checking UPP2C UPP2HDL … C/C++ Code VHDL/System Verilog Code … Composition Verification Interpretation Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 21/44

Checking UPP2C UPP2HDL … C/C++ Code VHDL/System Verilog Code … Composition Verification Interpretation UPPAAL model Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 21/44

Timed Automata Framework UPPAAL Model13 • A timed automaton is
a finite-state machine equipped with clocks (i.e., real-valued variables). • Timed Automata has been used successfully to model and verify time-critical systems (e.g., multimedia applications): ◦ Echeveste, Cont, Giavitto, and Jacquemard. “Operational semantics of a domain specific language for real time musician-computer interaction”. 2013 ◦ Poncelet and Jacquemard. “Model based testing of an interactive music system”. 2015 13Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda. “A Framework for Composition, Verification and Real-Time Performance of Multimedia Interactive Scenarios”. 15th International Conference on Application of Concurrency to System Design, ACSD 2015, Brussels, Belgium, June 21-26, 2015. IEEE, 2015, pp. 140–151. Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 22/44

Timed Automata Framework UPPAAL Model – Handling Intervals idle error
finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = true counter = 0 en = false urg = true Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 23/44

finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = false counter = 1 en = false urg = true Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 23/44

finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = false counter = 2 en = false urg = true Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 23/44

finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = false counter = 2 en = true urg = true Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 23/44

Timed Automata Framework UPPAAL Model – Multimedia Processes Brightness 100%
50% 30% 0% p0 p1 p2 p3 p4 p5 p6 ∆1 ∆2 ∆3 ∆4 ∆5 ∆6 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 24/44

50% 30% 0% p0 p1 p2 p3 p4 p5 p6 ∆1 ∆2 ∆3 ∆4 ∆5 ∆6 typedef struct { int value; int offset; } parameter_t; parameter_t process_brightness[7] = { {0,0}, {30,5}, {50,2}, ... }; Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 24/44

50% 30% 0% p0 p1 p2 p3 p4 p5 p6 ∆1 ∆2 ∆3 ∆4 ∆5 ∆6 typedef struct { int value; int offset; } parameter_t; parameter_t process_brightness[7] = { {0,0}, {30,5}, {50,2}, ... }; idle finished wait t ≤ mp[i].offset t ≥ 0, start?, {t = 0, i = 0} true, kill_p?, {t = 0} true, skip_p?, {t = 0} true, kill_p?, {t = 0} true, stop?, {t = 0} t = m p[i].offset ∧ i = lim it, send!, {data = m p[i].value,t = 0} t = mp[i].offset ∧ i < limit, send!, {t = 0, data = mp[i].value, i + +} Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 24/44

Timed Automata Framework Automatic Verification Some important properties are: •
Will the scenario finish ? ◦ A <> Scenario.finished Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 25/44

Will the scenario finish ? ◦ A <> Scenario .finished Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 25/44

Will the scenario finish ? ◦ A <> Scenario. finished Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 25/44

Will the scenario finish ? ◦ A <> Scenario.finished • Are all temporal relations always satisfied ? ◦ A[] !Control_Start_Group.error Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 25/44

Will the scenario finish ? ◦ A <> Scenario.finished • Are all temporal relations always satisfied ? ◦ A[] !Control_Start_Group.error • Is a specific temporal constraint always satisfied ? ◦ A[] (Structure_Group.wait_min imply clk >= 4468) Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 25/44

Will the scenario finish ? ◦ A <> Scenario.finished • Are all temporal relations always satisfied ? ◦ A[] !Control_Start_Group.error • Is a specific temporal constraint always satisfied ? ◦ A[] (Structure_Group.wait_min imply clk >= 4468) • Is a multimedia ressource used by two or more process at the same time ? ◦ A[] (!Texture_WH.wait || !Texture_LB.wait) Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 25/44

Checking UPP2C UPP2HDL … C/C++ Code VHDL/System Verilog Code … Composition Verification Interpretation VHDL/System Verilog Code Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 26/44

Timed Automata Framework True Parallel Implementation • Multimedia applications perform
real-time, compute-intensive, and data-intensive tasks that sometimes the performance of standard computers is not sufficient. • Most of the multimedia applications are executed on architectures and operating systems that do not provide low-latency and real-time performance (e.g., i-score). Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 27/44

Timed Automata Framework True Parallel Implementation • Multimedia applications perform
real-time, compute-intensive, and data-intensive tasks that sometimes the performance of standard computers is not sufficient. • Most of the multimedia applications are executed on architectures and operating systems that do not provide low-latency and real-time performance (e.g., i-score). • FPGAs offer reconfigurability, high-level design, physical parallelism, high-speed, reliability, and re-use14. • FPGAs have already been used with success in many different industrial applications: aerospace, automotive, medical, video and audio processing. 14Stephen Trimberger. “Three Ages of FPGAs: A Retrospective on the First Thirty Years of FPGA Technology”. Proceedings of the IEEE 103.3 (2015), pp. 318–331. doi: 10.1109/JPROC.2015.2392104 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 27/44

Timed Automata Framework True Parallel Implementation (a) Performance comparison. (b)
Cost/performance comparison. Figure: A comparison between FPGA and DSP platforms for real-time video applications15. 15Jeff Bier and Jennifer Eyre. “BDTI Study Certifies High-Level Synthesis Flows for DSP-Centric FPGA Design”. Xcell Journal 71 (2010), pp. 12–17. Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 28/44

Timed Automata Framework True Parallel Implementation16 inputs clock FPGA Next
State State Register Output Logic outputs Local Variables Clocks Variables Mealy FSM Timed Automaton Figure: Block diagram of the proposed hardware implementation of a timed automaton. 16Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda. “Exploiting Parallelism in FPGAs for the Real-Time Interpretation of Interactive Multimedia Scores”. Journées d’Informatique Musicale, JIM 2015, Montréal, Canada, May 7-9, 2015. 2015. Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 29/44

Timed Automata Framework True Parallel Implementation clock scenario current time
20 21 22 23 24 25 26 27 28 29 30 31 32 33 IP start start Group start Wolf Howl start Light Beam stop IP stop Group killed Wolf Howl killed Light Beam Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 30/44

Checking UPP2C UPP2HDL … C/C++ Code VHDL/System Verilog Code … Composition Verification Interpretation C/C++ Code17 … 17Tobias Amnell et al. “TIMES: A Tool for Schedulability Analysis and Code Generation of Real-Time Systems”. Formal Modeling and Analysis of Timed Systems: First International Workshop, FORMATS 2003, Marseille, France, September 6-7, 2003. Revised Papers. Ed. by Kim Guldstrand Larsen and Peter Niebert. Vol. 2791. Lecture Notes in Computer Science. Springer, 2003, pp. 60–72. doi: 10.1007/978-3-540-40903-8_6 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 31/44

Reactive Synchronous Interpreter • ReactiveML18 is a programming language for
implementing interactive systems. synchronous language + functional programming • ReactiveML has been used successfully in multimedia applications: ◦ Baudart, Mandel, and Pouzet. “Programming Mixed Music in ReactiveML”. 2013 ◦ Baudart, Jacquemard, Mandel, and Pouzet. “A synchronous embedding of Antescofo, a domain-specific language for interactive mixed music”. 2013 18Louis Mandel, Cédric Pasteur, and Marc Pouzet. “ReactiveML, ten years later”. Proceedings of the 17th International Symposium on Principles and Practice of Declarative Programming, Siena, Italy, July 14-16, 2015. Ed. by Moreno Falaschi and Elvira Albert. ACM, 2015, pp. 6–17. doi: 10.1145/2790449.2790509. Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 33/44

Reactive Synchronous Interpreter19 Overview Interpreter IS OSC Client Synchronous Observer
OSC Server ReactiveML Pure Data/ Max/MSP INScore Figure: Architecture of RItMos. 19Jaime Arias, Myriam Desainte-Catherine, Sylvain Salvati, and Camilo Rueda. “Executing Hierarchical Interactive Scores in ReactiveML”. . Journées d’Informatique Musicale, JIM 2014, Bourges, France, May 21-23, 2014. 2014, pp. 25–34. Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 34/44

Reactive Synchronous Interpreter Dynamic Graphical Interface (a) Colored boxes represent
the TO that are currently executing. (b) Dashed borders mean that the IP is enabled to be triggered. (c) Triggering IPs causes that a TO moves on the time-line or resizes its width (duration). Figure: Example of an interactive scenario visualized in INScore. Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 35/44

Handling Data Streams Colored Petri Net Model20 time value 3.7
v1 4.3 v2 4.3 v3 3.7 v4 2.5 v5 2.5 v6 1.8 v7 1.2 v8 1.2 v9 1.8 v10 2.5 v11 2.5 v12 stream 20Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda. “Modelling Data Processing for Interactive Scores Using Coloured Petri Nets”. 14th International Conference on Application of Concurrency to System Design, ACSD 2014, Tunis La Marsa, Tunisia, June 23-27, 2014. IEEE, 2014, pp. 186–195. doi: 10.1109/ACSD.2014.23 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 37/44

v1 4.3 v2 4.3 v3 3.7 v4 2.5 v5 2.5 v6 1.8 v7 1.2 v8 1.2 v9 1.8 v10 2.5 v11 2.5 v12 stream n_max f_dur f_dur n_max f_dur (n,f) n_max n f_dur@+f_dur n_max n+1 (n,f) number frames period start get duration int time file file int frame file next int [n <= n_max] t Declarations colset TIME = time; colset UNIT = unit; colset INT = int; colset BOOL = bool; colset DATA = INT; colset DURATION = TIME timed; colset FILE = product INT * DATA; var f_dur : TIME; var e : BOOL; var f : DATA; var n,n_max : INT; 12 2 1 (1,3.7) 20Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda. “Modelling Data Processing for Interactive Scores Using Coloured Petri Nets”. 14th International Conference on Application of Concurrency to System Design, ACSD 2014, Tunis La Marsa, Tunisia, June 23-27, 2014. IEEE, 2014, pp. 186–195. doi: 10.1109/ACSD.2014.23 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 37/44

v1 4.3 v2 4.3 v3 3.7 v4 2.5 v5 2.5 v6 1.8 v7 1.2 v8 1.2 v9 1.8 v10 2.5 v11 2.5 v12 stream n_max f_dur f_dur n_max f_dur (n,f) n_max n f_dur@+f_dur n_max n+1 (n,f) number frames period start get duration int time file file int frame file next int [n <= n_max] t Declarations colset TIME = time; colset UNIT = unit; colset INT = int; colset BOOL = bool; colset DATA = INT; colset DURATION = TIME timed; colset FILE = product INT * DATA; var f_dur : TIME; var e : BOOL; var f : DATA; var n,n_max : INT; 12 2@0 1 (1,3.7) 20Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda. “Modelling Data Processing for Interactive Scores Using Coloured Petri Nets”. 14th International Conference on Application of Concurrency to System Design, ACSD 2014, Tunis La Marsa, Tunisia, June 23-27, 2014. IEEE, 2014, pp. 186–195. doi: 10.1109/ACSD.2014.23 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 37/44

v1 4.3 v2 4.3 v3 3.7 v4 2.5 v5 2.5 v6 1.8 v7 1.2 v8 1.2 v9 1.8 v10 2.5 v11 2.5 v12 stream n_max f_dur f_dur n_max f_dur (n,f) n_max n f_dur@+f_dur n_max n+1 (n,f) number frames period start get duration int time file file int frame file next int [n <= n_max] t Declarations colset TIME = time; colset UNIT = unit; colset INT = int; colset BOOL = bool; colset DATA = INT; colset DURATION = TIME timed; colset FILE = product INT * DATA; var f_dur : TIME; var e : BOOL; var f : DATA; var n,n_max : INT; 2 12 2@2 (2,4.3) (1,3.7) 20Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda. “Modelling Data Processing for Interactive Scores Using Coloured Petri Nets”. 14th International Conference on Application of Concurrency to System Design, ACSD 2014, Tunis La Marsa, Tunisia, June 23-27, 2014. IEEE, 2014, pp. 186–195. doi: 10.1109/ACSD.2014.23 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 37/44

Interpreter 4. Handling Data Streams 5. Concluding Remarks 5.1. Overview 5.2. Future Work Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 38/44

Concluding Remarks Overview • We proposed a domain-specific language for
the specification and verification of interactive scenarios. ◦ Extensible. ◦ Specification not limited to the graphical environment. ◦ Intuitive operational semantics based on tree-like structures. ◦ Declarative interpretation of programs, allowing to reason about scenarios. Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 39/44

Concluding Remarks Overview • We proposed a domain-specific language for
the specification and verification of interactive scenarios. ◦ Extensible. ◦ Specification not limited to the graphical environment. ◦ Intuitive operational semantics based on tree-like structures. ◦ Declarative interpretation of programs, allowing to reason about scenarios. • We proposed a framework for the specification, verification and execution of interactive scenarios. ◦ Extension of interactive scenarios with guarded interaction points. ◦ Automatic verification of scenarios using mature and robust tools (i.e., UPPAAL). ◦ We showed the verification of some properties. ◦ Synthesizing of scenarios into reconfigurable hardware, allowing their low-latency and real-time execution. Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 39/44

Concluding Remarks Overview • We proposed a reactive interpreter of
interactive scenarios using ReactiveML. ◦ Dynamic creation of processes. ◦ Live coding. ◦ New features can be easily prototyped. ◦ Dynamic graphical interface using INScore and a synchronous observer. Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 40/44

Concluding Remarks Overview • We proposed a reactive interpreter of
interactive scenarios using ReactiveML. ◦ Dynamic creation of processes. ◦ Live coding. ◦ New features can be easily prototyped. ◦ Dynamic graphical interface using INScore and a synchronous observer. • We proposed a formalization of streams using Colored Petri Nets. ◦ Notion of asynchronous function composition. ◦ Modules for basic processing of audio files (e.g., reading, appending files). Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 40/44

Concluding Remarks Publications from this dissertation • Jaime Arias, Michell
Gúzman, and Carlos Olarte. “A Symbolic Model for Timed Concurrent Constraint Programming”. Electronic Notes in Theoretical Computer Science 312 (2015), pp. 161–177. doi: 10.1016/j.entcs.2015.04.010 • Jaime Arias, Myriam Desainte-Catherine, Carlos Olarte, and Camilo Rueda. “Foundations for Reliable and Flexible Interactive Multimedia Scores”. 5th International Conference on Mathematics and Computation in Music, MCM 2015, London, UK, June 22-25, 2015. Ed. by Tom Collins, David Meredith, and Anja Volk. Vol. 9110. Lecture Notes in Computer Science. Springer, 2015, pp. 29–41. doi: 10.1007/978-3-319-20603-5_3 • Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda. “A Framework for Composition, Verification and Real-Time Performance of Multimedia Interactive Scenarios”. 15th International Conference on Application of Concurrency to System Design, ACSD 2015, Brussels, Belgium, June 21-26, 2015. IEEE, 2015, pp. 140–151 • Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda. “Modelling Data Processing for Interactive Scores Using Coloured Petri Nets”. 14th International Conference on Application of Concurrency to System Design, ACSD 2014, Tunis La Marsa, Tunisia, June 23-27, 2014. IEEE, 2014, pp. 186–195. doi: 10.1109/ACSD.2014.23 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 41/44

Concluding Remarks Publications from this dissertation • Jaime Arias and
Jean-Michaël Celerier. “Le Séquenceur Interactif Multimédia i-score”. Journées Développement Logiciel de l’Enseignement Supérieur et de la Recherche, JDEV 2015, Bordeaux, France, June 30 - July 3, 2015. Poster. 2015 • Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda. “Exploiting Parallelism in FPGAs for the Real-Time Interpretation of Interactive Multimedia Scores”. Journées d’Informatique Musicale, JIM 2015, Montréal, Canada, May 7-9, 2015. 2015 • Jaime Arias, Myriam Desainte-Catherine, Sylvain Salvati, and Camilo Rueda. “Executing Hierarchical Interactive Scores in ReactiveML”. . Journées d’Informatique Musicale, JIM 2014, Bourges, France, May 21-23, 2014. 2014, pp. 25–34 Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 42/44

Concluding Remarks Future Work • Formalization in Coq of the
ReactiveIS semantics. • Extension of ReactiveIS with loops and conditionals. • Integration of IS2UPP to i-score. • Extension of the Timed Automata model with : ◦ A coherent model for loops21. ◦ Notions for improvisation22. • Colored Petri Net model to Timed Automata model. 21Abdeldjalil Boudjadar, Frits W. Vaandrager, Jean-Paul Bodeveix, and Mamoun Filali. “Extending UPPAAL for the Modeling and Verification of Dynamic Real-Time Systems”. Fundamentals of Software Engineering - 5th International Conference, FSEN 2013, Tehran, Iran, April 24-26, 2013, Revised Selected Papers. Vol. 8161. Lecture Notes in Computer Science. Springer, 2013, pp. 111–132. doi: 10.1007/978-3-642-40213-5_8. 22Cheng-i Wang and Shlomo Dubnov. “The Variable Markov Oracle: Algorithms for Human Gesture Applications”. IEEE MultiMedia 22.4 (2015), pp. 52–67. doi: 10.1109/MMUL.2015.76. Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 43/44

Concluding Remarks Future Work • Encoding of the Timed Automata
model into the tiled programming23. • Validation of the hardware implementation using SystemVerilog Assertions24 • Study of other verification and validation techniques, such as Model-Based Testing and game semantics. 23David Janin et al. “The T-calculus: Towards a Structured Programing of (Musical) Time and Space”. Proceedings of the First ACM SIGPLAN Workshop on Functional Art, Music, Modeling and Design. FARM ’13. Boston, Massachusetts, USA: ACM, 2013, pp. 23–34. isbn: 978-1-4503-2386-4. doi: 10.1145/2505341.2505347. 24Eduard Cerny, Surrendra Dudani, John Havlicek, and Dmitry Korchemny. SVA: The Power of Assertions in SystemVerilog. 2nd. Springer, 2014. isbn: 9783319071381. Jaime Arias Almeida (2015) Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice 44/44

Thank you for your attention!

Formal Semantics and Automatic Verification of Hierarchical Multimedia Scenarios with
Interactive Choice Jaime Arias Almeida Supervisors: Myriam Desainte-Catherine and Camilo Rueda Université de Bordeaux Laboratoire Bordelais de Recherche en Informatique (LaBRI) Doctoral Thesis Defense November 27, 2015 1

References Allombert, Antoine. “Aspects Temporels d’un Système de Partitions Musicales
Interactives pour la Composition et l’Exécution”. PhD thesis. Université de Bordeaux, 2009 (cited on slides 4, 5). Alur, Rajeev and David L. Dill. “A Theory of Timed Automata”. Theor. Comput. Sci. 126.2 (1994), pp. 183–235. doi: 10.1016/0304-3975(94)90010-8 (cited on slide 13). Amnell, Tobias et al. “TIMES: A Tool for Schedulability Analysis and Code Generation of Real-Time Systems”. Formal Modeling and Analysis of Timed Systems: First International Workshop, FORMATS 2003, Marseille, France, September 6-7, 2003. Revised Papers. Ed. by Kim Guldstrand Larsen and Peter Niebert. Vol. 2791. Lecture Notes in Computer Science. Springer, 2003, pp. 60–72. doi: 10.1007/978-3-540-40903-8_6 (cited on slide 86). Arias, Jaime and Jean-Michaël Celerier. “Le Séquenceur Interactif Multimédia i-score”. Journées Développement Logiciel de l’Enseignement Supérieur et de la Recherche, JDEV 2015, Bordeaux, France, June 30 - July 3, 2015. Poster. 2015 (cited on slide 102).

References Arias, Jaime, Myriam Desainte-Catherine, and Camilo Rueda. “A Framework
for Composition, Verification and Real-Time Performance of Multimedia Interactive Scenarios”. 15th International Conference on Application of Concurrency to System Design, ACSD 2015, Brussels, Belgium, June 21-26, 2015. IEEE, 2015, pp. 140–151 (cited on slides 48, 101). – .“Exploiting Parallelism in FPGAs for the Real-Time Interpretation of Interactive Multimedia Scores”. Journées d’Informatique Musicale, JIM 2015, Montréal, Canada, May 7-9, 2015. 2015 (cited on slides 84, 102). – .“Modelling Data Processing for Interactive Scores Using Coloured Petri Nets”. 14th International Conference on Application of Concurrency to System Design, ACSD 2014, Tunis La Marsa, Tunisia, June 23-27, 2014. IEEE, 2014, pp. 186–195. doi: 10.1109/ACSD.2014.23 (cited on slides 92–95, 101). Arias, Jaime, Michell Gúzman, and Carlos Olarte. “A Symbolic Model for Timed Concurrent Constraint Programming”. Electronic Notes in Theoretical Computer Science 312 (2015), pp. 161–177. doi: 10.1016/j.entcs.2015.04.010 (cited on slide 101). Arias, Jaime et al. “Executing Hierarchical Interactive Scores in ReactiveML”. Journées d’Informatique Musicale, JIM 2014, Bourges, France, May 21-23, 2014. 2014, pp. 25–34 (cited on slides 89, 102).

References Arias, Jaime et al. “Foundations for Reliable and Flexible
Interactive Multimedia Scores”. 5th International Conference on Mathematics and Computation in Music, MCM 2015, London, UK, June 22-25, 2015. Ed. by Tom Collins, David Meredith, and Anja Volk. Vol. 9110. Lecture Notes in Computer Science. Springer, 2015, pp. 29–41. doi: 10.1007/978-3-319-20603-5_3 (cited on slides 18, 101). Baudart, Guillaume, Louis Mandel, and Marc Pouzet. “Programming Mixed Music in ReactiveML”. ACM SIGPLAN Workshop on Functional Art, Music, Modeling and Design (FARM’13), Boston, Massachusetts, USA, September 28, 2013. Boston, USA: ACM, 2013, pp. 11–22. doi: 10.1145/2505341.2505344 (cited on slide 88). Baudart, Guillaume et al. “A synchronous embedding of Antescofo, a domain-specific language for interactive mixed music”. Proceedings of the International Conference on Embedded Software, EMSOFT 2013, Montreal, QC, Canada, September 29 - Oct. 4, 2013. IEEE, 2013, 1:1–1:12. doi: 10.1109/EMSOFT.2013.6658579 (cited on slide 88). Bier, Jeff and Jennifer Eyre. “BDTI Study Certifies High-Level Synthesis Flows for DSP-Centric FPGA Design”. Xcell Journal 71 (2010), pp. 12–17 (cited on slide 83).

References Boudjadar, Abdeldjalil et al. “Extending UPPAAL for the Modeling
and Verification of Dynamic Real-Time Systems”. Fundamentals of Software Engineering - 5th International Conference, FSEN 2013, Tehran, Iran, April 24-26, 2013, Revised Selected Papers. Vol. 8161. Lecture Notes in Computer Science. Springer, 2013, pp. 111–132. doi: 10.1007/978-3-642-40213-5_8 (cited on slide 103). Celerier, Jean-Michaël et al. “OSSIA: towards a unified interface for scoring time and interaction”. Proceedings of the First International Conference on Technologies for Music Notation and Representation - TENOR2015. Ed. by Marc Battier et al. Paris, France: Institut de Recherche en Musicologie, 2015, pp. 81–90. isbn: 978-2-9552905-0-7 (cited on slide 6). Cerny, Eduard et al. SVA: The Power of Assertions in SystemVerilog. 2nd. Springer, 2014. isbn: 9783319071381 (cited on slide 104). de la Hogue, Théo et al. “OSSIA: Open Scenario System for Interactive Applications”. Journées d’Informatique Musicale, JIM 2014, Bourges, France, May 21-23, 2014. 2014, pp. 78–84 (cited on slides 7–9). Desainte-Catherine, Myriam, Antoine Allombert, and Gérard Assayag. “Towards a Hybrid Temporal Paradigm for Musical Composition and Performance: The Case of Musical Interpretation”. Computer Music Journal 37.2 (2013), pp. 61–72. doi: 10.1162/COMJ_a_00179 (cited on slide 6).

References Echeveste, José et al. “Operational semantics of a domain
specific language for real time musician-computer interaction”. Discrete Event Dynamic Systems 23.4 (2013), pp. 343–383. doi: 10.1007/s10626-013-0166-2 (cited on slide 48). Girard, Jean-Yves. “Light Linear Logic”. Inf. Comput. 143.2 (1998), pp. 175–204. doi: 10.1006/inco.1998.2700 (cited on slide 42). Halbwachs, Nicolas, Fabienne Lagnier, and Pascal Raymond. “Synchronous Observers and the Verification of Reactive Systems”. Algebraic Methodology and Software Technology (AMAST ’93), Proceedings of the Third International Conference on Methodology and Software Technology, University of Twente, Enschede, The Netherlands, 21-25 June, 1993. Workshops in Computing. Springer, 1993, pp. 83–96. doi: 10.1007/978-1-4471-3227-1_8 (cited on slide 14). Janin, David et al. “The T-calculus: Towards a Structured Programing of (Musical) Time and Space”. Proceedings of the First ACM SIGPLAN Workshop on Functional Art, Music, Modeling and Design. FARM ’13. Boston, Massachusetts, USA: ACM, 2013, pp. 23–34. isbn: 978-1-4503-2386-4. doi: 10.1145/2505341.2505347 (cited on slide 104). Jensen, Kurt and Lars Michael Kristensen. “Colored Petri nets: a graphical language for formal modeling and validation of concurrent systems”. Commun. ACM 58.6 (2015), pp. 61–70. doi: 10.1145/2663340 (cited on slide 15).

References Mandel, Louis, Cédric Pasteur, and Marc Pouzet. “ReactiveML, ten
years later”. Proceedings of the 17th International Symposium on Principles and Practice of Declarative Programming, Siena, Italy, July 14-16, 2015. Ed. by Moreno Falaschi and Elvira Albert. ACM, 2015, pp. 6–17. doi: 10.1145/2790449.2790509 (cited on slide 88). Nigam, Vivek, Carlos Olarte, and Elaine Pimentel. “A General Proof System for Modalities in Concurrent Constraint Programming”. CONCUR 2013 - Concurrency Theory - 24th International Conference, CONCUR 2013, Buenos Aires, Argentina, August 27-30, 2013. Proceedings. Vol. 8052. Lecture Notes in Computer Science. Springer, 2013, pp. 410–424. doi: 10.1007/978-3-642-40184-8_29 (cited on slide 12). Olarte, Carlos and Camilo Rueda. “A Declarative Language for Dynamic Multimedia Interaction Systems”. 2nd International Conference on Mathematics and Computation in Music, MCM 2009, New Haven, CT, USA, June 19-22, 2009. Ed. by Elaine Chew, Adrian Childs, and Ching-Hua Chuan. Vol. 38. Communications in Computer and Information Science. Springer Berlin Heidelberg, 2009, pp. 218–227. isbn: 978-3-642-02393-4. doi: 10.1007/978-3-642-02394-1_20 (cited on slides 7–9).

References Poncelet, Clément and Florent Jacquemard. “Model based testing of
an interactive music system”. Proceedings of the 30th Annual ACM Symposium on Applied Computing, Salamanca, Spain, April 13-17, 2015. Ed. by Roger L. Wainwright et al. ACM, 2015, pp. 1759–1764. doi: 10.1145/2695664.2695804 (cited on slide 48). Toro, Mauricio, Myriam Desainte-Catherine, and Camilo Rueda. “Formal semantics for interactive music scores: a framework to design, specify properties and execute interactive scenarios”. Journal of Mathematics and Music 8.1 (2014), pp. 93–112. doi: 10.1080/17459737.2013.870610 (cited on slides 7–9). Trimberger, Stephen. “Three Ages of FPGAs: A Retrospective on the First Thirty Years of FPGA Technology”. Proceedings of the IEEE 103.3 (2015), pp. 318–331. doi: 10.1109/JPROC.2015.2392104 (cited on slide 82). Wang, Cheng-i and Shlomo Dubnov. “The Variable Markov Oracle: Algorithms for Human Gesture Applications”. IEEE MultiMedia 22.4 (2015), pp. 52–67. doi: 10.1109/MMUL.2015.76 (cited on slide 103).

Formal Semantcs and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice

Formal Semantcs and Automatic Verification of Hierarchical Multimedia Scenarios with Interactive Choice

More Decks by Jaime Arias Almeida

Other Decks in Research

Featured

Transcript