Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior Jaime Arias, Myriam Desainte-Catherine and Camilo Rueda Université de Bordeaux, LaBRI, UMR 5800 Inria - Bordeaux Sud-Ouest Colloquium Computer Science - University of Groningen March 2, 2016 

Motivation • Interactive multimedia = multimedia content + interaction with the environment + spatial and temporal organization • Some examples of interactive multimedia systems: ◦ Live performances. ◦ Museum installations. ◦ Plastic art installations. Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 1/17 1/17 

Motivation • Interactive Scores1(IS) is a formalism for composing and executing interactive multimedia scenarios. 1Antoine Allombert. “Aspects Temporels d’un Système de Partitions Musicales Interactives pour la Composition et l’Exécution”. PhD thesis. Université de Bordeaux, 2009 Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 2/17 2/17 

Motivation • Interactive Scores1(IS) is a formalism for composing and executing interactive multimedia scenarios. • An interactive scenario is composed of: ◦ Textures → multimedia processes ◦ Structures → hierarchy ◦ Temporal relations ◦ Interaction points 1Antoine Allombert. “Aspects Temporels d’un Système de Partitions Musicales Interactives pour la Composition et l’Exécution”. PhD thesis. Université de Bordeaux, 2009 Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 2/17 2/17 

Motivation • i-score2 is a software for interactive scenarios based on IS. ◦ Composition time → CSP problem ◦ Execution time → Timed Petri Net (TdPN) • IS mixes two temporal paradigms used in the current multimedia tools3: time-line and time-flow. • i-score has been used successfully during the last years4. 2Website: http://i-score.org/ 3Myriam Desainte-Catherine, Antoine Allombert, and Gérard Assayag. “Towards a Hybrid Temporal Paradigm for Musical Composition and Performance: The Case of Musical Interpretation”. Computer Music Journal 37.2 (2013), pp. 61–72. doi: 10.1162/COMJ_a_00179 4Jean-Michaël Celerier et al. “OSSIA: towards a unified interface for scoring time and interaction”. Proceedings of the First International Conference on Technologies for Music Notation and Representation - TENOR2015. Ed. by Marc Battier et al. Paris, France: Institut de Recherche en Musicologie, 2015, pp. 81–90. isbn: 978-2-9552905-0-7 Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 3/17 3/17 

Motivation • Nowadays, several applications (e.g., video games and interactive museum installations) increasingly demand5: ◦ Flexible control structures: conditionals and loops. ◦ Mechanisms for the automatic verification of scenarios. 5Théo de la Hogue et al. “OSSIA: Open Scenario System for Interactive Applications”. Journées d’Informatique Musicale, JIM 2014, Bourges, France, May 21-23, 2014. 2014, pp. 78–84 Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 4/17 4/17 

Motivation • Nowadays, several applications (e.g., video games and interactive museum installations) increasingly demand5: ◦ Flexible control structures: conditionals and loops. ◦ Mechanisms for the automatic verification of scenarios. • Several models have been proposed ◦ Olarte and Rueda. “A Declarative Language for Dynamic Multimedia Interaction Systems”. 2009 ◦ Toro, Desainte-Catherine, and Rueda. “Formal semantics for interactive music scores: a framework to design, specify properties and execute interactive scenarios”. 2014 5Théo de la Hogue et al. “OSSIA: Open Scenario System for Interactive Applications”. Journées d’Informatique Musicale, JIM 2014, Bourges, France, May 21-23, 2014. 2014, pp. 78–84 Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 4/17 4/17 

Motivation • Nowadays, several applications (e.g., video games and interactive museum installations) increasingly demand5: ◦ Flexible control structures: conditionals and loops. ◦ Mechanisms for the automatic verification of scenarios. • Several models have been proposed ◦ Olarte and Rueda. “A Declarative Language for Dynamic Multimedia Interaction Systems”. 2009 ◦ Toro, Desainte-Catherine, and Rueda. “Formal semantics for interactive music scores: a framework to design, specify properties and execute interactive scenarios”. 2014 • Disadvantages of existing models: ◦ They do not satisfy the composers’ demands. ◦ Real-time execution is not provided. ◦ There is no practical solutions for automatic verification. 5Théo de la Hogue et al. “OSSIA: Open Scenario System for Interactive Applications”. Journées d’Informatique Musicale, JIM 2014, Bourges, France, May 21-23, 2014. 2014, pp. 78–84 Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 4/17 4/17 

Our Goal We aim at developing a formal model for the specification and automatic verification of multimedia interactive scenarios with interactive choices. Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 5/17 5/17 

Timed Automata Framework 

Timed Automata Framework Overview i-score scenario IS2UPPAAL UPPAAL model Property Checking UPP2C UPP2HDL … C/C++ Code VHDL/System Verilog Code … Composition Verification Interpretation i-score scenario Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 6/17 6/17 

Timed Automata Framework Overview i-score scenario IS2UPPAAL UPPAAL model Property Checking UPP2C UPP2HDL … C/C++ Code VHDL/System Verilog Code … Composition Verification Interpretation Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 6/17 6/17 

Timed Automata Framework Overview i-score scenario IS2UPPAAL UPPAAL model Property Checking UPP2C UPP2HDL … C/C++ Code VHDL/System Verilog Code … Composition Verification Interpretation UPPAAL model Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 6/17 6/17 

Timed Automata Framework UPPAAL Model6 • A timed automaton is a finite-state machine equipped with clocks (i.e., real-valued variables). • Timed Automata has been used successfully to model and verify time-critical systems (e.g., multimedia applications): ◦ Echeveste, Cont, Giavitto, and Jacquemard. “Operational semantics of a domain specific language for real time musician-computer interaction”. 2013 ◦ Poncelet and Jacquemard. “Model based testing of an interactive music system”. 2015 6Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda. “A Framework for Composition, Verification and Real-Time Performance of Multimedia Interactive Scenarios”. 15th International Conference on Application of Concurrency to System Design, ACSD 2015, Brussels, Belgium, June 21-26, 2015. IEEE, 2015, pp. 140–151. Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 7/17 7/17 

Timed Automata Framework UPPAAL Model – Handling Intervals idle error finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = true counter = 0 en = false urg = true Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 8/17 8/17 

Timed Automata Framework UPPAAL Model – Handling Intervals idle error finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = true counter = 0 en = false urg = true Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 8/17 8/17 

Timed Automata Framework UPPAAL Model – Handling Intervals idle error finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = true counter = 0 en = false urg = true Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 8/17 8/17 

Timed Automata Framework UPPAAL Model – Handling Intervals idle error finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = true counter = 0 en = false urg = true Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 8/17 8/17 

Timed Automata Framework UPPAAL Model – Handling Intervals idle error finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = true counter = 0 en = false urg = true Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 8/17 8/17 

Timed Automata Framework UPPAAL Model – Handling Intervals idle error finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = true counter = 0 en = false urg = true Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 8/17 8/17 

Timed Automata Framework UPPAAL Model – Handling Intervals idle error finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = true counter = 0 en = false urg = true Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 8/17 8/17 

Timed Automata Framework UPPAAL Model – Handling Intervals idle error finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = true counter = 0 en = false urg = true Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 8/17 8/17 

Timed Automata Framework UPPAAL Model – Handling Intervals idle error finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = false counter = 1 en = false urg = true Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 8/17 8/17 

Timed Automata Framework UPPAAL Model – Handling Intervals idle error finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = false counter = 1 en = false urg = true Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 8/17 8/17 

Timed Automata Framework UPPAAL Model – Handling Intervals idle error finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = false counter = 1 en = false urg = true Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 8/17 8/17 

Timed Automata Framework UPPAAL Model – Handling Intervals idle error finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = false counter = 2 en = false urg = true Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 8/17 8/17 

Timed Automata Framework UPPAAL Model – Handling Intervals idle error finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = false counter = 2 en = true urg = true Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 8/17 8/17 

Timed Automata Framework UPPAAL Model – Handling Intervals idle error finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = false counter = 2 en = true urg = true Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 8/17 8/17 

Timed Automata Framework UPPAAL Model – Handling Intervals idle error finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = false counter = 2 en = true urg = true Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 8/17 8/17 

Timed Automata Framework UPPAAL Model – Handling Intervals idle error finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = false counter = 2 en = true urg = true Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 8/17 8/17 

Timed Automata Framework UPPAAL Model – Handling Intervals idle error finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = false counter = 2 en = true urg = true Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 8/17 8/17 

Timed Automata Framework UPPAAL Model – Handling Intervals idle error finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = false counter = 2 en = true urg = true Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 8/17 8/17 

Timed Automata Framework UPPAAL Model – Handling Intervals idle error finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = false counter = 2 en = true urg = true Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 8/17 8/17 

Timed Automata Framework UPPAAL Model – Handling Intervals idle error finished true, event_s2 , ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_m!, ∅ counter = n ∧ skip_v = true, skip!, ∅ idle finished skipped enabled timeout cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_m?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = eval(op, msg, v)} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = false counter = 2 en = true urg = true Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 8/17 8/17 

Timed Automata Framework UPPAAL Model – Multimedia Processes Brightness 100% 50% 30% 0% p0 p1 p2 p3 p4 p5 p6 ∆1 ∆2 ∆3 ∆4 ∆5 ∆6 Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 9/17 9/17 

Timed Automata Framework UPPAAL Model – Multimedia Processes Brightness 100% 50% 30% 0% p0 p1 p2 p3 p4 p5 p6 ∆1 ∆2 ∆3 ∆4 ∆5 ∆6 typedef struct { int value; int offset; } parameter_t; parameter_t process_brightness[7] = { {0,0}, {30,5}, {50,2}, ... }; Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 9/17 9/17 

Timed Automata Framework UPPAAL Model – Multimedia Processes Brightness 100% 50% 30% 0% p0 p1 p2 p3 p4 p5 p6 ∆1 ∆2 ∆3 ∆4 ∆5 ∆6 typedef struct { int value; int offset; } parameter_t; parameter_t process_brightness[7] = { {0,0}, {30,5}, {50,2}, ... }; idle finished wait t ≤ mp[i].offset t ≥ 0, start?, {t = 0, i = 0} true, kill_p?, {t = 0} true, skip_p?, {t = 0} true, kill_p?, {t = 0} true, stop?, {t = 0} t = m p[i].offset ∧ i = lim it, send!, {data = m p[i].value,t = 0} t = mp[i].offset ∧ i < limit, send!, {t = 0, data = mp[i].value, i + +} Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 9/17 9/17 

Timed Automata Framework Automatic Verification Some important properties are: • Will the scenario finish ? ◦ A <> Scenario.finished Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 10/17 10/17 

Timed Automata Framework Automatic Verification Some important properties are: • Will the scenario finish ? ◦ A <> Scenario.finished Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 10/17 10/17 

Timed Automata Framework Automatic Verification Some important properties are: • Will the scenario finish ? ◦ A <> Scenario.finished Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 10/17 10/17 

Timed Automata Framework Automatic Verification Some important properties are: • Will the scenario finish ? ◦ A <> Scenario .finished Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 10/17 10/17 

Timed Automata Framework Automatic Verification Some important properties are: • Will the scenario finish ? ◦ A <> Scenario. finished Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 10/17 10/17 

Timed Automata Framework Automatic Verification Some important properties are: • Will the scenario finish ? ◦ A <> Scenario.finished • Are all temporal relations always satisfied ? ◦ A[] !Control_Start_Group.error Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 10/17 10/17 

Timed Automata Framework Automatic Verification Some important properties are: • Will the scenario finish ? ◦ A <> Scenario.finished • Are all temporal relations always satisfied ? ◦ A[] !Control_Start_Group.error • Is a specific temporal constraint always satisfied ? ◦ A[] (Structure_Group.wait_min imply clk >= 4468) Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 10/17 10/17 

Timed Automata Framework Automatic Verification Some important properties are: • Will the scenario finish ? ◦ A <> Scenario.finished • Are all temporal relations always satisfied ? ◦ A[] !Control_Start_Group.error • Is a specific temporal constraint always satisfied ? ◦ A[] (Structure_Group.wait_min imply clk >= 4468) • Is a multimedia ressource used by two or more process at the same time ? ◦ A[] (!Texture_WH.wait || !Texture_LB.wait) Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 10/17 10/17 

Timed Automata Framework Overview i-score scenario IS2UPPAAL UPPAAL model Property Checking UPP2C UPP2HDL … C/C++ Code VHDL/System Verilog Code … Composition Verification Interpretation VHDL/System Verilog Code Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 11/17 11/17 

Timed Automata Framework True Parallel Implementation • Multimedia applications perform real-time, compute-intensive, and data-intensive tasks that sometimes the performance of standard computers is not sufficient. • Most of the multimedia applications are executed on architectures and operating systems that do not provide low-latency and real-time performance (e.g., i-score). Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 12/17 12/17 

Timed Automata Framework True Parallel Implementation • Multimedia applications perform real-time, compute-intensive, and data-intensive tasks that sometimes the performance of standard computers is not sufficient. • Most of the multimedia applications are executed on architectures and operating systems that do not provide low-latency and real-time performance (e.g., i-score). • FPGAs offer reconfigurability, high-level design, physical parallelism, high-speed, reliability, and re-use7. • FPGAs have already been used with success in many different industrial applications: aerospace, automotive, medical, video and audio processing. 7Stephen Trimberger. “Three Ages of FPGAs: A Retrospective on the First Thirty Years of FPGA Technology”. Proceedings of the IEEE 103.3 (2015), pp. 318–331. doi: 10.1109/JPROC.2015.2392104 Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 12/17 12/17 

Timed Automata Framework True Parallel Implementation (a) Performance comparison. (b) Cost/performance comparison. Figure: A comparison between FPGA and DSP platforms for real-time video applications8. 8Jeff Bier and Jennifer Eyre. “BDTI Study Certifies High-Level Synthesis Flows for DSP-Centric FPGA Design”. Xcell Journal 71 (2010), pp. 12–17. Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 13/17 13/17 

Timed Automata Framework True Parallel Implementation9 inputs clock FPGA Next State State Register Output Logic outputs Local Variables Clocks Variables Mealy FSM Timed Automaton Figure: Block diagram of the proposed hardware implementation of a timed automaton. 9Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda. “Exploiting Parallelism in FPGAs for the Real-Time Interpretation of Interactive Multimedia Scores”. Journées d’Informatique Musicale, JIM 2015, Montréal, Canada, May 7-9, 2015. 2015. Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 14/17 14/17 

Timed Automata Framework Overview i-score scenario IS2UPPAAL UPPAAL model Property Checking UPP2C UPP2HDL … C/C++ Code VHDL/System Verilog Code … Composition Verification Interpretation C/C++ Code10 … 10Tobias Amnell et al. “TIMES: A Tool for Schedulability Analysis and Code Generation of Real-Time Systems”. Formal Modeling and Analysis of Timed Systems: First International Workshop, FORMATS 2003, Marseille, France, September 6-7, 2003. Revised Papers. Ed. by Kim Guldstrand Larsen and Peter Niebert. Vol. 2791. Lecture Notes in Computer Science. Springer, 2003, pp. 60–72. doi: 10.1007/978-3-540-40903-8_6 Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 15/17 15/17 

Concluding Remarks 

Concluding Remarks Overview We proposed a framework for the specification, verification and execution of interactive scenarios. • Extension of interactive scenarios with guarded interaction points. • Automatic verification of scenarios using mature and robust tools (i.e., UPPAAL). • We showed the verification of some properties. • Synthesizing of scenarios into reconfigurable hardware, allowing their low-latency and real-time execution. Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 16/17 16/17 

Concluding Remarks Future Work • Extension of the Timed Automata model with : ◦ A coherent model for loops11. ◦ Notions for improvisation12. • Encoding of the Timed Automata model into the tiled programming13. • Validation of the hardware implementation using SystemVerilog Assertions14 • Study of other verification and validation techniques, such as Model-Based Testing and game semantics. 11Abdeldjalil Boudjadar, Frits W. Vaandrager, Jean-Paul Bodeveix, and Mamoun Filali. “Extending UPPAAL for the Modeling and Verification of Dynamic Real-Time Systems”. Fundamentals of Software Engineering - 5th International Conference, FSEN 2013, Tehran, Iran, April 24-26, 2013, Revised Selected Papers. Vol. 8161. Lecture Notes in Computer Science. Springer, 2013, pp. 111–132. doi: 10.1007/978-3-642-40213-5_8. 12Cheng-i Wang and Shlomo Dubnov. “The Variable Markov Oracle: Algorithms for Human Gesture Applications”. IEEE MultiMedia 22.4 (2015), pp. 52–67. doi: 10.1109/MMUL.2015.76. 13David Janin et al. “The T-calculus: Towards a Structured Programing of (Musical) Time and Space”. Proceedings of the First ACM SIGPLAN Workshop on Functional Art, Music, Modeling and Design. FARM ’13. Boston, Massachusetts, USA: ACM, 2013, pp. 23–34. isbn: 978-1-4503-2386-4. doi: 10.1145/2505341.2505347. 14Eduard Cerny, Surrendra Dudani, John Havlicek, and Dmitry Korchemny. SVA: The Power of Assertions in SystemVerilog. 2nd. Springer, 2014. isbn: 9783319071381. Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 17/17 17/17 

Thank you for your attention! 

Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior Jaime Arias, Myriam Desainte-Catherine and Camilo Rueda Université de Bordeaux, LaBRI, UMR 5800 Inria - Bordeaux Sud-Ouest Colloquium Computer Science - University of Groningen March 2, 2016 

References Allombert, Antoine. “Aspects Temporels d’un Système de Partitions Musicales Interactives pour la Composition et l’Exécution”. PhD thesis. Université de Bordeaux, 2009 (cited on slides 3, 4). Amnell, Tobias et al. “TIMES: A Tool for Schedulability Analysis and Code Generation of Real-Time Systems”. Formal Modeling and Analysis of Timed Systems: First International Workshop, FORMATS 2003, Marseille, France, September 6-7, 2003. Revised Papers. Ed. by Kim Guldstrand Larsen and Peter Niebert. Vol. 2791. Lecture Notes in Computer Science. Springer, 2003, pp. 60–72. doi: 10.1007/978-3-540-40903-8_6 (cited on slide 51). Arias, Jaime, Myriam Desainte-Catherine, and Camilo Rueda. “A Framework for Composition, Verification and Real-Time Performance of Multimedia Interactive Scenarios”. 15th International Conference on Application of Concurrency to System Design, ACSD 2015, Brussels, Belgium, June 21-26, 2015. IEEE, 2015, pp. 140–151 (cited on slide 14). – .“Exploiting Parallelism in FPGAs for the Real-Time Interpretation of Interactive Multimedia Scores”. Journées d’Informatique Musicale, JIM 2015, Montréal, Canada, May 7-9, 2015. 2015 (cited on slide 50). 

References Bier, Jeff and Jennifer Eyre. “BDTI Study Certifies High-Level Synthesis Flows for DSP-Centric FPGA Design”. Xcell Journal 71 (2010), pp. 12–17 (cited on slide 49). Boudjadar, Abdeldjalil et al. “Extending UPPAAL for the Modeling and Verification of Dynamic Real-Time Systems”. Fundamentals of Software Engineering - 5th International Conference, FSEN 2013, Tehran, Iran, April 24-26, 2013, Revised Selected Papers. Vol. 8161. Lecture Notes in Computer Science. Springer, 2013, pp. 111–132. doi: 10.1007/978-3-642-40213-5_8 (cited on slide 54). Celerier, Jean-Michaël et al. “OSSIA: towards a unified interface for scoring time and interaction”. Proceedings of the First International Conference on Technologies for Music Notation and Representation - TENOR2015. Ed. by Marc Battier et al. Paris, France: Institut de Recherche en Musicologie, 2015, pp. 81–90. isbn: 978-2-9552905-0-7 (cited on slide 5). Cerny, Eduard et al. SVA: The Power of Assertions in SystemVerilog. 2nd. Springer, 2014. isbn: 9783319071381 (cited on slide 54). de la Hogue, Théo et al. “OSSIA: Open Scenario System for Interactive Applications”. Journées d’Informatique Musicale, JIM 2014, Bourges, France, May 21-23, 2014. 2014, pp. 78–84 (cited on slides 6–8). 

References Desainte-Catherine, Myriam, Antoine Allombert, and Gérard Assayag. “Towards a Hybrid Temporal Paradigm for Musical Composition and Performance: The Case of Musical Interpretation”. Computer Music Journal 37.2 (2013), pp. 61–72. doi: 10.1162/COMJ_a_00179 (cited on slide 5). Echeveste, José et al. “Operational semantics of a domain specific language for real time musician-computer interaction”. Discrete Event Dynamic Systems 23.4 (2013), pp. 343–383. doi: 10.1007/s10626-013-0166-2 (cited on slide 14). Janin, David et al. “The T-calculus: Towards a Structured Programing of (Musical) Time and Space”. Proceedings of the First ACM SIGPLAN Workshop on Functional Art, Music, Modeling and Design. FARM ’13. Boston, Massachusetts, USA: ACM, 2013, pp. 23–34. isbn: 978-1-4503-2386-4. doi: 10.1145/2505341.2505347 (cited on slide 54). Olarte, Carlos and Camilo Rueda. “A Declarative Language for Dynamic Multimedia Interaction Systems”. 2nd International Conference on Mathematics and Computation in Music, MCM 2009, New Haven, CT, USA, June 19-22, 2009. Ed. by Elaine Chew, Adrian Childs, and Ching-Hua Chuan. Vol. 38. Communications in Computer and Information Science. Springer Berlin Heidelberg, 2009, pp. 218–227. isbn: 978-3-642-02393-4. doi: 10.1007/978-3-642-02394-1_20 (cited on slides 6–8). 

References Poncelet, Clément and Florent Jacquemard. “Model based testing of an interactive music system”. Proceedings of the 30th Annual ACM Symposium on Applied Computing, Salamanca, Spain, April 13-17, 2015. Ed. by Roger L. Wainwright et al. ACM, 2015, pp. 1759–1764. doi: 10.1145/2695664.2695804 (cited on slide 14). Toro, Mauricio, Myriam Desainte-Catherine, and Camilo Rueda. “Formal semantics for interactive music scores: a framework to design, specify properties and execute interactive scenarios”. Journal of Mathematics and Music 8.1 (2014), pp. 93–112. doi: 10.1080/17459737.2013.870610 (cited on slides 6–8). Trimberger, Stephen. “Three Ages of FPGAs: A Retrospective on the First Thirty Years of FPGA Technology”. Proceedings of the IEEE 103.3 (2015), pp. 318–331. doi: 10.1109/JPROC.2015.2392104 (cited on slide 48). Wang, Cheng-i and Shlomo Dubnov. “The Variable Markov Oracle: Algorithms for Human Gesture Applications”. IEEE MultiMedia 22.4 (2015), pp. 52–67. doi: 10.1109/MMUL.2015.76 (cited on slide 54).