Arias, Myriam Desainte-Catherine and Camilo Rueda Université de Bordeaux, LaBRI, UMR 5800 Inria - Bordeaux Sud-Ouest Colloquium Computer Science - University of Groningen March 2, 2016
the environment + spatial and temporal organization • Some examples of interactive multimedia systems: ◦ Live performances. ◦ Museum installations. ◦ Plastic art installations. Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 1/17 1/17
executing interactive multimedia scenarios. 1Antoine Allombert. “Aspects Temporels d’un Système de Partitions Musicales Interactives pour la Composition et l’Exécution”. PhD thesis. Université de Bordeaux, 2009 Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 2/17 2/17
on IS. ◦ Composition time → CSP problem ◦ Execution time → Timed Petri Net (TdPN) • IS mixes two temporal paradigms used in the current multimedia tools3: time-line and time-flow. • i-score has been used successfully during the last years4. 2Website: http://i-score.org/ 3Myriam Desainte-Catherine, Antoine Allombert, and Gérard Assayag. “Towards a Hybrid Temporal Paradigm for Musical Composition and Performance: The Case of Musical Interpretation”. Computer Music Journal 37.2 (2013), pp. 61–72. doi: 10.1162/COMJ_a_00179 4Jean-Michaël Celerier et al. “OSSIA: towards a unified interface for scoring time and interaction”. Proceedings of the First International Conference on Technologies for Music Notation and Representation - TENOR2015. Ed. by Marc Battier et al. Paris, France: Institut de Recherche en Musicologie, 2015, pp. 81–90. isbn: 978-2-9552905-0-7 Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 3/17 3/17
museum installations) increasingly demand5: ◦ Flexible control structures: conditionals and loops. ◦ Mechanisms for the automatic verification of scenarios. 5Théo de la Hogue et al. “OSSIA: Open Scenario System for Interactive Applications”. Journées d’Informatique Musicale, JIM 2014, Bourges, France, May 21-23, 2014. 2014, pp. 78–84 Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 4/17 4/17
museum installations) increasingly demand5: ◦ Flexible control structures: conditionals and loops. ◦ Mechanisms for the automatic verification of scenarios. • Several models have been proposed ◦ Olarte and Rueda. “A Declarative Language for Dynamic Multimedia Interaction Systems”. 2009 ◦ Toro, Desainte-Catherine, and Rueda. “Formal semantics for interactive music scores: a framework to design, specify properties and execute interactive scenarios”. 2014 5Théo de la Hogue et al. “OSSIA: Open Scenario System for Interactive Applications”. Journées d’Informatique Musicale, JIM 2014, Bourges, France, May 21-23, 2014. 2014, pp. 78–84 Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 4/17 4/17
museum installations) increasingly demand5: ◦ Flexible control structures: conditionals and loops. ◦ Mechanisms for the automatic verification of scenarios. • Several models have been proposed ◦ Olarte and Rueda. “A Declarative Language for Dynamic Multimedia Interaction Systems”. 2009 ◦ Toro, Desainte-Catherine, and Rueda. “Formal semantics for interactive music scores: a framework to design, specify properties and execute interactive scenarios”. 2014 • Disadvantages of existing models: ◦ They do not satisfy the composers’ demands. ◦ Real-time execution is not provided. ◦ There is no practical solutions for automatic verification. 5Théo de la Hogue et al. “OSSIA: Open Scenario System for Interactive Applications”. Journées d’Informatique Musicale, JIM 2014, Bourges, France, May 21-23, 2014. 2014, pp. 78–84 Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 4/17 4/17
a finite-state machine equipped with clocks (i.e., real-valued variables). • Timed Automata has been used successfully to model and verify time-critical systems (e.g., multimedia applications): ◦ Echeveste, Cont, Giavitto, and Jacquemard. “Operational semantics of a domain specific language for real time musician-computer interaction”. 2013 ◦ Poncelet and Jacquemard. “Model based testing of an interactive music system”. 2015 6Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda. “A Framework for Composition, Verification and Real-Time Performance of Multimedia Interactive Scenarios”. 15th International Conference on Application of Concurrency to System Design, ACSD 2015, Brussels, Belgium, June 21-26, 2015. IEEE, 2015, pp. 140–151. Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 7/17 7/17
Will the scenario finish ? ◦ A <> Scenario.finished • Are all temporal relations always satisfied ? ◦ A[] !Control_Start_Group.error • Is a specific temporal constraint always satisfied ? ◦ A[] (Structure_Group.wait_min imply clk >= 4468) • Is a multimedia ressource used by two or more process at the same time ? ◦ A[] (!Texture_WH.wait || !Texture_LB.wait) Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 10/17 10/17
real-time, compute-intensive, and data-intensive tasks that sometimes the performance of standard computers is not sufficient. • Most of the multimedia applications are executed on architectures and operating systems that do not provide low-latency and real-time performance (e.g., i-score). Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 12/17 12/17
real-time, compute-intensive, and data-intensive tasks that sometimes the performance of standard computers is not sufficient. • Most of the multimedia applications are executed on architectures and operating systems that do not provide low-latency and real-time performance (e.g., i-score). • FPGAs offer reconfigurability, high-level design, physical parallelism, high-speed, reliability, and re-use7. • FPGAs have already been used with success in many different industrial applications: aerospace, automotive, medical, video and audio processing. 7Stephen Trimberger. “Three Ages of FPGAs: A Retrospective on the First Thirty Years of FPGA Technology”. Proceedings of the IEEE 103.3 (2015), pp. 318–331. doi: 10.1109/JPROC.2015.2392104 Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 12/17 12/17
Cost/performance comparison. Figure: A comparison between FPGA and DSP platforms for real-time video applications8. 8Jeff Bier and Jennifer Eyre. “BDTI Study Certifies High-Level Synthesis Flows for DSP-Centric FPGA Design”. Xcell Journal 71 (2010), pp. 12–17. Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 13/17 13/17
State State Register Output Logic outputs Local Variables Clocks Variables Mealy FSM Timed Automaton Figure: Block diagram of the proposed hardware implementation of a timed automaton. 9Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda. “Exploiting Parallelism in FPGAs for the Real-Time Interpretation of Interactive Multimedia Scores”. Journées d’Informatique Musicale, JIM 2015, Montréal, Canada, May 7-9, 2015. 2015. Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 14/17 14/17
Checking UPP2C UPP2HDL … C/C++ Code VHDL/System Verilog Code … Composition Verification Interpretation C/C++ Code10 … 10Tobias Amnell et al. “TIMES: A Tool for Schedulability Analysis and Code Generation of Real-Time Systems”. Formal Modeling and Analysis of Timed Systems: First International Workshop, FORMATS 2003, Marseille, France, September 6-7, 2003. Revised Papers. Ed. by Kim Guldstrand Larsen and Peter Niebert. Vol. 2791. Lecture Notes in Computer Science. Springer, 2003, pp. 60–72. doi: 10.1007/978-3-540-40903-8_6 Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 15/17 15/17
verification and execution of interactive scenarios. • Extension of interactive scenarios with guarded interaction points. • Automatic verification of scenarios using mature and robust tools (i.e., UPPAAL). • We showed the verification of some properties. • Synthesizing of scenarios into reconfigurable hardware, allowing their low-latency and real-time execution. Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 16/17 16/17
model with : ◦ A coherent model for loops11. ◦ Notions for improvisation12. • Encoding of the Timed Automata model into the tiled programming13. • Validation of the hardware implementation using SystemVerilog Assertions14 • Study of other verification and validation techniques, such as Model-Based Testing and game semantics. 11Abdeldjalil Boudjadar, Frits W. Vaandrager, Jean-Paul Bodeveix, and Mamoun Filali. “Extending UPPAAL for the Modeling and Verification of Dynamic Real-Time Systems”. Fundamentals of Software Engineering - 5th International Conference, FSEN 2013, Tehran, Iran, April 24-26, 2013, Revised Selected Papers. Vol. 8161. Lecture Notes in Computer Science. Springer, 2013, pp. 111–132. doi: 10.1007/978-3-642-40213-5_8. 12Cheng-i Wang and Shlomo Dubnov. “The Variable Markov Oracle: Algorithms for Human Gesture Applications”. IEEE MultiMedia 22.4 (2015), pp. 52–67. doi: 10.1109/MMUL.2015.76. 13David Janin et al. “The T-calculus: Towards a Structured Programing of (Musical) Time and Space”. Proceedings of the First ACM SIGPLAN Workshop on Functional Art, Music, Modeling and Design. FARM ’13. Boston, Massachusetts, USA: ACM, 2013, pp. 23–34. isbn: 978-1-4503-2386-4. doi: 10.1145/2505341.2505347. 14Eduard Cerny, Surrendra Dudani, John Havlicek, and Dmitry Korchemny. SVA: The Power of Assertions in SystemVerilog. 2nd. Springer, 2014. isbn: 9783319071381. Jaime Arias - LaBRI - Inria Bordeaux Sud-Ouest (2016) Automatic Verification of Interactive Multimedia Scenarios with Branching Behavior 17/17 17/17
Arias, Myriam Desainte-Catherine and Camilo Rueda Université de Bordeaux, LaBRI, UMR 5800 Inria - Bordeaux Sud-Ouest Colloquium Computer Science - University of Groningen March 2, 2016
Interactives pour la Composition et l’Exécution”. PhD thesis. Université de Bordeaux, 2009 (cited on slides 3, 4). Amnell, Tobias et al. “TIMES: A Tool for Schedulability Analysis and Code Generation of Real-Time Systems”. Formal Modeling and Analysis of Timed Systems: First International Workshop, FORMATS 2003, Marseille, France, September 6-7, 2003. Revised Papers. Ed. by Kim Guldstrand Larsen and Peter Niebert. Vol. 2791. Lecture Notes in Computer Science. Springer, 2003, pp. 60–72. doi: 10.1007/978-3-540-40903-8_6 (cited on slide 51). Arias, Jaime, Myriam Desainte-Catherine, and Camilo Rueda. “A Framework for Composition, Verification and Real-Time Performance of Multimedia Interactive Scenarios”. 15th International Conference on Application of Concurrency to System Design, ACSD 2015, Brussels, Belgium, June 21-26, 2015. IEEE, 2015, pp. 140–151 (cited on slide 14). – .“Exploiting Parallelism in FPGAs for the Real-Time Interpretation of Interactive Multimedia Scores”. Journées d’Informatique Musicale, JIM 2015, Montréal, Canada, May 7-9, 2015. 2015 (cited on slide 50).
Synthesis Flows for DSP-Centric FPGA Design”. Xcell Journal 71 (2010), pp. 12–17 (cited on slide 49). Boudjadar, Abdeldjalil et al. “Extending UPPAAL for the Modeling and Verification of Dynamic Real-Time Systems”. Fundamentals of Software Engineering - 5th International Conference, FSEN 2013, Tehran, Iran, April 24-26, 2013, Revised Selected Papers. Vol. 8161. Lecture Notes in Computer Science. Springer, 2013, pp. 111–132. doi: 10.1007/978-3-642-40213-5_8 (cited on slide 54). Celerier, Jean-Michaël et al. “OSSIA: towards a unified interface for scoring time and interaction”. Proceedings of the First International Conference on Technologies for Music Notation and Representation - TENOR2015. Ed. by Marc Battier et al. Paris, France: Institut de Recherche en Musicologie, 2015, pp. 81–90. isbn: 978-2-9552905-0-7 (cited on slide 5). Cerny, Eduard et al. SVA: The Power of Assertions in SystemVerilog. 2nd. Springer, 2014. isbn: 9783319071381 (cited on slide 54). de la Hogue, Théo et al. “OSSIA: Open Scenario System for Interactive Applications”. Journées d’Informatique Musicale, JIM 2014, Bourges, France, May 21-23, 2014. 2014, pp. 78–84 (cited on slides 6–8).
Hybrid Temporal Paradigm for Musical Composition and Performance: The Case of Musical Interpretation”. Computer Music Journal 37.2 (2013), pp. 61–72. doi: 10.1162/COMJ_a_00179 (cited on slide 5). Echeveste, José et al. “Operational semantics of a domain specific language for real time musician-computer interaction”. Discrete Event Dynamic Systems 23.4 (2013), pp. 343–383. doi: 10.1007/s10626-013-0166-2 (cited on slide 14). Janin, David et al. “The T-calculus: Towards a Structured Programing of (Musical) Time and Space”. Proceedings of the First ACM SIGPLAN Workshop on Functional Art, Music, Modeling and Design. FARM ’13. Boston, Massachusetts, USA: ACM, 2013, pp. 23–34. isbn: 978-1-4503-2386-4. doi: 10.1145/2505341.2505347 (cited on slide 54). Olarte, Carlos and Camilo Rueda. “A Declarative Language for Dynamic Multimedia Interaction Systems”. 2nd International Conference on Mathematics and Computation in Music, MCM 2009, New Haven, CT, USA, June 19-22, 2009. Ed. by Elaine Chew, Adrian Childs, and Ching-Hua Chuan. Vol. 38. Communications in Computer and Information Science. Springer Berlin Heidelberg, 2009, pp. 218–227. isbn: 978-3-642-02393-4. doi: 10.1007/978-3-642-02394-1_20 (cited on slides 6–8).
an interactive music system”. Proceedings of the 30th Annual ACM Symposium on Applied Computing, Salamanca, Spain, April 13-17, 2015. Ed. by Roger L. Wainwright et al. ACM, 2015, pp. 1759–1764. doi: 10.1145/2695664.2695804 (cited on slide 14). Toro, Mauricio, Myriam Desainte-Catherine, and Camilo Rueda. “Formal semantics for interactive music scores: a framework to design, specify properties and execute interactive scenarios”. Journal of Mathematics and Music 8.1 (2014), pp. 93–112. doi: 10.1080/17459737.2013.870610 (cited on slides 6–8). Trimberger, Stephen. “Three Ages of FPGAs: A Retrospective on the First Thirty Years of FPGA Technology”. Proceedings of the IEEE 103.3 (2015), pp. 318–331. doi: 10.1109/JPROC.2015.2392104 (cited on slide 48). Wang, Cheng-i and Shlomo Dubnov. “The Variable Markov Oracle: Algorithms for Human Gesture Applications”. IEEE MultiMedia 22.4 (2015), pp. 52–67. doi: 10.1109/MMUL.2015.76 (cited on slide 54).
himito
ykaneko1992
umepon
tam17aki
waptech
masakat0
yuukit
yumulab
yururoi
tarugoconf
j20232
s_aiueo32
shinyorke
jasonvnalue
lemiorhan
bkeepers
brianwarren
sferik
marcelosomers
matthewcrist
addyosmani
lauravandoore
paulrobertlloyd
eitanlees
brad_frost
