A Framework for Composition, Verification and Real-Time Performance of Multimedia Interactive Scenarios

Transcript

A Framework for Composition, Verification and Real-Time Performance of Multimedia

Interactive Scenarios Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda Laboratoire Bordelais de Recherche en Informatique (LaBRI) Université de Bordeaux 15th International Conference on Application of Concurrency to System Design (ACSD 2015) Belgium, June 2015 1

Motivation Interactive Multimedia Scenarios • Interactive Scores† (IS) is a

formalism for composing and interpreting interactive multimedia scenarios. • Some applications of IS: ◦ Live performances. ◦ Museum installations. ◦ Plastic art installations. †A. Allombert. Aspects Temporels d’un Système de Partitions Musicales Interactives pour la Composition et l’Exécution. Ph.D. thesis, Université de Bordeaux, 2009 Arias, Desainte-Catherine, and Rueda (LaBRI) A Framework for Composition, Verification and Real-Time Performance of IS 1/13 1/13

Motivation Interactive Multimedia Scenarios • Currently, IS is implemented in

the software i-score†. • A Hierarchical Time Stream Petri Net (HTSPN) model represents and executes the partially ordered set of events. †Website: http://www.i-score.org/ Arias, Desainte-Catherine, and Rueda (LaBRI) A Framework for Composition, Verification and Real-Time Performance of IS 2/13 2/13

Motivation Interactive Multimedia Scenarios • Some applications demand two features

that are not supported by the IS model†: ◦ Flexible control structures such as branching behavior. ◦ Automatic verification of scenarios in order to avoid abnormal behaviors. • Multimedia applications perform real-time, compute-intensive, and data-intensive tasks that sometimes the performance of standard computers is not sufficient. • Most of the multimedia applications are executed on architectures and operating systems that do not provide low-latency and real-time performance (e.g., i-score). †T. De la Hogue, P. Baltazar, M. Desainte-Catherine, J. Chao, and C. Bossut. OSSIA : Open Scenario System for Interactive Applications. In Journées d’Informatique Musicale, pages 78–84, Bourges, 2014 Arias, Desainte-Catherine, and Rueda (LaBRI) A Framework for Composition, Verification and Real-Time Performance of IS 3/13 3/13

Motivation Field Programmable Gate Arrays (FPGAs) • FPGAs have risen

over the last years and at the same time their cost have been reduced. • FPGAs have already been used with success in many different industrial applications†: ◦ Aerospace ◦ Automotive ◦ Medical ◦ Video and audio processing †J. Rodriguez-Andina, M. Moure, and M. Valdes. Features, Design Tools, and Application Domains of FPGAs. IEEE Transactions on Industrial Electronics, 54(4):1810–1823, Aug. 2007. ISSN 0278-0046. doi: 10.1109/TIE.2007.898279 Arias, Desainte-Catherine, and Rueda (LaBRI) A Framework for Composition, Verification and Real-Time Performance of IS 4/13 4/13

Motivation Field Programmable Gate Arrays (FPGAs) • FPGAs offer the

following benefits†: ◦ Reconfigurability ◦ High-level design ◦ Physical parallelism ◦ High-speed ◦ Reliability ◦ Re-use D Q Q Look-up Table (LUT) Flip Flop Mux Clock I0 I1 I2 I3 Out Conﬁgurable Logic Blocks (CLBs) Programmable Interconnection Network Conﬁgurable Input/Output Blocks (IOBs) †R. Dubey. Introduction to Embedded System Design Using Field Programmable Gate Arrays. Springer London, London, 2009 Arias, Desainte-Catherine, and Rueda (LaBRI) A Framework for Composition, Verification and Real-Time Performance of IS 5/13 5/13

Motivation Field Programmable Gate Arrays (FPGAs) (a) Performance comparison. (b)

Cost/performance comparison. Figure: A comparison between FPGA and DSP platforms for real-time video applications*. *L. Langlois. Multirate digital signal processing for high-speed data converters for high-speed data converters. Xcell Journal, 73:50–53, Q4 2010 Arias, Desainte-Catherine, and Rueda (LaBRI) A Framework for Composition, Verification and Real-Time Performance of IS 6/13 6/13

This talk is about … A Timed Automata semantics for

the specification and automatic verification of interactive multimedia scenarios with branching behavior, and their real-time execution on FPGAs. Arias, Desainte-Catherine, and Rueda (LaBRI) A Framework for Composition, Verification and Real-Time Performance of IS 7/13 7/13

Outline i-score scenario IS2UPP UPPAAL model Property checking UPP2C UPP2HDL

C/C++ code VHDL/System Verilog code Composition Verification Interpretation

Outline i-score scenario IS2UPP UPPAAL model Property checking UPP2C UPP2HDL

Interactive Scores An overview • An interactive scenario is composed

of: ◦ Temporal objects • Textures → Multimedia processes • Structures → Hierarchy ◦ Temporal relations • Synchronization → ∆min = ∆max = 0 • Rigid relations → ∆min = ∆max > 0 • flexible relations → ∆min = 0 ∧ ∆max = ∞ • Semi-flexible relations → ∆min ̸= ∆max ∧ ∆max ̸= ∞ ◦ Interaction points • The system maintains the temporal relations during execution. Arias, Desainte-Catherine, and Rueda (LaBRI) A Framework for Composition, Verification and Real-Time Performance of IS 6/13 6/13

Interactive Scores An overview time (s) 0 1 2 3

4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Texture B Texture A Structure C Structure D Texture F Texture G Texture E Arias, Desainte-Catherine, and Rueda (LaBRI) A Framework for Composition, Verification and Real-Time Performance of IS 7/13 7/13

Outline i-score scenario IS2UPP UPPAAL model Property checking UPP2C UPP2HDL

Timed Automata Model Temporal Relation t ≤ γ0 t ≤

γ1 idle wait_min skipped finished killed flexible semi flexible t ≥ 0, event_s?, {t = 0} true, skip_p?, ∅ true, skip!, {t = 0} true, event_e2!, {t = 0} true, kill!, {t = 0} true, kill_p?, ∅ true, kill_p?, ∅ true, event_i?, ∅ true, kill_p?, ∅ true, event_i?, ∅ t = γ1 , τ, ∅ true, kill_p?, ∅ t = γ0 ∧ ﬁnite = false, event_e1!, ∅ t = γ0 ∧ ﬁnite = true, event_e1 !, ∅ time (s) 0 1 2 3 4 5 6 Texture A Texture B γ0 = 8 γ1 = 24 finite = true t = 0 Arias, Desainte-Catherine, and Rueda (LaBRI) A Framework for Composition, Verification and Real-Time Performance of IS 7/13 7/13

Timed Automata Model Temporal Relation t ≤ γ0 t ≤

Timed Automata Model Handling Intervals idle error finished true, event_s2

, ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_e!, ∅ counter = n ∧ skip_v = true, skip!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = true counter = 0 Arias, Desainte-Catherine, and Rueda (LaBRI) A Framework for Composition, Verification and Real-Time Performance of IS 8/13 8/13

Timed Automata Model Handling Intervals idle error finished true, event_s2

, ∅ true, kill_p?, ∅ true, skip_p?, {counter + +} true, event_s1 ?, {counter + +, skip_v = false} counter < n, τ, ∅ counter = n ∧ skip_v = false, event_e!, ∅ counter = n ∧ skip_v = true, skip!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point n = 2 skipv = false counter = 1 Arias, Desainte-Catherine, and Rueda (LaBRI) A Framework for Composition, Verification and Real-Time Performance of IS 8/13 8/13

Timed Automata Model Handling Intervals idle error finished true, event_s2

Timed Automata Model Interaction Points idle finished skipped enabled timeout

cond = true true, skip_p?, ∅ en = false, event_e?, ∅ true, event_s?, {en = true} en = true, event_e?, ∅ true, kill_p?, ∅ true, skip!, ∅ true, kill_p?, ∅ urg = true, event_t!, ∅ urg = false, skip!, ∅ true, event?, {cond = condition()} true, event_t!, {en = false} true, event_e!, ∅ time 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point urg =true en =true cond =false Arias, Desainte-Catherine, and Rueda (LaBRI) A Framework for Composition, Verification and Real-Time Performance of IS 9/13 9/13

Timed Automata Model Interaction Points idle finished skipped enabled timeout

Timed Automata Model Multimedia Processes Brightness 100% 50% 30% 0%

p0 p1 p2 p3 p4 p5 p6 ∆1 ∆2 ∆3 ∆4 ∆5 ∆6 typedef struct { int value; int offset; } parameter_t; parameter_t process_brightness[7] = { {0,0}, {30,5}, {50,2}, ... }; idle finished wait t ≤ mp[i].offset t ≥ 0, start?, {t = 0, i = 0} true, kill_p?, {t = 0} true, skip_p?, {t = 0} true, kill_p?, {t = 0} true, stop?, {t = 0} t = m p[i].offset ∧ i = lim it, send!, {data = m p[i].value,t = 0} t = mp[i].offset ∧ i < limit, send!, {t = 0, data = mp[i].value, i + +} Arias, Desainte-Catherine, and Rueda (LaBRI) A Framework for Composition, Verification and Real-Time Performance of IS 10/13 10/13

Outline i-score scenario IS2UPP UPPAAL model Property checking UPP2C UPP2HDL

Verification Proving some properties… Some important properties are: • Is

a temporal constraint always satisfied? • Are all temporal relations always satisfied? • Is a multimedia ressource accessed by two or more processes? • Will the scenario finish? Arias, Desainte-Catherine, and Rueda (LaBRI) A Framework for Composition, Verification and Real-Time Performance of IS 10/13 10/13

Outline i-score scenario IS2UPP UPPAAL model Property checking UPP2C UPP2HDL

Real-time Performance Executing IS on FPGAs Next State State Register

Output Logic Inputs Outputs Mealy FSM Local Variables Clock Variables Global Clock Inputs Outputs Figure: Block diagram of the hardware implementation.† †Y. Jiang, H. Zhang, Z. Li, Y. Deng, X. Song, M. Gu, and J. Sun. Design and Optimization of Multiclocked Embedded Systems Using Formal Techniques. IEEE Transactions on Industrial Electronics, 62(2):1270–1278, 2015 Arias, Desainte-Catherine, and Rueda (LaBRI) A Framework for Composition, Verification and Real-Time Performance of IS 10/13 10/13

Real-time Performance Executing IS on FPGAs Figure: Simulation of our

example. Arias, Desainte-Catherine, and Rueda (LaBRI) A Framework for Composition, Verification and Real-Time Performance of IS 11/13 11/13

Concluding Remarks Summary • We presented a novel framework for

composition, verification and real-time performance of interactive scenarios. • Compilers have been developed for the automatic construction of the UPPAAL models. • Synthesizable SystemVerilog templates of the UPPAAL models have been developed for their execution on FPGAs. • To the best of our knowledge, none of the existing models of IS supports the automatic verification of scenarios or provides an implementation that guarantees their real-time execution. Arias, Desainte-Catherine, and Rueda (LaBRI) A Framework for Composition, Verification and Real-Time Performance of IS 12/13 12/13

Concluding Remarks Future work • Front-end for composers in order

to specify the properties. • Validate the SystemVerilog code using the formal specification language SVA (SystemVerilog Assertions) • Integration of a Fast Ethernet module in order to provide a low-rate and reliable communication. • Synthesize DSP programs into FPGAs (e.g., programs written in FAUST – http://faust.grame.fr). Arias, Desainte-Catherine, and Rueda (LaBRI) A Framework for Composition, Verification and Real-Time Performance of IS 13/13 13/13

Concluding Remarks Future work • Front-end for composers in order

to specify the properties. • Validate the SystemVerilog code using the formal specification language SVA† (SystemVerilog Assertions) • Integration of a Fast Ethernet module in order to provide a low-rate and reliable communication. • Synthesize DSP programs into FPGAs (e.g., programs written in FAUST – http://faust.grame.fr). †Y. Jiang, H. Zhang, Z. Li, Y. Deng, X. Song, M. Gu, and J. Sun. Design and Optimization of Multiclocked Embedded Systems Using Formal Techniques. IEEE Transactions on Industrial Electronics, 62(2):1270–1278, 2015 Arias, Desainte-Catherine, and Rueda (LaBRI) A Framework for Composition, Verification and Real-Time Performance of IS 13/13 13/13

Concluding Remarks Future work • Front-end for composers in order

to specify the properties. • Validate the SystemVerilog code using the formal specification language SVA (SystemVerilog Assertions) • Integration of a Fast Ethernet module in order to provide a low-rate and reliable communication†. • Synthesize DSP programs into FPGAs (e.g., programs written in FAUST – http://faust.grame.fr). †R. Aviziensis, A. Freed, T. Suzuki, and D. Wessel. Scalable Connectivity Processor for Computer Music Performance Systems. In International Computer Music Conference, 2000 Arias, Desainte-Catherine, and Rueda (LaBRI) A Framework for Composition, Verification and Real-Time Performance of IS 13/13 13/13

Concluding Remarks Future work • Front-end for composers in order

to specify the properties. • Validate the SystemVerilog code using the formal specification language SVA (SystemVerilog Assertions) • Integration of a Fast Ethernet module in order to provide a low-rate and reliable communication. • Synthesize DSP programs into FPGAs (e.g., programs written in FAUST† – http://faust.grame.fr). †R. Trausmuth, C. Dusek, and Y. Orlarey. Using FAUST for FPGA Programming. In 9th International Conference on Digital Audio Effects, pages 18–20, 2006 Arias, Desainte-Catherine, and Rueda (LaBRI) A Framework for Composition, Verification and Real-Time Performance of IS 13/13 13/13

Thank you for your attention! [email protected]

A Framework for Composition, Verification and Real-Time Performance of Multimedia

Interactive Scenarios Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda Laboratoire Bordelais de Recherche en Informatique (LaBRI) Université de Bordeaux 15th International Conference on Application of Concurrency to System Design (ACSD 2015) Belgium, June 2015 1

References Application and Theory of Petri Nets 1995, 16th International

Conference, Turin, Italy, June 26-30, 1995, Proceedings, volume 935 of LNCS, 1995. Springer. ISBN 3-540-60029-9. A. Allombert. Aspects Temporels d’un Système de Partitions Musicales Interactives pour la Composition et l’Exécution. Ph.D. thesis, Université de Bordeaux, 2009. A. Allombert, R. Marczak, M. Desainte-Catherine, P. Baltazar, and GarnierLaurent. VIRAGE : Designing An Interactive Intermedia Sequencer From Users Requirements And Theoretical Background. In International Computer Music Conference, 2010. R. Alur and D. L. Dill. A theory of timed automata. Theoretical Computer Science, 126(2):183–235, Apr. 1994. R. Alur, T. A. Henzinger, and M. Y. Vardi. Parametric real-time reasoning. In Proceedings of the 25th annual ACM symposium on Theory of computing, pages 592–601. ACM Press, 1993.

References J. Arias, M. Desainte-Catherine, and C. Rueda. Modelling Data

Processing for Interactive Scores Using Coloured Petri Nets. In 14th International Conference On Applications Of Concurrency To System Design, pages 186–195, 2014. R. Aviziensis, A. Freed, T. Suzuki, and D. Wessel. Scalable Connectivity Processor for Computer Music Performance Systems. In International Computer Music Conference, 2000. G. Behrmann, A. David, and K. G. Larsen. A Tutorial on Uppaal. In Formal Methods for the Design of Real-Time Systems, pages 200–236. Springer Berlin Heidelberg, Berlin, Heidelberg, Jan. 2004. E. Cerny, S. Dudani, J. Havlicek, and D. Korchemny. The Power of Assertions in SystemVerilog. 2010.

References T. De la Hogue, P. Baltazar, M. Desainte-Catherine, J.

Chao, and C. Bossut. OSSIA : Open Scenario System for Interactive Applications. In Journées d’Informatique Musicale, pages 78–84, Bourges, 2014. M. Desainte-Catherine, A. Allombert, and G. Assayag. Towards a hybrid temporal paradigm for musical composition and performance: The case of musical interpretation. Computer Music Journal, 37(2):61–72, 2013. J. S. Dong, Y. Liu, J. Sun, and X. Zhang. Towards Verification of Computation Orchestration. Formal Aspects of Computing, 26(4): 729–759, 2014. R. Dubey. Introduction to Embedded System Design Using Field Programmable Gate Arrays. Springer London, London, 2009.

References J. Echeveste, A. Cont, J.-L. Giavitto, and F. Jacquemard.

Operational Semantics of a Domain Specific Language for Real Time Musician–Computer Interaction. Discrete Event Dynamic Systems, 23 (4):343–383, Dec. 2013. S. Hauck and A. DeHon. Reconfigurable Computing: The Theory and Practice of FPGA-Based Computation. San Francisco, CA, USA, 2007. K. Jensen and L. M. Kristensen. Coloured Petri Nets: Modelling and Validation of Concurrent Systems. Springer Berlin Heidelberg, 2009. Y. Jiang, H. Zhang, Z. Li, Y. Deng, X. Song, M. Gu, and J. Sun. Design and Optimization of Multiclocked Embedded Systems Using Formal Techniques. IEEE Transactions on Industrial Electronics, 62(2): 1270–1278, 2015. L. Langlois. Multirate digital signal processing for high-speed data converters for high-speed data converters. Xcell Journal, 73:50–53, Q4 2010.

References K. G. Larsen, P. Pettersson, and W. Yi. Uppaal

in a nutshell. International Journal on Software Tools for Technology Transfer, 1(1-2):134–152, Dec. 1997. E. A. Lee. The problem with threads. Computer, 39(5):33–42, May 2006. ISSN 0018-9162. R. Marczak, M. Desainte-Catherine, and A. Allombert. Real-time temporal control of musical processes. In The Third International Conferences on Advances in Multimedia, pages 12–17, 2011. ISBN 978-1-61208-129-8. R. Milner. Communication and concurrency. Prentice Hall, 1989. ISBN 978-0-13-115007-2. E. Monmasson, L. Idkhajine, M. N. Cirstea, I. Bahri, A. Tisan, and M. W. Naouar. FPGAs in Industrial Control Applications. IEEE Transactions on Industrial Informatics, 7(2):224–243, 2011. ISSN 1551-3203. doi: 10.1109/TII.2011.2123908.

References C. Olarte and C. Rueda. A Declarative Language for

Dynamic Multimedia Interaction Systems. In Mathematics and Computation in Music, volume 38, pages 218–227. Springer Berlin Heidelberg, 2009. J. Rodriguez-Andina, M. Moure, and M. Valdes. Features, Design Tools, and Application Domains of FPGAs. IEEE Transactions on Industrial Electronics, 54(4):1810–1823, Aug. 2007. ISSN 0278-0046. doi: 10.1109/TIE.2007.898279. P. Sénac, P. de Saqui-Sannes, and R. Willrich. Hierarchical time stream petri net: A model for hypermedia systems. In Application and Theory of Petri Nets DBL [1995], pages 451–470. ISBN 3-540-60029-9. M. Toro, M. Desainte-Catherine, and C. Rueda. Formal semantics for interactive music scores: a framework to design, specify properties and execute interactive scenarios. Journal of Mathematics and Music, 8 (1):93–112, 2014. ISSN 1745-9737.

References R. Trausmuth, C. Dusek, and Y. Orlarey. Using FAUST

for FPGA Programming. In 9th International Conference on Digital Audio Effects, pages 18–20, 2006.

A Framework for Composition, Verification and R...

A Framework for Composition, Verification and Real-Time Performance of Multimedia Interactive Scenarios

More Decks by Jaime Arias Almeida

Other Decks in Research

Featured

Transcript