Foundations for Reliable and Flexible Interactive Multimedia Scores

Transcript

1. Foundations for Reliable and Flexible Interactive Multimedia Scores Jaime Arias,

   Myriam Desainte-Catherine, Carlos Olarte, and Camilo Rueda Laboratoire Bordelais de Recherche en Informatique (LaBRI) Université de Bordeaux 5th International Conference on Mathematics and Computation in Music (MCM 2015) United Kingdom, June 2015 1 
   

2. Motivation Interactive Multimedia Scenarios • Interactive Scores† (IS) is a

   formalism for composing and interpreting interactive multimedia scenarios. • Some applications of IS: ◦ Live performances ◦ Museum installations ◦ Plastic art installations †A. Allombert. Aspects Temporels d’un Système de Partitions Musicales Interactives pour la Composition et l’Exécution. PhD thesis, Université de Bordeaux, 2009 Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 1/13 1/13

3. Motivation Interactive Multimedia Scenarios • Currently, IS is implemented in

   the software i-score†. • A Hierarchical Time Stream Petri Net (HTSPN) model represents and executes the partially ordered set of events. †Website: http://www.i-score.org/ Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 2/13 2/13

4. Motivation Interactive Multimedia Scenarios • Some applications demand two features

   that are not supported by the IS model†: ◦ Flexible control structures such as conditionals. ◦ Automatic verification of scenarios in order to avoid abnormal behaviors. • The specification of some behaviors could be cumbersome due to the horizontal time-line of i-score. †T. De la Hogue, P. Baltazar, M. Desainte-Catherine, J. Chao, and C. Bossut. OSSIA : Open Scenario System for Interactive Applications. In Journées d’Informatique Musicale, pages 78–84, Bourges, 2014 Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 3/13 3/13

5. This talk is about … ReactiveIS: a programming language for

   the specification and execution of interactive multimedia scenarios endowed with a tree-based operational semantics and a declarative interpretation in intuitionistic linear logic. Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 4/13 4/13

6. Outline ReactiveIS Syntax Program Tree State Tree Operational Semantics Logical

   Characterization Concluding Remarks

7. Outline ReactiveIS Syntax Program Tree State Tree Operational Semantics Logical

   Characterization Concluding Remarks

8. ReactiveIS Syntax ⟨score⟩ ::= ⟨structure⟩ ⟨texture⟩ ::= texture(⟨params⟩ ⟨msg⟩ ⟨msg⟩)

   ⟨structure⟩ ::= structure(⟨params⟩ ⟨TO-list⟩) ⟨params⟩ ::= ⟨name⟩ ⟨condition⟩ ⟨condition⟩ ⟨TO-event⟩ ::= start ⟨name⟩ | end ⟨name⟩ ⟨condition⟩ ::= wait(⟨TO-event⟩ ⟨min⟩ ⟨max⟩) | event ⟨msg⟩ | (⟨condition⟩ ∧ ⟨condition⟩) | (⟨condition⟩ ∨ ⟨condition⟩) time (s) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Texture B Texture A Structure C Structure D Texture E Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 5/13 5/13

9. ReactiveIS Syntax ... Structure C = { start.c = (Wait(End(A),5,5)

   & Wait(End(B),3,3)); stop.c = (Wait(End(D),0,INF) & Wait(End(E),0,INF)); Texture D = { start.c = ((Wait(Start(C),2,5) & Event("/mouse 1")) | Wait(Start(C),5,5)); stop.c = Wait(Start(D),1,1); start.msg = "/sound/1 on"; stop.msg = "/sound/1 off"; }; ... }; time (s) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Texture B Texture A Structure C Structure D Texture E Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 6/13 6/13

10. ReactiveIS Program Tree start: stop: True EndScenario Conditionals start: stop:

    WaitFromEnd(A,5,5) ∧ WaitFromEnd(B,3,3) WaitFromEnd(C.D,0,∞) ∧ WaitFromEnd(C.E,0,∞) Conditionals C start: stop: WaitFromStart(C,1,1) WaitFromStart(C.E,2,2) Conditionals Messages start: stop: /sound/2 on /sound/2 off E B A D Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 7/13 7/13

11. ReactiveIS State Tree start: stop: 0 ⊥ Times Arias, Desainte-Catherine,

    Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 8/13 8/13

12. ReactiveIS State Tree start: stop: 0 ⊥ Times start: stop:

    1 3 Times A Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 8/13 8/13

13. ReactiveIS State Tree start: stop: 0 ⊥ Times start: stop:

    3 5 Times B start: stop: 1 3 Times A Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 8/13 8/13

14. ReactiveIS State Tree start: stop: 0 ⊥ Times start: stop:

    3 5 Times B start: stop: 1 3 Times A start: stop: 8 ⊥ Times C start(S,C,8) Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 8/13 8/13

15. ReactiveIS State Tree start: stop: 0 ⊥ Times start: stop:

    3 5 Times B start: stop: 1 3 Times A start: stop: 8 14 Times C stop(S,C,14) Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 8/13 8/13

16. ReactiveIS Operational Semantics RSTART p ∈ canStart(S, P) ⟨P, S,

    I, t⟩ |= cs(n) ⟨St, O⟩I,t S −→P ⟨start(St, p, t), O ∪ {ms(n)}⟩I,t S where n = targetP(p) RSTOP p ∈ canStop(S) ⟨P, S, I, t⟩ |= ce(n) ⟨St, O⟩I,t S −→P ⟨stop(St, p, t), O ∪ {me(n)}⟩I,t S where n = targetP(p) RTIME ⟨S, ∅⟩I,t S −→∗ P ⟨S′, O⟩I,t S ̸ −→P ⟨S, t⟩ I,O =⇒P ⟨S′, t + 1⟩ Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 9/13 9/13

17. ReactiveIS Operational Semantics RSTART p ∈ canStart(S, P) ⟨P, S,

    I, t⟩ |= cs(n) ⟨St, O⟩I,t S −→P ⟨start(St, p, t), O ∪ {ms(n)}⟩I,t S where n = targetP(p) palive(S) = {p | p ∈ L(S) ∧ te(targetS (p)) = ⊥} 0 ⊥ 8 ⊥ 3 5 1 3 C A B palive (S) Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 9/13 9/13

18. ReactiveIS Operational Semantics RSTART p ∈ canStart(S, P) ⟨P, S,

    I, t⟩ |= cs(n) ⟨St, O⟩I,t S −→P ⟨start(St, p, t), O ∪ {ms(n)}⟩I,t S where n = targetP(p) palive(S) = {p | p ∈ L(S) ∧ te(targetS (p)) = ⊥} canStart(P, S) △ = {p | pparent ∈ palive(S) ∧ p ∈ Children(pparent )} \ L(S) C A B E D canStart(P, S) 0 ⊥ 8 ⊥ 3 5 1 3 C A B palive (S) Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 9/13 9/13

19. ReactiveIS Operational Semantics RSTART p ∈ canStart(S, P) ⟨P, S,

    I, t⟩ |= cs(n) ⟨St, O⟩I,t S −→P ⟨start(St, p, t), O ∪ {ms(n)}⟩I,t S where n = targetP(p) cs (targetP (C.E)) = WaitFromStart(C, 1, 1) ⟨P, S, I, t⟩ | = WaitFromStart(p, t1, t2) iff ∃n · n ∈ V (S) ∧ n = targetS (p) ∧ t1 ≤ t − ts (n) ≤ t2 0 ⊥ 8 ⊥ 3 5 1 3 C A B t = 9 Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 9/13 9/13

20. ReactiveIS Operational Semantics RSTOP p ∈ canStop(S) ⟨P, S, I,

    t⟩ |= ce(n) ⟨St, O⟩I,t S −→P ⟨stop(St, p, t), O ∪ {me(n)}⟩I,t S where n = targetP(p) canStop(S) △ = {p | p ∈ L(S) ∧ te(targetS (p)) = ⊥} = palive(S) Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 9/13 9/13

21. ReactiveIS Operational Semantics RSTART p ∈ canStart(S, P) ⟨P, S,

    I, t⟩ |= cs(n) ⟨St, O⟩I,t S −→P ⟨start(St, p, t), O ∪ {ms(n)}⟩I,t S where n = targetP(p) RSTOP p ∈ canStop(S) ⟨P, S, I, t⟩ |= ce(n) ⟨St, O⟩I,t S −→P ⟨stop(St, p, t), O ∪ {me(n)}⟩I,t S where n = targetP(p) RTIME ⟨S, ∅⟩I,t S −→∗ P ⟨S′, O⟩I,t S ̸ −→P ⟨S, t⟩ I,O =⇒P ⟨S′, t + 1⟩ Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 9/13 9/13

22. Outline ReactiveIS Syntax Program Tree State Tree Operational Semantics Logical

    Characterization Concluding Remarks

23. Logical Characterization Subexponentials in Linear Logic (SELL) Girard’s Linear Logic†:

    Formulas are seen as resources, e.g, c ⊗ c ̸−→ c. Subexponentials: meanings for !aF • “F holds in location a” (e.g., in a given structure) • “F holds with certain modality a” (e.g., in a time-unit t). • Subexp. are ordered in a poset. • The ordering determines the provability relation, e.g.,: !aF −→!bF iff b ⪯ a • We then have a better control of resources in proofs. 0+ 0 1+ 1 2+ 2 3+ 3 … where t is defined as t TI t.i t.o t.s.scenario t.s.A t.s.B t.s.C t.s.D t.s.E t.p TP †J. Girard. Linear logic. Theor. Comput. Sci., 50:1–102, 1987 Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 10/13 10/13

24. Logical Characterization Declarative Meaning of ReactiveIS programs We defined an

    encoding from ReactiveIS programs into SELL formulas. Adequacy P reaches a state s iff the sequent [[P]] −→ [[s]] is provable in SELL. This opens the possibility of reasoning about IS by using well established techniques in proof theory! We can verify properties such as: • Is it possible for the structure A and B to be played concurrently ? • Is the execution of A always preceded by an execution of B ? • Will all the structures be eventually played ? Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 11/13 11/13

25. Outline ReactiveIS Syntax Program Tree State Tree Operational Semantics Logical

    Characterization Concluding Remarks

26. Concluding Remarks Summary • We presented a new programming language

    for the specification, verification and interpretation of interactive scores. • ReactiveIS extends the full capacity of temporal organization of IS and also it allows the specification of conditionals. • The operational semantics based on labelled trees are simpler and more intuitive than the current model in HTSPN of i-score. • We presented a declarative interpretation of ReactiveIS programs as formulas in ILL with subexponentials that is adequate. Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 12/13 12/13

27. Concluding Remarks Future work • Coq for proving the correcteness

    of the semantics. • Front-end for the specification of properties. • Extension for loops. Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 13/13 13/13

28. Thank you for your attention! jaime.arias@labri.fr

29. Foundations for Reliable and Flexible Interactive Multimedia Scores Jaime Arias,

    Myriam Desainte-Catherine, Carlos Olarte, and Camilo Rueda Laboratoire Bordelais de Recherche en Informatique (LaBRI) Université de Bordeaux 5th International Conference on Mathematics and Computation in Music (MCM 2015) United Kingdom, June 2015 1 
    

30. References CONCUR 2013 - Concurrency Theory - 24th International Conference,

    CONCUR 2013, Buenos Aires, Argentina, August 27-30, 2013. Proceedings, volume 8052 of Lecture Notes in Computer Science, 2013. Springer. ISBN 978-3-642-40183-1. A. Allombert. Aspects Temporels d’un Système de Partitions Musicales Interactives pour la Composition et l’Exécution. PhD thesis, Université de Bordeaux, 2009. A. Allombert, R. Marczak, M. Desainte-Catherine, P. Baltazar, and GarnierLaurent. Virage : Designing An Interactive Intermedia Sequencer From Users Requirements And Theoretical Background. In International Computer Music Conference, 2010. J.-M. Andreoli. Logic programming with focusing proofs in linear logic. J. Log. Comput., 2(3):297–347, 1992.

31. References M. Andries, G. Engels, A. Habel, B. Hoffmann, H.

    Kreowski, S. Kuske, D. Plump, A. Schürr, and G. Taentzer. Graph transformation for specification and programming. Sci. Comput. Program., 34(1):1–54, 1999. doi: 10.1016/S0167-6423(98)00023-9. URL http://dx.doi.org/10.1016/S0167-6423(98)00023-9. J. Arias, M. Desainte-Catherine, C. Olarte, and C. Rueda. Foundations for reliable and flexible interactive multimedia scores. Technical report, Labri, University of Bordeaux, Mar. 2015. V. Danos, J. Joinet, and H. Schellinx. The structure of exponentials: Uncovering the dynamics of linear logic proofs. In Gottlob et al. [1993], pages 159–171. ISBN 3-540-57184-1. doi: 10.1007/BFb0022564. URL http://dx.doi.org/10.1007/BFb0022564.

32. References T. De la Hogue, P. Baltazar, M. Desainte-Catherine, J.

    Chao, and C. Bossut. OSSIA : Open Scenario System for Interactive Applications. In Journées d’Informatique Musicale, pages 78–84, Bourges, 2014. M. Desainte-Catherine, A. Allombert, and G. Assayag. Towards a hybrid temporal paradigm for musical composition and performance: The case of musical interpretation. Computer Music Journal, 37(2):61–72, 2013. doi: 10.1162/COMJ_a_00179. URL http://dx.doi.org/10.1162/COMJ_a_00179. J. Girard. Linear logic. Theor. Comput. Sci., 50:1–102, 1987. G. Gottlob, A. Leitsch, and D. Mundici, editors. Computational Logic and Proof Theory, Third Kurt Gödel Colloquium, KGC’93, Brno, Czech Republic, August 24-27, 1993, Proceedings, volume 713 of Lecture Notes in Computer Science, 1993. Springer. ISBN 3-540-57184-1.

33. References T. Hoare, G. Menzel, and J. Misra. A tree

    semantics of an orchestration language. In Engineering Theories of Software Intensive Systems, volume 195 of NATO Science Series II: Mathematics, Physics and Chemistry, pages 331–350. Springer, 2005. ISBN 978-1-4020-3530-2. doi: 10.1007/1-4020-3532-2_11. R. Marczak, M. Desainte-Catherine, and A. Allombert. Real-time temporal control of musical processes. In The Third International Conferences on Advances in Multimedia, MMEDIA 2011, pages 12–17, 2011. ISBN 978-1-61208-129-8. G. D. Michelis and M. Diaz, editors. Application and Theory of Petri Nets 1995, 16th International Conference, Turin, Italy, June 26-30, 1995, Proceedings, volume 935 of Lecture Notes in Computer Science, 1995. Springer. ISBN 3-540-60029-9.

34. References V. Nigam, C. Olarte, and E. Pimentel. A general

    proof system for modalities in concurrent constraint programming. In CONCUR DBL [2013], pages 410–424. ISBN 978-3-642-40183-1. C. Olarte and C. Rueda. A Declarative Language for Dynamic Multimedia Interaction Systems. In E. Chew, A. Childs, and C.-H. Chuan, editors, Mathematics and Computation in Music, volume 38 of Communications in Computer and Information Science, pages 218–227. Springer Berlin Heidelberg, Berlin, Heidelberg, 2009. ISBN 978-3-642-02393-4. doi: 10.1007/978-3-642-02394-1. URL http://www.springerlink.com/index/10.1007/ 978-3-642-02394-1http: //link.springer.com/10.1007/978-3-642-02394-1.

35. References P. Sénac, P. de Saqui-Sannes, and R. Willrich. Hierarchical

    time stream petri net: A model for hypermedia systems. In Michelis and Diaz [1995], pages 451–470. ISBN 3-540-60029-9. doi: 10.1007/3-540-60029-9_54. URL http://dx.doi.org/10.1007/3-540-60029-9_54. M. Toro, M. Desainte-Catherine, and C. Rueda. Formal semantics for interactive music scores: a framework to design, specify properties and execute interactive scenarios. Journal of Mathematics and Music, 8 (1):93–112, 2014. doi: 10.1080/17459737.2013.870610. D. A. Tran, K. A. Hua, and K. Vu. Videograph: A graphical object-based model for representing and querying video data. In ER, pages 383–396, 2000. doi: 10.1007/3-540-45393-8_28. URL http://dx.doi.org/10.1007/3-540-45393-8_28.