$30 off During Our Annual Pro Sale. View Details »

Foundations for Reliable and Flexible Interactive Multimedia Scores

Foundations for Reliable and Flexible Interactive Multimedia Scores

Jaime Arias Almeida

June 25, 2015
Tweet

More Decks by Jaime Arias Almeida

Other Decks in Research

Transcript

  1. Foundations for Reliable and Flexible Interactive
    Multimedia Scores
    Jaime Arias, Myriam Desainte-Catherine, Carlos Olarte, and Camilo
    Rueda
    Laboratoire Bordelais de Recherche en Informatique (LaBRI)
    Université de Bordeaux
    5th International Conference on Mathematics and Computation in
    Music (MCM 2015)
    United Kingdom, June 2015
    1

    View Slide

  2. Motivation
    Interactive Multimedia Scenarios
    • Interactive Scores† (IS) is a formalism for composing and interpreting
    interactive multimedia scenarios.
    • Some applications of IS:
    ◦ Live performances
    ◦ Museum installations
    ◦ Plastic art installations
    †A. Allombert. Aspects Temporels d’un Système de Partitions Musicales Interactives pour la
    Composition et l’Exécution. PhD thesis, Université de Bordeaux, 2009
    Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 1/13
    1/13

    View Slide

  3. Motivation
    Interactive Multimedia Scenarios
    • Currently, IS is implemented in the software i-score†.
    • A Hierarchical Time Stream Petri Net (HTSPN) model represents
    and executes the partially ordered set of events.
    †Website: http://www.i-score.org/
    Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 2/13
    2/13

    View Slide

  4. Motivation
    Interactive Multimedia Scenarios
    • Some applications demand two features that are not supported by
    the IS model†:
    ◦ Flexible control structures such as conditionals.
    ◦ Automatic verification of scenarios in order to avoid abnormal
    behaviors.
    • The specification of some behaviors could be cumbersome due to
    the horizontal time-line of i-score.
    †T. De la Hogue, P. Baltazar, M. Desainte-Catherine, J. Chao, and C. Bossut. OSSIA : Open
    Scenario System for Interactive Applications. In Journées d’Informatique Musicale, pages 78–84,
    Bourges, 2014
    Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 3/13
    3/13

    View Slide

  5. This talk is about …
    ReactiveIS: a programming language for the specification and execution
    of interactive multimedia scenarios endowed with a tree-based
    operational semantics and a declarative interpretation in intuitionistic
    linear logic.
    Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 4/13
    4/13

    View Slide

  6. Outline
    ReactiveIS
    Syntax
    Program Tree
    State Tree
    Operational Semantics
    Logical Characterization
    Concluding Remarks

    View Slide

  7. Outline
    ReactiveIS
    Syntax
    Program Tree
    State Tree
    Operational Semantics
    Logical Characterization
    Concluding Remarks

    View Slide

  8. ReactiveIS
    Syntax
    ⟨score⟩ ::= ⟨structure⟩
    ⟨texture⟩ ::= texture(⟨params⟩ ⟨msg⟩ ⟨msg⟩)
    ⟨structure⟩ ::= structure(⟨params⟩ ⟨TO-list⟩)
    ⟨params⟩ ::= ⟨name⟩ ⟨condition⟩ ⟨condition⟩
    ⟨TO-event⟩ ::= start ⟨name⟩ | end ⟨name⟩
    ⟨condition⟩ ::= wait(⟨TO-event⟩ ⟨min⟩ ⟨max⟩)
    | event ⟨msg⟩
    | (⟨condition⟩ ∧ ⟨condition⟩)
    | (⟨condition⟩ ∨ ⟨condition⟩)
    time (s)
    0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
    Texture B
    Texture A
    Structure C
    Structure D
    Texture E
    Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 5/13
    5/13

    View Slide

  9. ReactiveIS
    Syntax
    ...
    Structure C = {
    start.c = (Wait(End(A),5,5) &
    Wait(End(B),3,3));
    stop.c = (Wait(End(D),0,INF) &
    Wait(End(E),0,INF));
    Texture D = {
    start.c = ((Wait(Start(C),2,5) &
    Event("/mouse 1")) |
    Wait(Start(C),5,5));
    stop.c = Wait(Start(D),1,1);
    start.msg = "/sound/1 on";
    stop.msg = "/sound/1 off";
    };
    ...
    };
    time (s)
    0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
    Texture B
    Texture A
    Structure C
    Structure D
    Texture E
    Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 6/13
    6/13

    View Slide

  10. ReactiveIS
    Program Tree
    start:
    stop:
    True
    EndScenario
    Conditionals
    start:
    stop:
    WaitFromEnd(A,5,5) ∧ WaitFromEnd(B,3,3)
    WaitFromEnd(C.D,0,∞) ∧ WaitFromEnd(C.E,0,∞)
    Conditionals
    C
    start:
    stop:
    WaitFromStart(C,1,1)
    WaitFromStart(C.E,2,2)
    Conditionals
    Messages
    start:
    stop:
    /sound/2 on
    /sound/2 off
    E
    B A
    D
    Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 7/13
    7/13

    View Slide

  11. ReactiveIS
    State Tree
    start:
    stop:
    0

    Times
    Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 8/13
    8/13

    View Slide

  12. ReactiveIS
    State Tree
    start:
    stop:
    0

    Times
    start:
    stop:
    1
    3
    Times
    A
    Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 8/13
    8/13

    View Slide

  13. ReactiveIS
    State Tree
    start:
    stop:
    0

    Times
    start:
    stop:
    3
    5
    Times
    B
    start:
    stop:
    1
    3
    Times
    A
    Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 8/13
    8/13

    View Slide

  14. ReactiveIS
    State Tree
    start:
    stop:
    0

    Times
    start:
    stop:
    3
    5
    Times
    B
    start:
    stop:
    1
    3
    Times
    A
    start:
    stop:
    8

    Times
    C
    start(S,C,8)
    Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 8/13
    8/13

    View Slide

  15. ReactiveIS
    State Tree
    start:
    stop:
    0

    Times
    start:
    stop:
    3
    5
    Times
    B
    start:
    stop:
    1
    3
    Times
    A
    start:
    stop:
    8
    14
    Times
    C
    stop(S,C,14)
    Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 8/13
    8/13

    View Slide

  16. ReactiveIS
    Operational Semantics
    RSTART
    p ∈ canStart(S, P) ⟨P, S, I, t⟩ |= cs(n)
    ⟨St, O⟩I,t
    S
    −→P ⟨start(St, p, t), O ∪ {ms(n)}⟩I,t
    S
    where n = targetP(p)
    RSTOP
    p ∈ canStop(S) ⟨P, S, I, t⟩ |= ce(n)
    ⟨St, O⟩I,t
    S
    −→P ⟨stop(St, p, t), O ∪ {me(n)}⟩I,t
    S
    where n = targetP(p)
    RTIME
    ⟨S, ∅⟩I,t
    S
    −→∗
    P
    ⟨S′, O⟩I,t
    S
    ̸
    −→P
    ⟨S, t⟩ I,O
    =⇒P ⟨S′, t + 1⟩
    Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 9/13
    9/13

    View Slide

  17. ReactiveIS
    Operational Semantics
    RSTART
    p ∈ canStart(S, P) ⟨P, S, I, t⟩ |= cs(n)
    ⟨St, O⟩I,t
    S
    −→P ⟨start(St, p, t), O ∪ {ms(n)}⟩I,t
    S
    where n = targetP(p)
    palive(S) = {p | p ∈ L(S) ∧ te(targetS (p)) = ⊥}
    0

    8

    3
    5
    1
    3
    C
    A
    B
    palive (S)
    Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 9/13
    9/13

    View Slide

  18. ReactiveIS
    Operational Semantics
    RSTART
    p ∈ canStart(S, P) ⟨P, S, I, t⟩ |= cs(n)
    ⟨St, O⟩I,t
    S
    −→P ⟨start(St, p, t), O ∪ {ms(n)}⟩I,t
    S
    where n = targetP(p)
    palive(S) = {p | p ∈ L(S) ∧ te(targetS (p)) = ⊥}
    canStart(P, S) △
    = {p | pparent ∈ palive(S) ∧ p ∈ Children(pparent )} \ L(S)
    C
    A
    B
    E
    D
    canStart(P, S)
    0

    8

    3
    5
    1
    3
    C
    A
    B
    palive (S)
    Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 9/13
    9/13

    View Slide

  19. ReactiveIS
    Operational Semantics
    RSTART
    p ∈ canStart(S, P) ⟨P, S, I, t⟩ |= cs(n)
    ⟨St, O⟩I,t
    S
    −→P ⟨start(St, p, t), O ∪ {ms(n)}⟩I,t
    S
    where n = targetP(p)
    cs (targetP (C.E)) = WaitFromStart(C, 1, 1)
    ⟨P, S, I, t⟩ |
    = WaitFromStart(p, t1, t2) iff
    ∃n · n ∈ V (S) ∧ n = targetS (p) ∧ t1 ≤ t − ts (n) ≤ t2
    0

    8

    3
    5
    1
    3
    C
    A
    B
    t = 9
    Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 9/13
    9/13

    View Slide

  20. ReactiveIS
    Operational Semantics
    RSTOP
    p ∈ canStop(S) ⟨P, S, I, t⟩ |= ce(n)
    ⟨St, O⟩I,t
    S
    −→P ⟨stop(St, p, t), O ∪ {me(n)}⟩I,t
    S
    where n = targetP(p)
    canStop(S) △
    = {p | p ∈ L(S) ∧ te(targetS (p)) = ⊥} = palive(S)
    Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 9/13
    9/13

    View Slide

  21. ReactiveIS
    Operational Semantics
    RSTART
    p ∈ canStart(S, P) ⟨P, S, I, t⟩ |= cs(n)
    ⟨St, O⟩I,t
    S
    −→P ⟨start(St, p, t), O ∪ {ms(n)}⟩I,t
    S
    where n = targetP(p)
    RSTOP
    p ∈ canStop(S) ⟨P, S, I, t⟩ |= ce(n)
    ⟨St, O⟩I,t
    S
    −→P ⟨stop(St, p, t), O ∪ {me(n)}⟩I,t
    S
    where n = targetP(p)
    RTIME
    ⟨S, ∅⟩I,t
    S
    −→∗
    P
    ⟨S′, O⟩I,t
    S
    ̸
    −→P
    ⟨S, t⟩ I,O
    =⇒P ⟨S′, t + 1⟩
    Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 9/13
    9/13

    View Slide

  22. Outline
    ReactiveIS
    Syntax
    Program Tree
    State Tree
    Operational Semantics
    Logical Characterization
    Concluding Remarks

    View Slide

  23. Logical Characterization
    Subexponentials in Linear Logic (SELL)
    Girard’s Linear Logic†: Formulas are seen as resources, e.g, c ⊗ c ̸−→ c.
    Subexponentials: meanings for !aF
    • “F holds in location a” (e.g., in a given structure)
    • “F holds with certain modality a” (e.g., in a time-unit t).
    • Subexp. are ordered in a poset.
    • The ordering determines the
    provability relation, e.g.,:
    !aF −→!bF iff b ⪯ a
    • We then have a better control
    of resources in proofs.
    0+
    0
    1+
    1
    2+
    2
    3+
    3

    where t is defined as
    t
    TI
    t.i t.o
    t.s.scenario
    t.s.A t.s.B t.s.C
    t.s.D t.s.E
    t.p
    TP
    †J. Girard. Linear logic. Theor. Comput. Sci., 50:1–102, 1987
    Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 10/13
    10/13

    View Slide

  24. Logical Characterization
    Declarative Meaning of ReactiveIS programs
    We defined an encoding from ReactiveIS programs into SELL formulas.
    Adequacy
    P reaches a state s iff the sequent [[P]] −→ [[s]] is provable in SELL.
    This opens the possibility of reasoning about IS by using well established
    techniques in proof theory!
    We can verify properties such as:
    • Is it possible for the structure A and B to be played concurrently ?
    • Is the execution of A always preceded by an execution of B ?
    • Will all the structures be eventually played ?
    Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 11/13
    11/13

    View Slide

  25. Outline
    ReactiveIS
    Syntax
    Program Tree
    State Tree
    Operational Semantics
    Logical Characterization
    Concluding Remarks

    View Slide

  26. Concluding Remarks
    Summary
    • We presented a new programming language for the specification,
    verification and interpretation of interactive scores.
    • ReactiveIS extends the full capacity of temporal organization of IS and
    also it allows the specification of conditionals.
    • The operational semantics based on labelled trees are simpler and
    more intuitive than the current model in HTSPN of i-score.
    • We presented a declarative interpretation of ReactiveIS programs as
    formulas in ILL with subexponentials that is adequate.
    Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 12/13
    12/13

    View Slide

  27. Concluding Remarks
    Future work
    • Coq for proving the correcteness of the semantics.
    • Front-end for the specification of properties.
    • Extension for loops.
    Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 13/13
    13/13

    View Slide

  28. Thank you for your attention!
    [email protected]

    View Slide

  29. Foundations for Reliable and Flexible Interactive
    Multimedia Scores
    Jaime Arias, Myriam Desainte-Catherine, Carlos Olarte, and Camilo
    Rueda
    Laboratoire Bordelais de Recherche en Informatique (LaBRI)
    Université de Bordeaux
    5th International Conference on Mathematics and Computation in
    Music (MCM 2015)
    United Kingdom, June 2015
    1

    View Slide

  30. References
    CONCUR 2013 - Concurrency Theory - 24th International Conference,
    CONCUR 2013, Buenos Aires, Argentina, August 27-30, 2013.
    Proceedings, volume 8052 of Lecture Notes in Computer Science, 2013.
    Springer. ISBN 978-3-642-40183-1.
    A. Allombert. Aspects Temporels d’un Système de Partitions Musicales
    Interactives pour la Composition et l’Exécution. PhD thesis, Université de
    Bordeaux, 2009.
    A. Allombert, R. Marczak, M. Desainte-Catherine, P. Baltazar, and
    GarnierLaurent. Virage : Designing An Interactive Intermedia
    Sequencer From Users Requirements And Theoretical Background.
    In International Computer Music Conference, 2010.
    J.-M. Andreoli. Logic programming with focusing proofs in linear logic. J.
    Log. Comput., 2(3):297–347, 1992.

    View Slide

  31. References
    M. Andries, G. Engels, A. Habel, B. Hoffmann, H. Kreowski, S. Kuske,
    D. Plump, A. Schürr, and G. Taentzer. Graph transformation for
    specification and programming. Sci. Comput. Program., 34(1):1–54,
    1999. doi: 10.1016/S0167-6423(98)00023-9. URL
    http://dx.doi.org/10.1016/S0167-6423(98)00023-9.
    J. Arias, M. Desainte-Catherine, C. Olarte, and C. Rueda. Foundations
    for reliable and flexible interactive multimedia scores. Technical
    report, Labri, University of Bordeaux, Mar. 2015.
    V. Danos, J. Joinet, and H. Schellinx. The structure of exponentials:
    Uncovering the dynamics of linear logic proofs. In Gottlob et al.
    [1993], pages 159–171. ISBN 3-540-57184-1. doi:
    10.1007/BFb0022564. URL
    http://dx.doi.org/10.1007/BFb0022564.

    View Slide

  32. References
    T. De la Hogue, P. Baltazar, M. Desainte-Catherine, J. Chao, and
    C. Bossut. OSSIA : Open Scenario System for Interactive
    Applications. In Journées d’Informatique Musicale, pages 78–84,
    Bourges, 2014.
    M. Desainte-Catherine, A. Allombert, and G. Assayag. Towards a hybrid
    temporal paradigm for musical composition and performance: The
    case of musical interpretation. Computer Music Journal, 37(2):61–72,
    2013. doi: 10.1162/COMJ_a_00179. URL
    http://dx.doi.org/10.1162/COMJ_a_00179.
    J. Girard. Linear logic. Theor. Comput. Sci., 50:1–102, 1987.
    G. Gottlob, A. Leitsch, and D. Mundici, editors. Computational Logic and
    Proof Theory, Third Kurt Gödel Colloquium, KGC’93, Brno, Czech Republic,
    August 24-27, 1993, Proceedings, volume 713 of Lecture Notes in
    Computer Science, 1993. Springer. ISBN 3-540-57184-1.

    View Slide

  33. References
    T. Hoare, G. Menzel, and J. Misra. A tree semantics of an orchestration
    language. In Engineering Theories of Software Intensive Systems, volume
    195 of NATO Science Series II: Mathematics, Physics and Chemistry, pages
    331–350. Springer, 2005. ISBN 978-1-4020-3530-2. doi:
    10.1007/1-4020-3532-2_11.
    R. Marczak, M. Desainte-Catherine, and A. Allombert. Real-time
    temporal control of musical processes. In The Third International
    Conferences on Advances in Multimedia, MMEDIA 2011, pages 12–17,
    2011. ISBN 978-1-61208-129-8.
    G. D. Michelis and M. Diaz, editors. Application and Theory of Petri Nets
    1995, 16th International Conference, Turin, Italy, June 26-30, 1995,
    Proceedings, volume 935 of Lecture Notes in Computer Science, 1995.
    Springer. ISBN 3-540-60029-9.

    View Slide

  34. References
    V. Nigam, C. Olarte, and E. Pimentel. A general proof system for
    modalities in concurrent constraint programming. In CONCUR DBL
    [2013], pages 410–424. ISBN 978-3-642-40183-1.
    C. Olarte and C. Rueda. A Declarative Language for Dynamic
    Multimedia Interaction Systems. In E. Chew, A. Childs, and C.-H.
    Chuan, editors, Mathematics and Computation in Music, volume 38 of
    Communications in Computer and Information Science, pages 218–227.
    Springer Berlin Heidelberg, Berlin, Heidelberg, 2009. ISBN
    978-3-642-02393-4. doi: 10.1007/978-3-642-02394-1. URL
    http://www.springerlink.com/index/10.1007/
    978-3-642-02394-1http:
    //link.springer.com/10.1007/978-3-642-02394-1.

    View Slide

  35. References
    P. Sénac, P. de Saqui-Sannes, and R. Willrich. Hierarchical time stream
    petri net: A model for hypermedia systems. In Michelis and Diaz
    [1995], pages 451–470. ISBN 3-540-60029-9. doi:
    10.1007/3-540-60029-9_54. URL
    http://dx.doi.org/10.1007/3-540-60029-9_54.
    M. Toro, M. Desainte-Catherine, and C. Rueda. Formal semantics for
    interactive music scores: a framework to design, specify properties
    and execute interactive scenarios. Journal of Mathematics and Music, 8
    (1):93–112, 2014. doi: 10.1080/17459737.2013.870610.
    D. A. Tran, K. A. Hua, and K. Vu. Videograph: A graphical object-based
    model for representing and querying video data. In ER, pages
    383–396, 2000. doi: 10.1007/3-540-45393-8_28. URL
    http://dx.doi.org/10.1007/3-540-45393-8_28.

    View Slide