Foundations for Reliable and Flexible Interactive Multimedia Scores

Foundations for Reliable and Flexible Interactive
Multimedia Scores
Jaime Arias, Myriam Desainte-Catherine, Carlos Olarte, and Camilo
Rueda
Laboratoire Bordelais de Recherche en Informatique (LaBRI)
Université de Bordeaux
5th International Conference on Mathematics and Computation in
Music (MCM 2015)
United Kingdom, June 2015
1

View Slide

Motivation
Interactive Multimedia Scenarios
• Interactive Scores† (IS) is a formalism for composing and interpreting
interactive multimedia scenarios.
• Some applications of IS:
◦ Live performances
◦ Museum installations
◦ Plastic art installations
†A. Allombert. Aspects Temporels d’un Système de Partitions Musicales Interactives pour la
Composition et l’Exécution. PhD thesis, Université de Bordeaux, 2009
Arias, Desainte-Catherine, Olarte, and Rueda (LaBRI) Foundations for Reliable and Flexible Interactive Multimedia Scores 1/13
1/13

View Slide

Motivation
• Currently, IS is implemented in the software i-score†.
• A Hierarchical Time Stream Petri Net (HTSPN) model represents
and executes the partially ordered set of events.
†Website: http://www.i-score.org/
2/13

View Slide

Motivation
• Some applications demand two features that are not supported by
the IS model†:
◦ Flexible control structures such as conditionals.
◦ Automatic verification of scenarios in order to avoid abnormal
behaviors.
• The specification of some behaviors could be cumbersome due to
the horizontal time-line of i-score.
†T. De la Hogue, P. Baltazar, M. Desainte-Catherine, J. Chao, and C. Bossut. OSSIA : Open
Scenario System for Interactive Applications. In Journées d’Informatique Musicale, pages 78–84,
Bourges, 2014
3/13

View Slide

This talk is about …
ReactiveIS: a programming language for the specification and execution
of interactive multimedia scenarios endowed with a tree-based
operational semantics and a declarative interpretation in intuitionistic
linear logic.
4/13

View Slide

Outline
ReactiveIS
Syntax
Program Tree
State Tree
Operational Semantics
Logical Characterization
Concluding Remarks

View Slide

ReactiveIS
Syntax
⟨score⟩ ::= ⟨structure⟩
⟨texture⟩ ::= texture(⟨params⟩ ⟨msg⟩ ⟨msg⟩)
⟨structure⟩ ::= structure(⟨params⟩ ⟨TO-list⟩)
⟨params⟩ ::= ⟨name⟩ ⟨condition⟩ ⟨condition⟩
⟨TO-event⟩ ::= start ⟨name⟩ | end ⟨name⟩
⟨condition⟩ ::= wait(⟨TO-event⟩ ⟨min⟩ ⟨max⟩)
| event ⟨msg⟩
| (⟨condition⟩ ∧ ⟨condition⟩)
| (⟨condition⟩ ∨ ⟨condition⟩)
time (s)
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Texture B
Texture A
Structure C
Structure D
Texture E
5/13

View Slide

ReactiveIS
Syntax
...
Structure C = {
start.c = (Wait(End(A),5,5) &
Wait(End(B),3,3));
stop.c = (Wait(End(D),0,INF) &
Wait(End(E),0,INF));
Texture D = {
start.c = ((Wait(Start(C),2,5) &
Event("/mouse 1")) |
Wait(Start(C),5,5));
stop.c = Wait(Start(D),1,1);
start.msg = "/sound/1 on";
stop.msg = "/sound/1 off";
};
...
};
time (s)
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Texture B
Texture A
Structure C
Structure D
Texture E
6/13

View Slide

ReactiveIS
Program Tree
start:
stop:
True
EndScenario
Conditionals
start:
stop:
WaitFromEnd(A,5,5) ∧ WaitFromEnd(B,3,3)
WaitFromEnd(C.D,0,∞) ∧ WaitFromEnd(C.E,0,∞)
Conditionals
C
start:
stop:
WaitFromStart(C,1,1)
WaitFromStart(C.E,2,2)
Conditionals
Messages
start:
stop:
/sound/2 on
/sound/2 off
E
B A
D
7/13

View Slide

ReactiveIS
State Tree
start:
stop:
0
⊥
Times
8/13

View Slide

ReactiveIS
State Tree
start:
stop:
0
⊥
Times
start:
stop:
1
3
Times
A
8/13

View Slide

ReactiveIS
State Tree
start:
stop:
0
⊥
Times
start:
stop:
3
5
Times
B
start:
stop:
1
3
Times
A
8/13

View Slide

ReactiveIS
State Tree
start:
stop:
0
⊥
Times
start:
stop:
3
5
Times
B
start:
stop:
1
3
Times
A
start:
stop:
8
⊥
Times
C
start(S,C,8)
8/13

View Slide

ReactiveIS
State Tree
start:
stop:
0
⊥
Times
start:
stop:
3
5
Times
B
start:
stop:
1
3
Times
A
start:
stop:
8
14
Times
C
stop(S,C,14)
8/13

View Slide

ReactiveIS
RSTART
p ∈ canStart(S, P) ⟨P, S, I, t⟩ |= cs(n)
⟨St, O⟩I,t
S
−→P ⟨start(St, p, t), O ∪ {ms(n)}⟩I,t
S
where n = targetP(p)
RSTOP
p ∈ canStop(S) ⟨P, S, I, t⟩ |= ce(n)
⟨St, O⟩I,t
S
−→P ⟨stop(St, p, t), O ∪ {me(n)}⟩I,t
S
RTIME
⟨S, ∅⟩I,t
S
−→∗
P
⟨S′, O⟩I,t
S
̸
−→P
⟨S, t⟩ I,O
=⇒P ⟨S′, t + 1⟩
9/13

View Slide

ReactiveIS
RSTART
⟨St, O⟩I,t
S
S
palive(S) = {p | p ∈ L(S) ∧ te(targetS (p)) = ⊥}
0
⊥
8
⊥
3
5
1
3
C
A
B
palive (S)
9/13

View Slide

ReactiveIS
RSTART
⟨St, O⟩I,t
S
S
palive(S) = {p | p ∈ L(S) ∧ te(targetS (p)) = ⊥}
canStart(P, S) △
= {p | pparent ∈ palive(S) ∧ p ∈ Children(pparent )} \ L(S)
C
A
B
E
D
canStart(P, S)
0
⊥
8
⊥
3
5
1
3
C
A
B
palive (S)
9/13

View Slide

ReactiveIS
RSTART
⟨St, O⟩I,t
S
S
cs (targetP (C.E)) = WaitFromStart(C, 1, 1)
⟨P, S, I, t⟩ |
= WaitFromStart(p, t1, t2) iff
∃n · n ∈ V (S) ∧ n = targetS (p) ∧ t1 ≤ t − ts (n) ≤ t2
0
⊥
8
⊥
3
5
1
3
C
A
B
t = 9
9/13

View Slide

ReactiveIS
RSTOP
⟨St, O⟩I,t
S
S
canStop(S) △
= {p | p ∈ L(S) ∧ te(targetS (p)) = ⊥} = palive(S)
9/13

View Slide

ReactiveIS
RSTART
⟨St, O⟩I,t
S
S
RSTOP
⟨St, O⟩I,t
S
S
RTIME
⟨S, ∅⟩I,t
S
−→∗
P
⟨S′, O⟩I,t
S
̸
−→P
⟨S, t⟩ I,O
=⇒P ⟨S′, t + 1⟩
9/13

View Slide

Outline
ReactiveIS
Syntax
Program Tree
State Tree
Concluding Remarks

View Slide

Subexponentials in Linear Logic (SELL)
Girard’s Linear Logic†: Formulas are seen as resources, e.g, c ⊗ c ̸−→ c.
Subexponentials: meanings for !aF
• “F holds in location a” (e.g., in a given structure)
• “F holds with certain modality a” (e.g., in a time-unit t).
• Subexp. are ordered in a poset.
• The ordering determines the
provability relation, e.g.,:
!aF −→!bF iff b ⪯ a
• We then have a better control
of resources in proofs.
0+
0
1+
1
2+
2
3+
3
…
where t is deﬁned as
t
TI
t.i t.o
t.s.scenario
t.s.A t.s.B t.s.C
t.s.D t.s.E
t.p
TP
†J. Girard. Linear logic. Theor. Comput. Sci., 50:1–102, 1987
10/13

View Slide

Declarative Meaning of ReactiveIS programs
We defined an encoding from ReactiveIS programs into SELL formulas.
Adequacy
P reaches a state s iff the sequent [[P]] −→ [[s]] is provable in SELL.
This opens the possibility of reasoning about IS by using well established
techniques in proof theory!
We can verify properties such as:
• Is it possible for the structure A and B to be played concurrently ?
• Is the execution of A always preceded by an execution of B ?
• Will all the structures be eventually played ?
11/13

View Slide

Outline
ReactiveIS
Syntax
Program Tree
State Tree
Concluding Remarks

View Slide

Concluding Remarks
Summary
• We presented a new programming language for the specification,
verification and interpretation of interactive scores.
• ReactiveIS extends the full capacity of temporal organization of IS and
also it allows the specification of conditionals.
• The operational semantics based on labelled trees are simpler and
more intuitive than the current model in HTSPN of i-score.
• We presented a declarative interpretation of ReactiveIS programs as
formulas in ILL with subexponentials that is adequate.
12/13

View Slide

Concluding Remarks
Future work
• Coq for proving the correcteness of the semantics.
• Front-end for the specification of properties.
• Extension for loops.
13/13

View Slide

Thank you for your attention!
[email protected]

View Slide

Foundations for Reliable and Flexible Interactive
Multimedia Scores
Jaime Arias, Myriam Desainte-Catherine, Carlos Olarte, and Camilo
Rueda
Laboratoire Bordelais de Recherche en Informatique (LaBRI)
Université de Bordeaux
5th International Conference on Mathematics and Computation in
Music (MCM 2015)
United Kingdom, June 2015
1

View Slide

References
CONCUR 2013 - Concurrency Theory - 24th International Conference,
CONCUR 2013, Buenos Aires, Argentina, August 27-30, 2013.
Proceedings, volume 8052 of Lecture Notes in Computer Science, 2013.
Springer. ISBN 978-3-642-40183-1.
A. Allombert. Aspects Temporels d’un Système de Partitions Musicales
Interactives pour la Composition et l’Exécution. PhD thesis, Université de
Bordeaux, 2009.
A. Allombert, R. Marczak, M. Desainte-Catherine, P. Baltazar, and
GarnierLaurent. Virage : Designing An Interactive Intermedia
Sequencer From Users Requirements And Theoretical Background.
In International Computer Music Conference, 2010.
J.-M. Andreoli. Logic programming with focusing proofs in linear logic. J.
Log. Comput., 2(3):297–347, 1992.

View Slide

References
M. Andries, G. Engels, A. Habel, B. Hoffmann, H. Kreowski, S. Kuske,
D. Plump, A. Schürr, and G. Taentzer. Graph transformation for
specification and programming. Sci. Comput. Program., 34(1):1–54,
1999. doi: 10.1016/S0167-6423(98)00023-9. URL
http://dx.doi.org/10.1016/S0167-6423(98)00023-9.
J. Arias, M. Desainte-Catherine, C. Olarte, and C. Rueda. Foundations
for reliable and flexible interactive multimedia scores. Technical
report, Labri, University of Bordeaux, Mar. 2015.
V. Danos, J. Joinet, and H. Schellinx. The structure of exponentials:
Uncovering the dynamics of linear logic proofs. In Gottlob et al.
[1993], pages 159–171. ISBN 3-540-57184-1. doi:
10.1007/BFb0022564. URL
http://dx.doi.org/10.1007/BFb0022564.

View Slide

References
T. De la Hogue, P. Baltazar, M. Desainte-Catherine, J. Chao, and
C. Bossut. OSSIA : Open Scenario System for Interactive
Applications. In Journées d’Informatique Musicale, pages 78–84,
Bourges, 2014.
M. Desainte-Catherine, A. Allombert, and G. Assayag. Towards a hybrid
temporal paradigm for musical composition and performance: The
case of musical interpretation. Computer Music Journal, 37(2):61–72,
2013. doi: 10.1162/COMJ_a_00179. URL
http://dx.doi.org/10.1162/COMJ_a_00179.
J. Girard. Linear logic. Theor. Comput. Sci., 50:1–102, 1987.
G. Gottlob, A. Leitsch, and D. Mundici, editors. Computational Logic and
Proof Theory, Third Kurt Gödel Colloquium, KGC’93, Brno, Czech Republic,
August 24-27, 1993, Proceedings, volume 713 of Lecture Notes in
Computer Science, 1993. Springer. ISBN 3-540-57184-1.

View Slide

References
T. Hoare, G. Menzel, and J. Misra. A tree semantics of an orchestration
language. In Engineering Theories of Software Intensive Systems, volume
195 of NATO Science Series II: Mathematics, Physics and Chemistry, pages
331–350. Springer, 2005. ISBN 978-1-4020-3530-2. doi:
10.1007/1-4020-3532-2_11.
R. Marczak, M. Desainte-Catherine, and A. Allombert. Real-time
temporal control of musical processes. In The Third International
Conferences on Advances in Multimedia, MMEDIA 2011, pages 12–17,
2011. ISBN 978-1-61208-129-8.
G. D. Michelis and M. Diaz, editors. Application and Theory of Petri Nets
1995, 16th International Conference, Turin, Italy, June 26-30, 1995,
Proceedings, volume 935 of Lecture Notes in Computer Science, 1995.
Springer. ISBN 3-540-60029-9.

View Slide

References
V. Nigam, C. Olarte, and E. Pimentel. A general proof system for
modalities in concurrent constraint programming. In CONCUR DBL
[2013], pages 410–424. ISBN 978-3-642-40183-1.
C. Olarte and C. Rueda. A Declarative Language for Dynamic
Multimedia Interaction Systems. In E. Chew, A. Childs, and C.-H.
Chuan, editors, Mathematics and Computation in Music, volume 38 of
Communications in Computer and Information Science, pages 218–227.
Springer Berlin Heidelberg, Berlin, Heidelberg, 2009. ISBN
978-3-642-02393-4. doi: 10.1007/978-3-642-02394-1. URL
http://www.springerlink.com/index/10.1007/
978-3-642-02394-1http:
//link.springer.com/10.1007/978-3-642-02394-1.

View Slide

References
P. Sénac, P. de Saqui-Sannes, and R. Willrich. Hierarchical time stream
petri net: A model for hypermedia systems. In Michelis and Diaz
[1995], pages 451–470. ISBN 3-540-60029-9. doi:
10.1007/3-540-60029-9_54. URL
http://dx.doi.org/10.1007/3-540-60029-9_54.
M. Toro, M. Desainte-Catherine, and C. Rueda. Formal semantics for
interactive music scores: a framework to design, specify properties
and execute interactive scenarios. Journal of Mathematics and Music, 8
(1):93–112, 2014. doi: 10.1080/17459737.2013.870610.
D. A. Tran, K. A. Hua, and K. Vu. Videograph: A graphical object-based
model for representing and querying video data. In ER, pages
383–396, 2000. doi: 10.1007/3-540-45393-8_28. URL
http://dx.doi.org/10.1007/3-540-45393-8_28.

View Slide

Foundations for Reliable and Flexible Interactive Multimedia Scores

Foundations for Reliable and Flexible Interactive Multimedia Scores

Jaime Arias Almeida

More Decks by Jaime Arias Almeida

Other Decks in Research

Featured

Transcript