Towards a Rewriting Logic Semantics for Interactive Scores Jaime Arias, Jean-Michaël Celerier, Myriam Desainte-Catherine, Carlos Olarte, Bernard P. Serpette Journées d’Informatique Musicale (JIM 2024) May 7th, 2024 

Interactive Scores Interactive multimedia = multimedia content + interaction with the environment + spatial and temporal organization (a) Live performances (b) Museum installations (c) Plastic art installations Figure: Some examples of interactive multimedia systems. Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 2/32 

Interactive Scores The potential high complexity of these systems requires adequate specification languages for their full description and computer-assisted verification Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 3/32 

Interactive Scores • Interactive Scores (IS) is a formalism for composing and executing interactive multimedia scenarios. • In IS, the environment has the possibility to modify, during the interpretation of the score, its temporal organization. Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 4/32 

Interactive Scores • Interactive Scores (IS) is a formalism for composing and executing interactive multimedia scenarios. • In IS, the environment has the possibility to modify, during the interpretation of the score, its temporal organization. • A score consists of: Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 4/32 

Interactive Scores • Interactive Scores (IS) is a formalism for composing and executing interactive multimedia scenarios. • In IS, the environment has the possibility to modify, during the interpretation of the score, its temporal organization. • A score consists of: ◦ Temporal Objects (TOs): textures (multimedia processes) and structures (hierarchy). Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 4/32 

Interactive Scores • Interactive Scores (IS) is a formalism for composing and executing interactive multimedia scenarios. • In IS, the environment has the possibility to modify, during the interpretation of the score, its temporal organization. • A score consists of: ◦ Temporal Objects (TOs): textures (multimedia processes) and structures (hierarchy). ◦ Temporal Relations: temporal (i.e. quantitative) and logical (i.e. qualitative) relations between TOs. Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 4/32 

Interactive Scores • Interactive Scores (IS) is a formalism for composing and executing interactive multimedia scenarios. • In IS, the environment has the possibility to modify, during the interpretation of the score, its temporal organization. • A score consists of: ◦ Temporal Objects (TOs): textures (multimedia processes) and structures (hierarchy). ◦ Temporal Relations: temporal (i.e. quantitative) and logical (i.e. qualitative) relations between TOs. ◦ Interaction Points: allow to change the start and stop times of TOs during execution. Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 4/32 

Interactive Scores Figure: ossia score – https://ossia.io/ Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 5/32 

Interactive Scores • Several models have been proposed to give formal semantics: ◦ Petri nets: Desainte-Catherine et al. (2013) ◦ Process Calculi: Olarte and Rueda (2009); Toro et al. (2014) ◦ Reactive Programming: Arias et al. (2014) ◦ Linear Logic: Arias et al. (2015) ◦ Timed automata: Arias et al. (2016) Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 6/32 

Interactive Scores • Several models have been proposed to give formal semantics: ◦ Petri nets: Desainte-Catherine et al. (2013) ◦ Process Calculi: Olarte and Rueda (2009); Toro et al. (2014) ◦ Reactive Programming: Arias et al. (2014) ◦ Linear Logic: Arias et al. (2015) ◦ Timed automata: Arias et al. (2016) However Õ: • They cannot be straightforwardly implemented or extended. • There are no practical solutions for the automatic verification. Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 6/32 

ReactiveIS ReactiveIS is a declarative programming language for the specification and interpretation of IS. Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 7/32 

ReactiveIS ReactiveIS is a declarative programming language for the specification and interpretation of IS. # hierarchical structure structure : g { name : "Group"; start-when : (wait(start(scenario),0,0) /\ wait(ev("/keyboard f"))); stop-when : wait(stop(lb), 0, INF); content : [ # light beam texture texture : lb { name : "Light Beam" ; start-when : wait(start(g),0, 1000) ; stop-when : wait(start(lb), 1184, 1184) ; msg-start : "/lightBeam on" ; msg-stop : "/lightBeam off" ; } ] ; } Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 7/32 

ReactiveIS Some Features: • More flexible temporal organization of TOs. • Support control structures such as conditionals. • Tree-based operational semantics. • Logical semantics based on Intuitionistic Linear Logic (ILL). Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 8/32 

ReactiveIS Some Features: • More flexible temporal organization of TOs. • Support control structures such as conditionals. • Tree-based operational semantics. • Logical semantics based on Intuitionistic Linear Logic (ILL). Some Drawbacks: • Artists would need to have knowledge of ILL to prove properties of their score. • Maybe new constructors’ behaviour may be captured in one semantic but not in the other. Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 8/32 

Our Goal Define an easily extensible executable semantics for the specification, interpretation and automatic verification of interactive scores written in ReactiveIS. Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 9/32 

Our Goal Define an easily extensible executable semantics for the specification, interpretation and automatic verification of interactive scores written in ReactiveIS. Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 9/32 

Table of contents 1. Rewriting Logic 2. A Rewrite Theory for ReactiveIS 3. Analysis of ReactiveIS Programs 4. Conclusions Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 10/32 

1. Rewriting Logic Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 11/32 

Rewriting Logic • Rewriting logic (RL) is a powerful framework that can express both concurrent computation and logical deduction. • Features: ◦ Very expressive and general ◦ User-defined datatypes ◦ Large applications (e.g. biology, security, etc) ◦ Executable specification ◦ Maude system: full LTL model checking, reachability, … Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 12/32 

Rewriting Logic In RL, the specification unit is a rewrite theory: R = equational theory Σ , E ⊎ B , R signature axioms equations rewrite rules Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 13/32 

Rewriting Logic In RL, the specification unit is a rewrite theory: R = equational theory Σ , E ⊎ B , R signature axioms equations rewrite rules • The equational theory specifies the static behaviour. • The rewrite rules model the dynamic behaviour. Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 13/32 

Rewriting Logic sorts Note Scale . subsorts Note < Scale . --- Notes op E : -> Note [ctor] . op F : -> Note [ctor] . --- ... --- compute next half tone op half : Note -> Note . eq half(E) = F . eq half(F) = Gb . --- ... --- Compute next whole tone var N : Note . op whole : Note -> Note . eq whole(N) = half(half(N)) . Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 14/32 

Rewriting Logic sorts Note Scale . subsorts Note < Scale . --- Notes op E : -> Note [ctor] . op F : -> Note [ctor] . --- ... --- compute next half tone op half : Note -> Note . eq half(E) = F . eq half(F) = Gb . --- ... --- Compute next whole tone var N : Note . op whole : Note -> Note . eq whole(N) = half(half(N)) . Maude> red whole(D) . reduce in SCALE : whole(D) . rewrites: 3 in 0ms cpu (0ms real) (230769 rewrites/second) result Note: E Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 14/32 

Rewriting Logic --- scale (note concatenation) var S : Scale . op _;_ : Scale Scale -> Scale [ctor] . --- Major scale : whole ; whole ; half ; whole ; whole ; whole ; half crl [major-whole] : S => S ; whole(tail(S)) if steps(S) =/= 2 /\ steps(S) =/= 6 . crl [major-half] : S => S ; half(tail(S)) if steps(S) = 2 . crl [major-half] : S => S ; half(tail(S)) if steps(S) = 6 . Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 15/32 

Rewriting Logic --- scale (note concatenation) var S : Scale . op _;_ : Scale Scale -> Scale [ctor] . --- Major scale : whole ; whole ; half ; whole ; whole ; whole ; half crl [major-whole] : S => S ; whole(tail(S)) if steps(S) =/= 2 /\ steps(S) =/= 6 . crl [major-half] : S => S ; half(tail(S)) if steps(S) = 2 . crl [major-half] : S => S ; half(tail(S)) if steps(S) = 6 . Maude> rew [7] C . rewrite [7] in SCALE : C . rewrites: 102 in 0ms cpu (0ms real) (4857142 rewrites/second) result Scale: ((((((C ; D) ; E) ; F) ; G) ; A) ; B) ; C Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 15/32 

2. A Rewrite Theory for ReactiveIS Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 16/32 

A Rewrite Theory for ReactiveIS • RL allows specifying systems using an OO-like notation. • Entities are modelled as record-like structures (sort Object): O : C | a1 : v1 , . . . , an : vn object identifier (Oid) class identifier (Cid) attributes (Attribute) values Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 17/32 

A Rewrite Theory for ReactiveIS • RL allows specifying systems using an OO-like notation. • Entities are modelled as record-like structures (sort Object): O : C | a1 : v1 , . . . , an : vn object identifier (Oid) class identifier (Cid) attributes (Attribute) values --- C Major triad op Cmajor : -> Object [ctor]. eq Cmajor = < cmajor : Chord | root: C, third: E, fifth: G > . Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 17/32 

A Rewrite Theory for ReactiveIS Syntax • Thanks to the almost zero representation distance in RL, our specification follows very closely the syntax of ReactiveIS. --- attributes for textures op start:_ : Condition -> Attribute . --- start condition op stop:_ : Condition -> Attribute . --- stop condition Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 18/32 

A Rewrite Theory for ReactiveIS Syntax • Thanks to the almost zero representation distance in RL, our specification follows very closely the syntax of ReactiveIS. --- attributes for textures op start:_ : Condition -> Attribute . --- start condition op stop:_ : Condition -> Attribute . --- stop condition op mstart:_ : Msg -> Attribute . --- start message op mstop:_ : Msg -> Attribute . --- stop message Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 18/32 

A Rewrite Theory for ReactiveIS Syntax • Thanks to the almost zero representation distance in RL, our specification follows very closely the syntax of ReactiveIS. --- attributes for textures op start:_ : Condition -> Attribute . --- start condition op stop:_ : Condition -> Attribute . --- stop condition op mstart:_ : Msg -> Attribute . --- start message op mstop:_ : Msg -> Attribute . --- stop message --- infinity for intervals op inf : -> Nat . Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 18/32 

A Rewrite Theory for ReactiveIS Syntax • Thanks to the almost zero representation distance in RL, our specification follows very closely the syntax of ReactiveIS. --- attributes for textures op start:_ : Condition -> Attribute . --- start condition op stop:_ : Condition -> Attribute . --- stop condition op mstart:_ : Msg -> Attribute . --- start message op mstop:_ : Msg -> Attribute . --- stop message --- infinity for intervals op inf : -> Nat . --- waiting conditions: event, lower, and upper bound op wait : WaitCondition Nat Nat -> Condition . Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 18/32 

A Rewrite Theory for ReactiveIS Syntax • Thanks to the almost zero representation distance in RL, our specification follows very closely the syntax of ReactiveIS. --- attributes for textures op start:_ : Condition -> Attribute . --- start condition op stop:_ : Condition -> Attribute . --- stop condition op mstart:_ : Msg -> Attribute . --- start message op mstop:_ : Msg -> Attribute . --- stop message --- infinity for intervals op inf : -> Nat . --- waiting conditions: event, lower, and upper bound op wait : WaitCondition Nat Nat -> Condition . --- external/internal events op event : String -> WaitCondition . --- external event Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 18/32 

A Rewrite Theory for ReactiveIS Syntax • Thanks to the almost zero representation distance in RL, our specification follows very closely the syntax of ReactiveIS. --- attributes for textures op start:_ : Condition -> Attribute . --- start condition op stop:_ : Condition -> Attribute . --- stop condition op mstart:_ : Msg -> Attribute . --- start message op mstop:_ : Msg -> Attribute . --- stop message --- infinity for intervals op inf : -> Nat . --- waiting conditions: event, lower, and upper bound op wait : WaitCondition Nat Nat -> Condition . --- external/internal events op event : String -> WaitCondition . --- external event op start : Oid -> WaitCondition . --- TO started op stop : Oid -> WaitCondition . --- TO stopped Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 18/32 

A Rewrite Theory for ReactiveIS Syntax --- Identifiers for temporal objects op TO : -> Cid [ctor] . Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 19/32 

A Rewrite Theory for ReactiveIS Syntax --- Identifiers for temporal objects op TO : -> Cid [ctor] . --- Variables vars O1 : Oid . vars C1 C2 : Condition . vars M1 M2 : Msg . Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 19/32 

A Rewrite Theory for ReactiveIS Syntax --- Identifiers for temporal objects op TO : -> Cid [ctor] . --- Variables vars O1 : Oid . vars C1 C2 : Condition . vars M1 M2 : Msg . --- Making TOs op make-texture : Oid Condition Condition Msg Msg -> Object . eq make-texture(O1, C1, C2, M1, M2) = < O1 : TO | start : C1, stop : C2, mstart : M1, mstop : M2 > . Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 19/32 

A Rewrite Theory for ReactiveIS Syntax --- Identifiers for temporal objects op TO : -> Cid [ctor] . --- Variables vars O1 : Oid . vars C1 C2 : Condition . vars M1 M2 : Msg . --- Making TOs op make-texture : Oid Condition Condition Msg Msg -> Object . eq make-texture(O1, C1, C2, M1, M2) = < O1 : TO | start : C1, stop : C2, mstart : M1, mstop : M2 > . --- Light Beam texture op lightBeam : -> Object . eq lightBeam = make-texture('lb, wait(start('g),0,1000), wait(start('lb),1184,1184), msg("/lightBeam on"), msg("/lightBeam off")) . Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 19/32 

A Rewrite Theory for ReactiveIS Semantics • A state of execution of a ReactiveIS program is an object of class System. op System : -> Cid --- class ID for the system --- attributes of a program's state op prog:_ : Configuration -> Attribute . op idle:_ : TOSet -> Attribute . op run:_ : TOSet -> Attribute . op done:_ : TOSet -> Attribute . op cant-start:_ : TOSet -> Attribute . op cant-stop:_ : TOSet -> Attribute . op pmsg:_ : Configuration -> Attribute . op in:_ : Configuration -> Attribute . op out:_ : Configuration -> Attribute . op time:_ : Nat -> Attribute . Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 20/32 

A Rewrite Theory for ReactiveIS Semantics Based on the tree-based semantics of ReactiveIS, we define two possible transitions: internal and observable transition. Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 21/32 

A Rewrite Theory for ReactiveIS Semantics Based on the tree-based semantics of ReactiveIS, we define two possible transitions: internal and observable transition. Internal Transition (4 rewrite rules): • It’s instantaneous: starting and stopping of TOs. • The start and stop conditions are evaluated, and the TOs’ state change, accordingly. Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 21/32 

A Rewrite Theory for ReactiveIS Semantics crl [start] : Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 22/32 

A Rewrite Theory for ReactiveIS Semantics crl [start] : < top : System | prog: (Sys < O1 : TO | start: C1, mstart: M1, ats >), idle: (O1, SB), run: RUN, done: DONE, cant-start: NSTART, cant-stop: NSTOP, pmsg: PMSEG, in: IN, out: OUT, time: T > Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 22/32 

A Rewrite Theory for ReactiveIS Semantics crl [start] : < top : System | prog: (Sys < O1 : TO | start: C1, mstart: M1, ats >), idle: (O1, SB), run: RUN, done: DONE, cant-start: NSTART, cant-stop: NSTOP, pmsg: PMSEG, in: IN, out: OUT, time: T > => < top : System | prog: (Sys < O1 : TO | start: C1, mstart: M1, ats >), idle: (SB, children-start(ats)), run: (O1, RUN), done: DONE, cant-start: NSTART, cant-stop: NSTOP, pmsg: (PMSEG start(O1, T)), in: IN, out: (OUT M1), time: T > Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 22/32 

A Rewrite Theory for ReactiveIS Semantics crl [start] : < top : System | prog: (Sys < O1 : TO | start: C1, mstart: M1, ats >), idle: (O1, SB), run: RUN, done: DONE, cant-start: NSTART, cant-stop: NSTOP, pmsg: PMSEG, in: IN, out: OUT, time: T > => < top : System | prog: (Sys < O1 : TO | start: C1, mstart: M1, ats >), idle: (SB, children-start(ats)), run: (O1, RUN), done: DONE, cant-start: NSTART, cant-stop: NSTOP, pmsg: (PMSEG start(O1, T)), in: IN, out: (OUT M1), time: T > if { T ; IN ; PMSEG ; C1 } => { T ; IN ; PMSEG ; True } . Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 22/32 

A Rewrite Theory for ReactiveIS Semantics Observable Transition (1 rewrite rule) • It represents time elapse in the system (i.e. from T to T+1). • The interaction with the environment takes place: ◦ by emitting the start and stop messages, and ◦ listening to inputs (messages) from the environment. Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 23/32 

3. Analysis of ReactiveIS Programs Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 24/32 

Analysis of ReactiveIS Programs • Maude’s strategy language allows us to define strategies for applying the rewriting rules in a specific order. • The parser IS2Maude automatically generates the executable RL specification of a ReactiveIS program. Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 25/32 

Analysis of ReactiveIS Programs • We can “simulate” some steps of the score by rewriting: Maude> rew [5] initState . result Object: < top : System | prog: ..., idle: ('g, 'ws), run: empty, done: 'scenario, cant-start: 'f, cant-stop: empty, pmsg: (start('scenario, 0) stop('scenario, 0)), in: none, out: none, time: 1 > Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 26/32 

Analysis of ReactiveIS Programs • We can rewrite the initial term following a strategy to find out whether the whole scenario was successfully executed. Maude> srew [1] initState using one(step ; tick) * ; (match < top : System | done: ('scenario, TOSET'), ATS >) . Solution 1 result Object: < top : System | prog: ..., done: ('f, 'g, 'lb, 'scenario, 'wh, 'ws), ..., time: 7193 > Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 27/32 

4. Conclusions Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 28/32 

Conclusions Contributions: • Executable rewriting logic semantics, implemented in Maude. • Automatic reachability analysis. • Extensible framework for new and complex constructors. • IS2Maude: ReactiveIS programs into Maude (interpreter). Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 29/32 

Conclusions Contributions: • Executable rewriting logic semantics, implemented in Maude. • Automatic reachability analysis. • Extensible framework for new and complex constructors. • IS2Maude: ReactiveIS programs into Maude (interpreter). Future Work: • Enhance ReactiveIS with complex constructors,e.g. loops. • Symbolic real-time RL semantics (Maude and SMT solvers). • Extend analysis techniques to encompass full model-checking. Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 29/32 

Thank you for your attention! [email protected] Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 30/32 

References Jaime Arias, Myriam Desainte-Catherine, Sylvain Salvati, and Camilo Rueda. Executing Hierarchical Interactive Scores in ReactiveML. In Journées d’Informatique Musicale, JIM 2014, pages 25–34, 2014. Jaime Arias, Myriam Desainte-Catherine, Carlos Olarte, and Camilo Rueda. Foundations for reliable and flexible interactive multimedia scores. In MCM, volume 9110 of LNCS, pages 29–41. Springer, 2015. Jaime Arias, Jean-Michaël Celerier, and Myriam Desainte-Catherine. Authoring and automatic verification of interactive multimedia scores. Journal of New Music Research, 46(1):15–33, 2016. Myriam Desainte-Catherine, Antoine Allombert, and Gérard Assayag. Towards a hybrid temporal paradigm for musical composition and performance: The case of musical interpretation. Comput. Music. J., 37(2):61–72, 2013. Carlos Olarte and Camilo Rueda. A declarative language for dynamic multimedia interaction systems. In MCM, volume 38, pages 218–227. Springer Berlin Heidelberg, 2009. Mauricio Toro, Myriam Desainte-Catherine, and Camilo Rueda. Formal semantics for interactive music scores: a framework to design, specify properties and execute interactive scenarios. Journal of Mathematics and Music, 8(1):93–112, 2014. Arias et al. (JIM 2024) Towards a Rewriting Logic Semantics for Interactive Scores 31/32 

Towards a Rewriting Logic Semantics for Interactive Scores Jaime Arias, Jean-Michaël Celerier, Myriam Desainte-Catherine, Carlos Olarte, Bernard P. Serpette Journées d’Informatique Musicale (JIM 2024) May 7th, 2024