Exploiting Parallelism in FPGAs for the Real-Time Interpretation of Interactive Multimedia Scores

Exploiting Parallelism in FPGAs for the Real-Time Interpretation of Interactive
Multimedia Scores* Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda Laboratoire Bordelais de Recherche en Informatique (LaBRI) Université de Bordeaux Journées d’Informatique Musicale (JIM 2015) Montreal - Canada, May 2015 1 *Supported by the ANR Project OSSIA and SCRIME

Motivation Interactive Multimedia Scenarios • Interactive Scores† (IS) is a
formalism for composing and executing interactive multimedia scenarios. (a) Live-art performances. (b) Interactive museum installations. (c) Plastic art installations. Figure: Some applications of IS †Antoine Allombert. Aspects Temporels d’un Système de Partitions Musicales Interactives pour la Composition et l’Exécution. PhD thesis, Université de Bordeaux, 2009 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 1/23 1/23

Motivation Interactive Multimedia Scenarios • Currently, IS is implemented in
the software i-score†. Texture Structure Rigid Interval Semi-Flexible Interval Interaction Point OSC Messages Multimedia Processes Figure: Graphical interface of i-score †I-score website: http://i-score.org/ Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 2/23 2/23

Motivation Interactive Multimedia Scenarios • Some multimedia applications perform: ◦
real-time tasks ◦ compute-intensive tasks ◦ data-intensive tasks • Sometimes the performance of standard computers is not sufficient. • Most of the multimedia applications are executed on architectures and operating systems that do not provide low-latency and real-time performance. • I-score is implemented using threads†: non-determinism and unreliability. †Edward A. Lee. The problem with threads. Computer, 39(5):33–42, May 2006 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 3/23 3/23

Motivation Field Programmable Gate Arrays (FPGAs) • FPGAs† have risen
over the last years and at the same time their cost have been reduced. • FPGAs have already been used with success in many different industrial applications: ◦ Aerospace ◦ Automotive ◦ Medical ◦ Video and audio processing †Réseau de Portes Programmables in situ Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 4/23 4/23

Motivation Field Programmable Gate Arrays (FPGAs) • FPGAs have risen
over the last years and at the same time their cost have been reduced. • FPGAs have already been used with success in many different industrial applications†: ◦ Aerospace ◦ Automotive ◦ Medical ◦ Video and audio processing †J.J. Rodriguez-Andina, M.J. Moure, and M.D. Valdes. Features, Design Tools, and Application Domains of FPGAs. IEEE Transactions on Industrial Electronics, 54(4):1810–1823, August 2007 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 4/23 4/23

Motivation Field Programmable Gate Arrays (FPGAs) • FPGAs offer the
following benefits†: ◦ Reconfigurability ◦ High-level design ◦ Physical parallelism ◦ High-speed ◦ Reliability ◦ Re-use D Q Q Look-up Table (LUT) Flip Flop Mux Clock I0 I1 I2 I3 Out Conﬁgurable Logic Blocks (CLBs) Programmable Interconnection Network Conﬁgurable Input/Output Blocks (IOBs) †Rahul Dubey. Introduction to Embedded System Design Using Field Programmable Gate Arrays. Springer London, London, 2009 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 5/23 5/23

Motivation Field Programmable Gate Arrays (FPGAs)* *Jeff Bier and Jennifer
Eyre. BDTI study certifies high-level synthesis flows for DSP-centric FPGA design. Xcell Journal, 71:12–17, Q2 2010 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 6/23 6/23

Motivation Field Programmable Gate Arrays (FPGAs)* *Luc Langlois. Multirate digital
signal processing for high-speed data converters for high-speed data converters. Xcell Journal, 73:50–53, Q4 2010 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 7/23 7/23

This talk is about …* An approach for a true
parallel implementation on FPGAs of the execution model of interactive multimedia scenarios. *David Wessel and Matthew Wright. Problems and prospects for intimate musical control of computers. In Ivan Poupyrev, Michael J. Lyons, Sidney Fels, and Tina Blaine, editors, New Interfaces for Musical Expression, NIME-01, Proceedings, Seattle, April 1-2, 2001, pages 11–14, 2001 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 8/23 8/23

Outline An Overview of IS Hardware Implementation Example Concluding Remarks

An Overview of IS • An interactive scenario is composed
of: ◦ Temporal objects • Textures → Multimedia processes • Structures → Hierarchy ◦ Temporal relations • Synchronization → ∆min = ∆max = 0 • Rigid relations → ∆min = ∆max > 0 • flexible relations → ∆min = 0 ∧ ∆max = ∞ • Semi-flexible relations → ∆min ̸= ∆max ∧ ∆max ̸= ∞ ◦ Interaction points • The system maintains the temporal relations during execution. Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 9/23 9/23

An Overview of IS time (s) 0 1 2 3
4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Texture B Texture A Structure C Structure D Texture F Texture G Texture E Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 10/23 10/23

Hardware Implementation Building our clock • We need to generate
our clock signal: frequency divider. cycles = Tclockscore TclockFPGA Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 11/23 11/23

our clock signal: frequency divider. cycles = Tclockscore TclockFPGA = 400ns 10ns Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 11/23 11/23

our clock signal: frequency divider. cycles = Tclockscore TclockFPGA = 400ns 10ns = 20 cycles Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 11/23 11/23

our clock signal: frequency divider. cycles = Tclockscore TclockFPGA = 400ns 10ns = 20 cycles Figure: Frequency divider. Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 11/23 11/23

Hardware Implementation The main module: a temporal relation • A
temporal relation: ◦ It is defined between two points. ◦ Defines a precedence relation. ◦ Defines an interval of time. • An interaction point allows for agogic modifications. Texture A Texture B Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 12/23 12/23

temporal relation: ◦ It is defined between two points. ◦ Defines a precedence relation. ◦ Defines an interval of time. • An interaction point allows for agogic modifications. Texture A Texture B end A start B Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 12/23 12/23

temporal relation: ◦ It is defined between two points. ◦ Defines a precedence relation. ◦ Defines an interval of time. • An interaction point allows for agogic modifications. Texture A Texture B end A start B B after A Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 12/23 12/23

temporal relation: ◦ It is defined between two points. ◦ Defines a precedence relation. ◦ Defines an interval of time. • An interaction point allows for agogic modifications. Texture A Texture B end A start B B after A min max Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 12/23 12/23

temporal relation: ◦ It is defined between two points. ◦ Defines a precedence relation. ◦ Defines an interval of time. • An interaction point allows for agogic modifications. Texture A Texture B end A start B B after A min max interaction point Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 12/23 12/23

Hardware Implementation The main module: a temporal relation time (s)
0 1 2 3 4 5 6 Texture A Texture B timer 1 timer 2 – – idle new wait min. wait max. end eventstart eventmin eventext /eventend eventmax /eventend Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23 13/23

0 1 2 3 4 5 6 Texture A Texture B timer 1 timer 2 8ms 24ms idle new wait min. wait max. end eventstart eventmin eventext /eventend eventmax /eventend Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23 13/23

Hardware Implementation Textures • A texture is the same as
a temporal relation but it has an attached multimedia process. Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 14/23 14/23

Hardware Implementation Handling complex temporal properties • Composers can add
one or more temporal relations in order to define the starting time of a temporal object. Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 15/23 15/23

Hardware Implementation Handling complex temporal properties time 1 2 3
4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23

4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. wait max. end eventstart eventmin eventext /eventend eventmax /eventend – – timer 1 timer 2 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23

4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. update wait max. end eventstart eventmin eventext /eventend eventmax /eventend eventstart – – timer 1 timer 2 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23

4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. update wait max. end eventstart eventmin eventext /eventend eventmax /eventend eventstart 5ms 10ms timer 1 timer 2 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23

4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. update wait max. end eventstart eventmin eventext /eventend eventmax /eventend eventstart 3ms 8ms timer 1 timer 2 max(3, 5) = 3 min(8, 10) = 8 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23

Hardware Implementation Hierarchical organization • A structure contains other temporal
objects and defines its own temporal organization. • A multimedia scenario is a structure (root). • We can model a structure as a temporal relation. • The end of a structure defines the end of its children. Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 17/23 17/23

objects and defines its own temporal organization. • A multimedia scenario is a structure (root). • We can model a structure as a temporal relation. • The end of a structure defines the end of its children. idle new wait min. update wait max. end eventstart eventm in event ext /event end event m ax /event end eventstart Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 17/23 17/23

objects and defines its own temporal organization. • A multimedia scenario is a structure (root). • We can model a structure as a temporal relation. • The end of a structure defines the end of its children. idle new wait min. update wait max. end eventstart eventm in event ext /event end event m ax /event end eventstart kill/killed kill/killed kill/killed kill/killed kill/killed Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 17/23 17/23

Hardware Implementation Hierarchical organization • Conditions for stopping a structure:
◦ with IP: 1. its maximum duration has elapsed. 2. the interaction point has been triggered. ◦ normal: 1. the maximum duration is infinity. 2. its minimum duration has elapsed. 3. its children have stopped. Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 18/23 18/23

Hardware Implementation Hierarchical organization • Conditions for stopping a structure:
◦ with IP: 1. its maximum duration has elapsed. 2. the interaction point has been triggered. ◦ normal: 1. the maximum duration is infinity. 2. its minimum duration has elapsed. 3. its children have stopped. eventmin eventend1 eventend2 eventendn eventext Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 18/23 18/23

Hardware Implementation Code in SystemVerilog 1 ... 2 // FSM
3 always_ff @(posedge clk_fsm, posedge rst_s) begin: FSM 4 if ( rst_s ) State = idle; 5 else begin: FSM_Sequencer 6 unique case (State) 7 ... 8 wait_min: begin: wait_min_state 9 rst_counter <= 0; enable_counter <= 1; 10 if (kill_s || max_timeout) State = stop; 11 else if (min_timeout) State = wait_end; 12 else if (set_s) State = new_interval; 13 end: wait_min_state 14 endcase 15 end: FSM_Sequencer 16 end: FSM 17 18 // Output 19 always_comb begin 20 event_out = 1'b0; killed = 1'b0; 21 unique case (State) 22 ... 23 wait_min : begin 24 if (kill_s) killed = 1; 25 else if (max_timeout) begin 26 event_out = (clk_s & min_timeout); killed = event_out; 27 end 28 else event_out = 0; 29 end 30 endcase 31 end Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 19/23 19/23

Example Texture B Texture A Structure C Structure D Texture
F Texture G Texture E triggered at 21 s 2 s 1 s 4 s ∆ = 6 s ∆ = 14 s ∆ = 10 s ∆ = 4 s Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 20/23 20/23

Example Texture B Texture A Structure C Structure D Texture
F Texture G Texture E triggered at 21 s triggered at 31 s 2 s 1 s 4 s ∆ = 6 s ∆ = 14 s ∆ = 10 s ∆ = 4 s Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 20/23 20/23

Example Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs
for the Real-Time Interpretation of IS 21/23 21/23

Concluding Remarks Summary • A novel physical parallel implementation of
IS. • Low-latency and real-time performance. • Reaction to events is almost instant. • It is not affected by the complex behavior of operating systems (e.g., interrupt handling). Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 22/23 22/23

Concluding Remarks Future Work • Integrate multimedia processes to FPGAs
(e.g., FAUST†). • Implement a Fast Ethernet module to provide a reliable and low-rate communication between a FPGA and external applications running on standard computers. • Compile the Timed Automata semantics of IS into FPGA. • Distributed extension. †Robert Trausmuth, Christian Dusek, and Yann Orlarey. Using FAUST for FPGA Programming. In Proceedings of the 9th International Conference on Digital Audio Effects, pages 18–20, 2006 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 23/23 23/23

(e.g., FAUST). • Implement a Fast Ethernet module to provide a reliable and low-rate communication between a FPGA and external applications running on standard computers†. • Compile the Timed Automata semantics of IS into FPGA. • Distributed extension. †Rimas Aviziensis, Adrian Freed, Takahiko Suzuki, and David Wessel. Scalable Connectivity Processor for Computer Music Performance Systems. In Proceedings of the International Computer Music Conference, 2000 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 23/23 23/23

(e.g., FAUST). • Implement a Fast Ethernet module to provide a reliable and low-rate communication between a FPGA and external applications running on standard computers. • Compile the Timed Automata semantics of IS into FPGA† [Done]. • Distributed extension. †Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda. A framework for composition, verification and real-time performance of multimedia interactive scenarios. In 15th International Conference on Application of Concurrency to System Design (ACSD’15), 2015. To appear Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 23/23 23/23

(e.g., FAUST). • Implement a Fast Ethernet module to provide a reliable and low-rate communication between a FPGA and external applications running on standard computers. • Compile the Timed Automata semantics of IS into FPGA. • Distributed extension†. †Frank Opitz, Edris Sahak, and Bernd Schwarz. Accelerating distributed computing with FPGAs. Xcell Journal, 79:20–27, Q2 2012 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 23/23 23/23

Thank you for your attention! e-mail:[email protected]

Exploiting Parallelism in FPGAs for the Real-Time Interpretation of Interactive
Multimedia Scores* Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda Laboratoire Bordelais de Recherche en Informatique (LaBRI) Université de Bordeaux Journées d’Informatique Musicale (JIM 2015) Montreal - Canada, May 2015 1 *Supported by the ANR Project OSSIA and SCRIME

References Antoine Allombert. Aspects Temporels d’un Système de Partitions Musicales
Interactives pour la Composition et l’Exécution. PhD thesis, Université de Bordeaux, 2009. Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda. A framework for composition, verification and real-time performance of multimedia interactive scenarios. In 15th International Conference on Application of Concurrency to System Design (ACSD’15), 2015. To appear. Rimas Aviziensis, Adrian Freed, Takahiko Suzuki, and David Wessel. Scalable Connectivity Processor for Computer Music Performance Systems. In Proceedings of the International Computer Music Conference, 2000.

References Jeff Bier and Jennifer Eyre. BDTI study certifies high-level
synthesis flows for DSP-centric FPGA design. Xcell Journal, 71:12–17, Q2 2010. Rahul Dubey. Introduction to Embedded System Design Using Field Programmable Gate Arrays. Springer London, London, 2009. Luc Langlois. Multirate digital signal processing for high-speed data converters for high-speed data converters. Xcell Journal, 73:50–53, Q4 2010. Edward A. Lee. The problem with threads. Computer, 39(5):33–42, May 2006. Frank Opitz, Edris Sahak, and Bernd Schwarz. Accelerating distributed computing with FPGAs. Xcell Journal, 79:20–27, Q2 2012.

References J.J. Rodriguez-Andina, M.J. Moure, and M.D. Valdes. Features, Design
Tools, and Application Domains of FPGAs. IEEE Transactions on Industrial Electronics, 54(4):1810–1823, August 2007. Robert Trausmuth, Christian Dusek, and Yann Orlarey. Using FAUST for FPGA Programming. In Proceedings of the 9th International Conference on Digital Audio Effects, pages 18–20, 2006. David Wessel and Matthew Wright. Problems and prospects for intimate musical control of computers. In Ivan Poupyrev, Michael J. Lyons, Sidney Fels, and Tina Blaine, editors, New Interfaces for Musical Expression, NIME-01, Proceedings, Seattle, April 1-2, 2001, pages 11–14, 2001.

Exploiting Parallelism in FPGAs for the Real-Time Interpretation of Interactive Multimedia Scores

Exploiting Parallelism in FPGAs for the Real-Time Interpretation of Interactive Multimedia Scores

More Decks by Jaime Arias Almeida

Other Decks in Research

Featured

Transcript