Exploiting Parallelism in FPGAs for the Real-Time Interpretation of Interactive Multimedia Scores

Exploiting Parallelism in FPGAs for the Real-Time Interpretation of Interactive Multimedia Scores

E49d1bfa9f9443686cbb9b4aaf6298af?s=128

Jaime Arias Almeida

May 01, 2015
Tweet

Transcript

  1. Exploiting Parallelism in FPGAs for the Real-Time Interpretation of Interactive

    Multimedia Scores* Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda Laboratoire Bordelais de Recherche en Informatique (LaBRI) Université de Bordeaux Journées d’Informatique Musicale (JIM 2015) Montreal - Canada, May 2015 1  *Supported by the ANR Project OSSIA and SCRIME
  2. Motivation Interactive Multimedia Scenarios • Interactive Scores† (IS) is a

    formalism for composing and executing interactive multimedia scenarios. (a) Live-art performances. (b) Interactive museum installations. (c) Plastic art installations. Figure: Some applications of IS †Antoine Allombert. Aspects Temporels d’un Système de Partitions Musicales Interactives pour la Composition et l’Exécution. PhD thesis, Université de Bordeaux, 2009 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 1/23 1/23
  3. Motivation Interactive Multimedia Scenarios • Currently, IS is implemented in

    the software i-score†. Texture Structure Rigid Interval Semi-Flexible Interval Interaction Point OSC Messages Multimedia Processes Figure: Graphical interface of i-score †I-score website: http://i-score.org/ Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 2/23 2/23
  4. Motivation Interactive Multimedia Scenarios • Some multimedia applications perform: ◦

    real-time tasks ◦ compute-intensive tasks ◦ data-intensive tasks • Sometimes the performance of standard computers is not sufficient. • Most of the multimedia applications are executed on architectures and operating systems that do not provide low-latency and real-time performance. • I-score is implemented using threads†: non-determinism and unreliability. †Edward A. Lee. The problem with threads. Computer, 39(5):33–42, May 2006 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 3/23 3/23
  5. Motivation Field Programmable Gate Arrays (FPGAs) • FPGAs† have risen

    over the last years and at the same time their cost have been reduced. • FPGAs have already been used with success in many different industrial applications: ◦ Aerospace ◦ Automotive ◦ Medical ◦ Video and audio processing †Réseau de Portes Programmables in situ Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 4/23 4/23
  6. Motivation Field Programmable Gate Arrays (FPGAs) • FPGAs have risen

    over the last years and at the same time their cost have been reduced. • FPGAs have already been used with success in many different industrial applications†: ◦ Aerospace ◦ Automotive ◦ Medical ◦ Video and audio processing †J.J. Rodriguez-Andina, M.J. Moure, and M.D. Valdes. Features, Design Tools, and Application Domains of FPGAs. IEEE Transactions on Industrial Electronics, 54(4):1810–1823, August 2007 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 4/23 4/23
  7. Motivation Field Programmable Gate Arrays (FPGAs) • FPGAs offer the

    following benefits†: ◦ Reconfigurability ◦ High-level design ◦ Physical parallelism ◦ High-speed ◦ Reliability ◦ Re-use D Q Q Look-up Table (LUT) Flip Flop Mux Clock I0 I1 I2 I3 Out Configurable Logic Blocks (CLBs) Programmable Interconnection Network Configurable Input/Output Blocks (IOBs) †Rahul Dubey. Introduction to Embedded System Design Using Field Programmable Gate Arrays. Springer London, London, 2009 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 5/23 5/23
  8. Motivation Field Programmable Gate Arrays (FPGAs)* *Jeff Bier and Jennifer

    Eyre. BDTI study certifies high-level synthesis flows for DSP-centric FPGA design. Xcell Journal, 71:12–17, Q2 2010 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 6/23 6/23
  9. Motivation Field Programmable Gate Arrays (FPGAs)* *Luc Langlois. Multirate digital

    signal processing for high-speed data converters for high-speed data converters. Xcell Journal, 73:50–53, Q4 2010 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 7/23 7/23
  10. This talk is about …* An approach for a true

    parallel implementation on FPGAs of the execution model of interactive multimedia scenarios. *David Wessel and Matthew Wright. Problems and prospects for intimate musical control of computers. In Ivan Poupyrev, Michael J. Lyons, Sidney Fels, and Tina Blaine, editors, New Interfaces for Musical Expression, NIME-01, Proceedings, Seattle, April 1-2, 2001, pages 11–14, 2001 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 8/23 8/23
  11. Outline An Overview of IS Hardware Implementation Example Concluding Remarks

  12. Outline An Overview of IS Hardware Implementation Example Concluding Remarks

  13. An Overview of IS • An interactive scenario is composed

    of: ◦ Temporal objects • Textures → Multimedia processes • Structures → Hierarchy ◦ Temporal relations • Synchronization → ∆min = ∆max = 0 • Rigid relations → ∆min = ∆max > 0 • flexible relations → ∆min = 0 ∧ ∆max = ∞ • Semi-flexible relations → ∆min ̸= ∆max ∧ ∆max ̸= ∞ ◦ Interaction points • The system maintains the temporal relations during execution. Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 9/23 9/23
  14. An Overview of IS time (s) 0 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Texture B Texture A Structure C Structure D Texture F Texture G Texture E Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 10/23 10/23
  15. Outline An Overview of IS Hardware Implementation Example Concluding Remarks

  16. Hardware Implementation Building our clock • We need to generate

    our clock signal: frequency divider. cycles = Tclockscore TclockFPGA Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 11/23 11/23
  17. Hardware Implementation Building our clock • We need to generate

    our clock signal: frequency divider. cycles = Tclockscore TclockFPGA = 400ns 10ns Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 11/23 11/23
  18. Hardware Implementation Building our clock • We need to generate

    our clock signal: frequency divider. cycles = Tclockscore TclockFPGA = 400ns 10ns = 20 cycles Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 11/23 11/23
  19. Hardware Implementation Building our clock • We need to generate

    our clock signal: frequency divider. cycles = Tclockscore TclockFPGA = 400ns 10ns = 20 cycles Figure: Frequency divider. Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 11/23 11/23
  20. Hardware Implementation The main module: a temporal relation • A

    temporal relation: ◦ It is defined between two points. ◦ Defines a precedence relation. ◦ Defines an interval of time. • An interaction point allows for agogic modifications. Texture A Texture B Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 12/23 12/23
  21. Hardware Implementation The main module: a temporal relation • A

    temporal relation: ◦ It is defined between two points. ◦ Defines a precedence relation. ◦ Defines an interval of time. • An interaction point allows for agogic modifications. Texture A Texture B end A start B Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 12/23 12/23
  22. Hardware Implementation The main module: a temporal relation • A

    temporal relation: ◦ It is defined between two points. ◦ Defines a precedence relation. ◦ Defines an interval of time. • An interaction point allows for agogic modifications. Texture A Texture B end A start B B after A Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 12/23 12/23
  23. Hardware Implementation The main module: a temporal relation • A

    temporal relation: ◦ It is defined between two points. ◦ Defines a precedence relation. ◦ Defines an interval of time. • An interaction point allows for agogic modifications. Texture A Texture B end A start B B after A min max Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 12/23 12/23
  24. Hardware Implementation The main module: a temporal relation • A

    temporal relation: ◦ It is defined between two points. ◦ Defines a precedence relation. ◦ Defines an interval of time. • An interaction point allows for agogic modifications. Texture A Texture B end A start B B after A min max interaction point Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 12/23 12/23
  25. Hardware Implementation The main module: a temporal relation time (s)

    0 1 2 3 4 5 6 Texture A Texture B timer 1 timer 2 – – idle new wait min. wait max. end eventstart eventmin eventext /eventend eventmax /eventend Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23 13/23
  26. Hardware Implementation The main module: a temporal relation time (s)

    0 1 2 3 4 5 6 Texture A Texture B timer 1 timer 2 – – idle new wait min. wait max. end eventstart eventmin eventext /eventend eventmax /eventend Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23 13/23
  27. Hardware Implementation The main module: a temporal relation time (s)

    0 1 2 3 4 5 6 Texture A Texture B timer 1 timer 2 – – idle new wait min. wait max. end eventstart eventmin eventext /eventend eventmax /eventend Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23 13/23
  28. Hardware Implementation The main module: a temporal relation time (s)

    0 1 2 3 4 5 6 Texture A Texture B timer 1 timer 2 – – idle new wait min. wait max. end eventstart eventmin eventext /eventend eventmax /eventend Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23 13/23
  29. Hardware Implementation The main module: a temporal relation time (s)

    0 1 2 3 4 5 6 Texture A Texture B timer 1 timer 2 – – idle new wait min. wait max. end eventstart eventmin eventext /eventend eventmax /eventend Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23 13/23
  30. Hardware Implementation The main module: a temporal relation time (s)

    0 1 2 3 4 5 6 Texture A Texture B timer 1 timer 2 – – idle new wait min. wait max. end eventstart eventmin eventext /eventend eventmax /eventend Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23 13/23
  31. Hardware Implementation The main module: a temporal relation time (s)

    0 1 2 3 4 5 6 Texture A Texture B timer 1 timer 2 – – idle new wait min. wait max. end eventstart eventmin eventext /eventend eventmax /eventend Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23 13/23
  32. Hardware Implementation The main module: a temporal relation time (s)

    0 1 2 3 4 5 6 Texture A Texture B timer 1 timer 2 8ms 24ms idle new wait min. wait max. end eventstart eventmin eventext /eventend eventmax /eventend Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23 13/23
  33. Hardware Implementation The main module: a temporal relation time (s)

    0 1 2 3 4 5 6 Texture A Texture B timer 1 timer 2 4ms 20ms idle new wait min. wait max. end eventstart eventmin eventext /eventend eventmax /eventend Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23 13/23
  34. Hardware Implementation The main module: a temporal relation time (s)

    0 1 2 3 4 5 6 Texture A Texture B timer 2 timer 1 0ms 16ms idle new wait min. wait max. end eventstart eventmin eventext /eventend eventmax /eventend Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23 13/23
  35. Hardware Implementation The main module: a temporal relation time (s)

    0 1 2 3 4 5 6 Texture A Texture B timer 1 timer 2 0ms 12ms idle new wait min. wait max. end eventstart eventmin eventext /eventend eventmax /eventend Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23 13/23
  36. Hardware Implementation The main module: a temporal relation time (s)

    0 1 2 3 4 5 6 Texture A Texture B timer 1 timer 2 0ms 8ms idle new wait min. wait max. end eventstart eventmin eventext /eventend eventmax /eventend Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23 13/23
  37. Hardware Implementation The main module: a temporal relation time (s)

    0 1 2 3 4 5 6 Texture A Texture B timer 1 timer 2 0ms 4ms idle new wait min. wait max. end eventstart eventmin eventext /eventend eventmax /eventend Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23 13/23
  38. Hardware Implementation The main module: a temporal relation time (s)

    0 1 2 3 4 5 6 Texture A Texture B timer 1 timer 2 0ms 0ms idle new wait min. wait max. end eventstart eventmin eventext /eventend eventmax /eventend Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23 13/23
  39. Hardware Implementation The main module: a temporal relation time (s)

    0 1 2 3 4 5 6 Texture A Texture B timer 1 timer 2 0ms 0ms idle new wait min. wait max. end eventstart eventmin eventext /eventend eventmax /eventend Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23 13/23
  40. Hardware Implementation The main module: a temporal relation time (s)

    0 1 2 3 4 5 6 Texture A Texture B timer 1 timer 2 0ms 0ms idle new wait min. wait max. end eventstart eventmin eventext /eventend eventmax /eventend Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23 13/23
  41. Hardware Implementation The main module: a temporal relation time (s)

    0 1 2 3 4 5 6 Texture A Texture B timer 1 timer 2 0ms 0ms idle new wait min. wait max. end eventstart eventmin eventext /eventend eventmax /eventend Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23 13/23
  42. Hardware Implementation The main module: a temporal relation time (s)

    0 1 2 3 4 5 6 Texture A Texture B timer 1 timer 2 0ms 0ms idle new wait min. wait max. end eventstart eventmin eventext /eventend eventmax /eventend Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23 13/23
  43. Hardware Implementation Textures • A texture is the same as

    a temporal relation but it has an attached multimedia process. Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 14/23 14/23
  44. Hardware Implementation Handling complex temporal properties • Composers can add

    one or more temporal relations in order to define the starting time of a temporal object. Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 15/23 15/23
  45. Hardware Implementation Handling complex temporal properties time 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23
  46. Hardware Implementation Handling complex temporal properties time 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. wait max. end eventstart eventmin eventext /eventend eventmax /eventend – – timer 1 timer 2 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23
  47. Hardware Implementation Handling complex temporal properties time 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. update wait max. end eventstart eventmin eventext /eventend eventmax /eventend eventstart – – timer 1 timer 2 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23
  48. Hardware Implementation Handling complex temporal properties time 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. update wait max. end eventstart eventmin eventext /eventend eventmax /eventend eventstart – – timer 1 timer 2 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23
  49. Hardware Implementation Handling complex temporal properties time 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. update wait max. end eventstart eventmin eventext /eventend eventmax /eventend eventstart – – timer 1 timer 2 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23
  50. Hardware Implementation Handling complex temporal properties time 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. update wait max. end eventstart eventmin eventext /eventend eventmax /eventend eventstart – – timer 1 timer 2 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23
  51. Hardware Implementation Handling complex temporal properties time 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. update wait max. end eventstart eventmin eventext /eventend eventmax /eventend eventstart 5ms 10ms timer 1 timer 2 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23
  52. Hardware Implementation Handling complex temporal properties time 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. update wait max. end eventstart eventmin eventext /eventend eventmax /eventend eventstart 4ms 9ms timer 1 timer 2 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23
  53. Hardware Implementation Handling complex temporal properties time 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. update wait max. end eventstart eventmin eventext /eventend eventmax /eventend eventstart 3ms 8ms timer 1 timer 2 max(3, 5) = 3 min(8, 10) = 8 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23
  54. Hardware Implementation Handling complex temporal properties time 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. update wait max. end eventstart eventmin eventext /eventend eventmax /eventend eventstart 5ms 8ms timer 1 timer 2 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23
  55. Hardware Implementation Handling complex temporal properties time 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. update wait max. end eventstart eventmin eventext /eventend eventmax /eventend eventstart 4ms 7ms timer 1 timer 2 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23
  56. Hardware Implementation Handling complex temporal properties time 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. update wait max. end eventstart eventmin eventext /eventend eventmax /eventend eventstart 3ms 6ms timer 1 timer 2 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23
  57. Hardware Implementation Handling complex temporal properties time 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. update wait max. end eventstart eventmin eventext /eventend eventmax /eventend eventstart 2ms 5ms timer 1 timer 2 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23
  58. Hardware Implementation Handling complex temporal properties time 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. update wait max. end eventstart eventmin eventext /eventend eventmax /eventend eventstart 1ms 4ms timer 1 timer 2 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23
  59. Hardware Implementation Handling complex temporal properties time 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. update wait max. end eventstart eventmin eventext /eventend eventmax /eventend eventstart 0ms 3ms timer 2 timer 1 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23
  60. Hardware Implementation Handling complex temporal properties time 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. update wait max. end eventstart eventmin eventext /eventend eventmax /eventend eventstart 0ms 2ms timer 1 timer 2 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23
  61. Hardware Implementation Handling complex temporal properties time 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. update wait max. end eventstart eventmin eventext /eventend eventmax /eventend eventstart 0ms 1ms timer 1 timer 2 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23
  62. Hardware Implementation Handling complex temporal properties time 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. update wait max. end eventstart eventmin eventext /eventend eventmax /eventend eventstart 0ms 0ms timer 1 timer 2 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23
  63. Hardware Implementation Handling complex temporal properties time 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. update wait max. end eventstart eventmin eventext /eventend eventmax /eventend eventstart 0ms 0ms timer 1 timer 2 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23
  64. Hardware Implementation Handling complex temporal properties time 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. update wait max. end eventstart eventmin eventext /eventend eventmax /eventend eventstart 0ms 0ms timer 1 timer 2 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23
  65. Hardware Implementation Handling complex temporal properties time 1 2 3

    4 5 6 7 8 9 10 11 12 13 14 15 16 min max TR 2 min max TR 1 min max interaction point idle new wait min. update wait max. end eventstart eventmin eventext /eventend eventmax /eventend eventstart 0ms 0ms timer 1 timer 2 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23 16/23
  66. Hardware Implementation Hierarchical organization • A structure contains other temporal

    objects and defines its own temporal organization. • A multimedia scenario is a structure (root). • We can model a structure as a temporal relation. • The end of a structure defines the end of its children. Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 17/23 17/23
  67. Hardware Implementation Hierarchical organization • A structure contains other temporal

    objects and defines its own temporal organization. • A multimedia scenario is a structure (root). • We can model a structure as a temporal relation. • The end of a structure defines the end of its children. idle new wait min. update wait max. end eventstart eventm in event ext /event end event m ax /event end eventstart Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 17/23 17/23
  68. Hardware Implementation Hierarchical organization • A structure contains other temporal

    objects and defines its own temporal organization. • A multimedia scenario is a structure (root). • We can model a structure as a temporal relation. • The end of a structure defines the end of its children. idle new wait min. update wait max. end eventstart eventm in event ext /event end event m ax /event end eventstart kill/killed kill/killed kill/killed kill/killed kill/killed Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 17/23 17/23
  69. Hardware Implementation Hierarchical organization • Conditions for stopping a structure:

    ◦ with IP: 1. its maximum duration has elapsed. 2. the interaction point has been triggered. ◦ normal: 1. the maximum duration is infinity. 2. its minimum duration has elapsed. 3. its children have stopped. Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 18/23 18/23
  70. Hardware Implementation Hierarchical organization • Conditions for stopping a structure:

    ◦ with IP: 1. its maximum duration has elapsed. 2. the interaction point has been triggered. ◦ normal: 1. the maximum duration is infinity. 2. its minimum duration has elapsed. 3. its children have stopped. Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 18/23 18/23
  71. Hardware Implementation Hierarchical organization • Conditions for stopping a structure:

    ◦ with IP: 1. its maximum duration has elapsed. 2. the interaction point has been triggered. ◦ normal: 1. the maximum duration is infinity. 2. its minimum duration has elapsed. 3. its children have stopped. eventmin eventend1 eventend2 eventendn eventext Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 18/23 18/23
  72. Hardware Implementation Code in SystemVerilog 1 ... 2 // FSM

    3 always_ff @(posedge clk_fsm, posedge rst_s) begin: FSM 4 if ( rst_s ) State = idle; 5 else begin: FSM_Sequencer 6 unique case (State) 7 ... 8 wait_min: begin: wait_min_state 9 rst_counter <= 0; enable_counter <= 1; 10 if (kill_s || max_timeout) State = stop; 11 else if (min_timeout) State = wait_end; 12 else if (set_s) State = new_interval; 13 end: wait_min_state 14 endcase 15 end: FSM_Sequencer 16 end: FSM 17 18 // Output 19 always_comb begin 20 event_out = 1'b0; killed = 1'b0; 21 unique case (State) 22 ... 23 wait_min : begin 24 if (kill_s) killed = 1; 25 else if (max_timeout) begin 26 event_out = (clk_s & min_timeout); killed = event_out; 27 end 28 else event_out = 0; 29 end 30 endcase 31 end Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 19/23 19/23
  73. Outline An Overview of IS Hardware Implementation Example Concluding Remarks

  74. Example Texture B Texture A Structure C Structure D Texture

    F Texture G Texture E triggered at 21 s 2 s 1 s 4 s ∆ = 6 s ∆ = 14 s ∆ = 10 s ∆ = 4 s Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 20/23 20/23
  75. Example Texture B Texture A Structure C Structure D Texture

    F Texture G Texture E triggered at 21 s triggered at 31 s 2 s 1 s 4 s ∆ = 6 s ∆ = 14 s ∆ = 10 s ∆ = 4 s Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 20/23 20/23
  76. Example Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs

    for the Real-Time Interpretation of IS 21/23 21/23
  77. Example Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs

    for the Real-Time Interpretation of IS 21/23 21/23
  78. Example Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs

    for the Real-Time Interpretation of IS 21/23 21/23
  79. Example Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs

    for the Real-Time Interpretation of IS 21/23 21/23
  80. Example Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs

    for the Real-Time Interpretation of IS 21/23 21/23
  81. Example Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs

    for the Real-Time Interpretation of IS 21/23 21/23
  82. Example Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs

    for the Real-Time Interpretation of IS 21/23 21/23
  83. Outline An Overview of IS Hardware Implementation Example Concluding Remarks

  84. Concluding Remarks Summary • A novel physical parallel implementation of

    IS. • Low-latency and real-time performance. • Reaction to events is almost instant. • It is not affected by the complex behavior of operating systems (e.g., interrupt handling). Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 22/23 22/23
  85. Concluding Remarks Future Work • Integrate multimedia processes to FPGAs

    (e.g., FAUST†). • Implement a Fast Ethernet module to provide a reliable and low-rate communication between a FPGA and external applications running on standard computers. • Compile the Timed Automata semantics of IS into FPGA. • Distributed extension. †Robert Trausmuth, Christian Dusek, and Yann Orlarey. Using FAUST for FPGA Programming. In Proceedings of the 9th International Conference on Digital Audio Effects, pages 18–20, 2006 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 23/23 23/23
  86. Concluding Remarks Future Work • Integrate multimedia processes to FPGAs

    (e.g., FAUST). • Implement a Fast Ethernet module to provide a reliable and low-rate communication between a FPGA and external applications running on standard computers†. • Compile the Timed Automata semantics of IS into FPGA. • Distributed extension. †Rimas Aviziensis, Adrian Freed, Takahiko Suzuki, and David Wessel. Scalable Connectivity Processor for Computer Music Performance Systems. In Proceedings of the International Computer Music Conference, 2000 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 23/23 23/23
  87. Concluding Remarks Future Work • Integrate multimedia processes to FPGAs

    (e.g., FAUST). • Implement a Fast Ethernet module to provide a reliable and low-rate communication between a FPGA and external applications running on standard computers. • Compile the Timed Automata semantics of IS into FPGA† [Done]. • Distributed extension. †Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda. A framework for composition, verification and real-time performance of multimedia interactive scenarios. In 15th International Conference on Application of Concurrency to System Design (ACSD’15), 2015. To appear Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 23/23 23/23
  88. Concluding Remarks Future Work • Integrate multimedia processes to FPGAs

    (e.g., FAUST). • Implement a Fast Ethernet module to provide a reliable and low-rate communication between a FPGA and external applications running on standard computers. • Compile the Timed Automata semantics of IS into FPGA. • Distributed extension†. †Frank Opitz, Edris Sahak, and Bernd Schwarz. Accelerating distributed computing with FPGAs. Xcell Journal, 79:20–27, Q2 2012 Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 23/23 23/23
  89. Thank you for your attention! e-mail:jaime.arias@labri.fr

  90. Exploiting Parallelism in FPGAs for the Real-Time Interpretation of Interactive

    Multimedia Scores* Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda Laboratoire Bordelais de Recherche en Informatique (LaBRI) Université de Bordeaux Journées d’Informatique Musicale (JIM 2015) Montreal - Canada, May 2015 1  *Supported by the ANR Project OSSIA and SCRIME
  91. References Antoine Allombert. Aspects Temporels d’un Système de Partitions Musicales

    Interactives pour la Composition et l’Exécution. PhD thesis, Université de Bordeaux, 2009. Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda. A framework for composition, verification and real-time performance of multimedia interactive scenarios. In 15th International Conference on Application of Concurrency to System Design (ACSD’15), 2015. To appear. Rimas Aviziensis, Adrian Freed, Takahiko Suzuki, and David Wessel. Scalable Connectivity Processor for Computer Music Performance Systems. In Proceedings of the International Computer Music Conference, 2000.
  92. References Jeff Bier and Jennifer Eyre. BDTI study certifies high-level

    synthesis flows for DSP-centric FPGA design. Xcell Journal, 71:12–17, Q2 2010. Rahul Dubey. Introduction to Embedded System Design Using Field Programmable Gate Arrays. Springer London, London, 2009. Luc Langlois. Multirate digital signal processing for high-speed data converters for high-speed data converters. Xcell Journal, 73:50–53, Q4 2010. Edward A. Lee. The problem with threads. Computer, 39(5):33–42, May 2006. Frank Opitz, Edris Sahak, and Bernd Schwarz. Accelerating distributed computing with FPGAs. Xcell Journal, 79:20–27, Q2 2012.
  93. References J.J. Rodriguez-Andina, M.J. Moure, and M.D. Valdes. Features, Design

    Tools, and Application Domains of FPGAs. IEEE Transactions on Industrial Electronics, 54(4):1810–1823, August 2007. Robert Trausmuth, Christian Dusek, and Yann Orlarey. Using FAUST for FPGA Programming. In Proceedings of the 9th International Conference on Digital Audio Effects, pages 18–20, 2006. David Wessel and Matthew Wright. Problems and prospects for intimate musical control of computers. In Ivan Poupyrev, Michael J. Lyons, Sidney Fels, and Tina Blaine, editors, New Interfaces for Musical Expression, NIME-01, Proceedings, Seattle, April 1-2, 2001, pages 11–14, 2001.