Exploiting Parallelism in FPGAs for the Real-Time Interpretation of Interactive Multimedia Scores

Transcript

1. Exploiting Parallelism in FPGAs for the Real-Time
   Interpretation of Interactive Multimedia Scores*
   Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda
   Laboratoire Bordelais de Recherche en Informatique (LaBRI)
   Université de Bordeaux
   Journées d’Informatique Musicale (JIM 2015)
   Montreal - Canada, May 2015
   1
   
   *Supported by the ANR Project OSSIA and SCRIME
   

   View Slide

2. Motivation
   Interactive Multimedia Scenarios
   • Interactive Scores† (IS) is a formalism for composing and executing
   interactive multimedia scenarios.
   (a) Live-art performances. (b) Interactive museum
   installations.
   (c) Plastic art installations.
   Figure: Some applications of IS
   †Antoine Allombert. Aspects Temporels d’un Système de Partitions Musicales Interactives pour la
   Composition et l’Exécution. PhD thesis, Université de Bordeaux, 2009
   Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 1/23
   1/23
   

   View Slide

3. Motivation
   Interactive Multimedia Scenarios
   • Currently, IS is implemented in the software i-score†.
   Texture
   Structure
   Rigid Interval
   Semi-Flexible Interval
   Interaction Point
   OSC Messages Multimedia Processes
   Figure: Graphical interface of i-score
   †I-score website: http://i-score.org/
   Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 2/23
   2/23
   

   View Slide

4. Motivation
   Interactive Multimedia Scenarios
   • Some multimedia applications perform:
   ◦ real-time tasks
   ◦ compute-intensive tasks
   ◦ data-intensive tasks
   • Sometimes the performance of standard computers is not sufficient.
   • Most of the multimedia applications are executed on architectures and
   operating systems that do not provide low-latency and real-time
   performance.
   • I-score is implemented using threads†: non-determinism and
   unreliability.
   †Edward A. Lee. The problem with threads. Computer, 39(5):33–42, May 2006
   Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 3/23
   3/23
   

   View Slide

5. Motivation
   Field Programmable Gate Arrays (FPGAs)
   • FPGAs† have risen over the last years and at the same time their cost
   have been reduced.
   • FPGAs have already been used with success in many different
   industrial applications:
   ◦ Aerospace
   ◦ Automotive
   ◦ Medical
   ◦ Video and audio processing
   †Réseau de Portes Programmables in situ
   Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 4/23
   4/23
   

   View Slide

6. Motivation
   Field Programmable Gate Arrays (FPGAs)
   • FPGAs have risen over the last years and at the same time their cost
   have been reduced.
   • FPGAs have already been used with success in many different
   industrial applications†:
   ◦ Aerospace
   ◦ Automotive
   ◦ Medical
   ◦ Video and audio processing
   †J.J. Rodriguez-Andina, M.J. Moure, and M.D. Valdes. Features, Design Tools, and Application
   Domains of FPGAs. IEEE Transactions on Industrial Electronics, 54(4):1810–1823, August 2007
   Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 4/23
   4/23
   

   View Slide

7. Motivation
   Field Programmable Gate Arrays (FPGAs)
   • FPGAs offer the following benefits†:
   ◦ Reconfigurability
   ◦ High-level design
   ◦ Physical parallelism
   ◦ High-speed
   ◦ Reliability
   ◦ Re-use
   D Q
   Q
   Look-up Table
   (LUT)
   Flip Flop
   Mux
   Clock
   I0
   I1
   I2
   I3
   Out
   Configurable Logic Blocks (CLBs)
   Programmable Interconnection
   Network
   Configurable Input/Output Blocks (IOBs)
   †Rahul Dubey. Introduction to Embedded System Design Using Field Programmable Gate Arrays.
   Springer London, London, 2009
   Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 5/23
   5/23
   

   View Slide

8. Motivation
   Field Programmable Gate Arrays (FPGAs)*
   *Jeff Bier and Jennifer Eyre. BDTI study certifies high-level synthesis flows for DSP-centric
   FPGA design. Xcell Journal, 71:12–17, Q2 2010
   Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 6/23
   6/23
   

   View Slide

9. Motivation
   Field Programmable Gate Arrays (FPGAs)*
   *Luc Langlois. Multirate digital signal processing for high-speed data converters for high-speed
   data converters. Xcell Journal, 73:50–53, Q4 2010
   Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 7/23
   7/23
   

   View Slide

10. This talk is about …*
    An approach for a true parallel implementation on FPGAs of the
    execution model of interactive multimedia scenarios.
    *David Wessel and Matthew Wright. Problems and prospects for intimate musical control of
    computers. In Ivan Poupyrev, Michael J. Lyons, Sidney Fels, and Tina Blaine, editors, New Interfaces
    for Musical Expression, NIME-01, Proceedings, Seattle, April 1-2, 2001, pages 11–14, 2001
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 8/23
    8/23
    

    View Slide

11. Outline
    An Overview of IS
    Hardware Implementation
    Example
    Concluding Remarks
    

    View Slide

12. Outline
    An Overview of IS
    Hardware Implementation
    Example
    Concluding Remarks
    

    View Slide

13. An Overview of IS
    • An interactive scenario is composed of:
    ◦ Temporal objects
    • Textures → Multimedia processes
    • Structures → Hierarchy
    ◦ Temporal relations
    • Synchronization → ∆min
    = ∆max
    = 0
    • Rigid relations → ∆min
    = ∆max > 0
    • flexible relations → ∆min
    = 0 ∧ ∆max
    = ∞
    • Semi-flexible relations → ∆min ̸= ∆max ∧ ∆max ̸= ∞
    ◦ Interaction points
    • The system maintains the temporal relations during execution.
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 9/23
    9/23
    

    View Slide

14. An Overview of IS
    time (s)
    0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
    Texture B
    Texture A
    Structure C
    Structure D
    Texture F
    Texture G
    Texture E
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 10/23
    10/23
    

    View Slide

15. Outline
    An Overview of IS
    Hardware Implementation
    Example
    Concluding Remarks
    

    View Slide

16. Hardware Implementation
    Building our clock
    • We need to generate our clock signal: frequency divider.
    cycles =
    Tclockscore
    TclockFPGA
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 11/23
    11/23
    

    View Slide

17. Hardware Implementation
    Building our clock
    • We need to generate our clock signal: frequency divider.
    cycles =
    Tclockscore
    TclockFPGA
    =
    400ns
    10ns
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 11/23
    11/23
    

    View Slide

18. Hardware Implementation
    Building our clock
    • We need to generate our clock signal: frequency divider.
    cycles =
    Tclockscore
    TclockFPGA
    =
    400ns
    10ns
    = 20 cycles
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 11/23
    11/23
    

    View Slide

19. Hardware Implementation
    Building our clock
    • We need to generate our clock signal: frequency divider.
    cycles =
    Tclockscore
    TclockFPGA
    =
    400ns
    10ns
    = 20 cycles
    Figure: Frequency divider.
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 11/23
    11/23
    

    View Slide

20. Hardware Implementation
    The main module: a temporal relation
    • A temporal relation:
    ◦ It is defined between two points.
    ◦ Defines a precedence relation.
    ◦ Defines an interval of time.
    • An interaction point allows for agogic modifications.
    Texture A Texture B
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 12/23
    12/23
    

    View Slide

21. Hardware Implementation
    The main module: a temporal relation
    • A temporal relation:
    ◦ It is defined between two points.
    ◦ Defines a precedence relation.
    ◦ Defines an interval of time.
    • An interaction point allows for agogic modifications.
    Texture A Texture B
    end A start B
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 12/23
    12/23
    

    View Slide

22. Hardware Implementation
    The main module: a temporal relation
    • A temporal relation:
    ◦ It is defined between two points.
    ◦ Defines a precedence relation.
    ◦ Defines an interval of time.
    • An interaction point allows for agogic modifications.
    Texture A Texture B
    end A start B
    B after A
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 12/23
    12/23
    

    View Slide

23. Hardware Implementation
    The main module: a temporal relation
    • A temporal relation:
    ◦ It is defined between two points.
    ◦ Defines a precedence relation.
    ◦ Defines an interval of time.
    • An interaction point allows for agogic modifications.
    Texture A Texture B
    end A start B
    B after A
    min max
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 12/23
    12/23
    

    View Slide

24. Hardware Implementation
    The main module: a temporal relation
    • A temporal relation:
    ◦ It is defined between two points.
    ◦ Defines a precedence relation.
    ◦ Defines an interval of time.
    • An interaction point allows for agogic modifications.
    Texture A Texture B
    end A start B
    B after A
    min max
    interaction point
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 12/23
    12/23
    

    View Slide

25. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    –
    –
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23
    

    View Slide

26. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    –
    –
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23
    

    View Slide

27. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    –
    –
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23
    

    View Slide

28. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    –
    –
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23
    

    View Slide

29. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    –
    –
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23
    

    View Slide

30. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    –
    –
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23
    

    View Slide

31. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    –
    –
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23
    

    View Slide

32. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    8ms
    24ms
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23
    

    View Slide

33. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    4ms
    20ms
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23
    

    View Slide

34. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 2
    timer 1 0ms
    16ms
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23
    

    View Slide

35. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    0ms
    12ms
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23
    

    View Slide

36. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    0ms
    8ms
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23
    

    View Slide

37. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    0ms
    4ms
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23
    

    View Slide

38. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    0ms
    0ms
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23
    

    View Slide

39. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    0ms
    0ms
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23
    

    View Slide

40. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    0ms
    0ms
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23
    

    View Slide

41. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    0ms
    0ms
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23
    

    View Slide

42. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    0ms
    0ms
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23
    

    View Slide

43. Hardware Implementation
    Textures
    • A texture is the same as a temporal relation but it has an attached
    multimedia process.
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 14/23
    14/23
    

    View Slide

44. Hardware Implementation
    Handling complex temporal properties
    • Composers can add one or more temporal relations in order to
    define the starting time of a temporal object.
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 15/23
    15/23
    

    View Slide

45. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23
    

    View Slide

46. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    – –
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23
    

    View Slide

47. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    – –
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23
    

    View Slide

48. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    – –
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23
    

    View Slide

49. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    – –
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23
    

    View Slide

50. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    – –
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23
    

    View Slide

51. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    5ms 10ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23
    

    View Slide

52. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    4ms 9ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23
    

    View Slide

53. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    3ms 8ms
    timer 1 timer 2
    max(3, 5) = 3 min(8, 10) = 8
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23
    

    View Slide

54. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    5ms 8ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23
    

    View Slide

55. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    4ms 7ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23
    

    View Slide

56. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    3ms 6ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23
    

    View Slide

57. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    2ms 5ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23
    

    View Slide

58. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    1ms 4ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23
    

    View Slide

59. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    0ms 3ms
    timer 2
    timer 1
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23
    

    View Slide

60. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    0ms 2ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23
    

    View Slide

61. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    0ms 1ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23
    

    View Slide

62. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    0ms 0ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23
    

    View Slide

63. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    0ms 0ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23
    

    View Slide

64. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    0ms 0ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23
    

    View Slide

65. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    0ms 0ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23
    

    View Slide

66. Hardware Implementation
    Hierarchical organization
    • A structure contains other temporal objects and defines its own
    temporal organization.
    • A multimedia scenario is a structure (root).
    • We can model a structure as a temporal relation.
    • The end of a structure defines the end of its children.
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 17/23
    17/23
    

    View Slide

67. Hardware Implementation
    Hierarchical organization
    • A structure contains other temporal objects and defines its own
    temporal organization.
    • A multimedia scenario is a structure (root).
    • We can model a structure as a temporal relation.
    • The end of a structure defines the end of its children.
    idle
    new
    wait
    min.
    update
    wait
    max.
    end
    eventstart
    eventm
    in
    event
    ext /event
    end
    event
    m
    ax /event
    end
    eventstart
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 17/23
    17/23
    

    View Slide

68. Hardware Implementation
    Hierarchical organization
    • A structure contains other temporal objects and defines its own
    temporal organization.
    • A multimedia scenario is a structure (root).
    • We can model a structure as a temporal relation.
    • The end of a structure defines the end of its children.
    idle
    new
    wait
    min.
    update
    wait
    max.
    end
    eventstart
    eventm
    in
    event
    ext /event
    end
    event
    m
    ax /event
    end
    eventstart
    kill/killed
    kill/killed
    kill/killed
    kill/killed
    kill/killed
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 17/23
    17/23
    

    View Slide

69. Hardware Implementation
    Hierarchical organization
    • Conditions for stopping a structure:
    ◦ with IP:
    1. its maximum duration has elapsed.
    2. the interaction point has been triggered.
    ◦ normal:
    1. the maximum duration is infinity.
    2. its minimum duration has elapsed.
    3. its children have stopped.
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 18/23
    18/23
    

    View Slide

70. Hardware Implementation
    Hierarchical organization
    • Conditions for stopping a structure:
    ◦ with IP:
    1. its maximum duration has elapsed.
    2. the interaction point has been triggered.
    ◦ normal:
    1. the maximum duration is infinity.
    2. its minimum duration has elapsed.
    3. its children have stopped.
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 18/23
    18/23
    

    View Slide

71. Hardware Implementation
    Hierarchical organization
    • Conditions for stopping a structure:
    ◦ with IP:
    1. its maximum duration has elapsed.
    2. the interaction point has been triggered.
    ◦ normal:
    1. the maximum duration is infinity.
    2. its minimum duration has elapsed.
    3. its children have stopped.
    eventmin
    eventend1
    eventend2
    eventendn
    eventext
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 18/23
    18/23
    

    View Slide

72. Hardware Implementation
    Code in SystemVerilog
    1 ...
    2 // FSM
    3 always_ff @(posedge clk_fsm, posedge rst_s) begin: FSM
    4 if ( rst_s ) State = idle;
    5 else begin: FSM_Sequencer
    6 unique case (State)
    7 ...
    8 wait_min: begin: wait_min_state
    9 rst_counter <= 0; enable_counter <= 1;
    10 if (kill_s || max_timeout) State = stop;
    11 else if (min_timeout) State = wait_end;
    12 else if (set_s) State = new_interval;
    13 end: wait_min_state
    14 endcase
    15 end: FSM_Sequencer
    16 end: FSM
    17
    18 // Output
    19 always_comb begin
    20 event_out = 1'b0; killed = 1'b0;
    21 unique case (State)
    22 ...
    23 wait_min : begin
    24 if (kill_s) killed = 1;
    25 else if (max_timeout) begin
    26 event_out = (clk_s & min_timeout); killed = event_out;
    27 end
    28 else event_out = 0;
    29 end
    30 endcase
    31 end
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 19/23
    19/23
    

    View Slide

73. Outline
    An Overview of IS
    Hardware Implementation
    Example
    Concluding Remarks
    

    View Slide

74. Example
    Texture B
    Texture A
    Structure C
    Structure D
    Texture F
    Texture G
    Texture E
    triggered at 21 s
    2 s
    1 s
    4 s
    ∆ = 6 s
    ∆ = 14 s
    ∆ = 10 s
    ∆ = 4 s
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 20/23
    20/23
    

    View Slide

75. Example
    Texture B
    Texture A
    Structure C
    Structure D
    Texture F
    Texture G
    Texture E
    triggered at 21 s triggered at 31 s
    2 s
    1 s
    4 s
    ∆ = 6 s
    ∆ = 14 s
    ∆ = 10 s
    ∆ = 4 s
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 20/23
    20/23
    

    View Slide

76. Example
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 21/23
    21/23
    

    View Slide

77. Example
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 21/23
    21/23
    

    View Slide

78. Example
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 21/23
    21/23
    

    View Slide

79. Example
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 21/23
    21/23
    

    View Slide

80. Example
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 21/23
    21/23
    

    View Slide

81. Example
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 21/23
    21/23
    

    View Slide

82. Example
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 21/23
    21/23
    

    View Slide

83. Outline
    An Overview of IS
    Hardware Implementation
    Example
    Concluding Remarks
    

    View Slide

84. Concluding Remarks
    Summary
    • A novel physical parallel implementation of IS.
    • Low-latency and real-time performance.
    • Reaction to events is almost instant.
    • It is not affected by the complex behavior of operating systems (e.g.,
    interrupt handling).
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 22/23
    22/23
    

    View Slide

85. Concluding Remarks
    Future Work
    • Integrate multimedia processes to FPGAs (e.g., FAUST†).
    • Implement a Fast Ethernet module to provide a reliable and low-rate
    communication between a FPGA and external applications running
    on standard computers.
    • Compile the Timed Automata semantics of IS into FPGA.
    • Distributed extension.
    †Robert Trausmuth, Christian Dusek, and Yann Orlarey. Using FAUST for FPGA Programming.
    In Proceedings of the 9th International Conference on Digital Audio Effects, pages 18–20, 2006
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 23/23
    23/23
    

    View Slide

86. Concluding Remarks
    Future Work
    • Integrate multimedia processes to FPGAs (e.g., FAUST).
    • Implement a Fast Ethernet module to provide a reliable and low-rate
    communication between a FPGA and external applications running
    on standard computers†.
    • Compile the Timed Automata semantics of IS into FPGA.
    • Distributed extension.
    †Rimas Aviziensis, Adrian Freed, Takahiko Suzuki, and David Wessel. Scalable Connectivity
    Processor for Computer Music Performance Systems. In Proceedings of the International Computer
    Music Conference, 2000
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 23/23
    23/23
    

    View Slide

87. Concluding Remarks
    Future Work
    • Integrate multimedia processes to FPGAs (e.g., FAUST).
    • Implement a Fast Ethernet module to provide a reliable and low-rate
    communication between a FPGA and external applications running
    on standard computers.
    • Compile the Timed Automata semantics of IS into FPGA† [Done].
    • Distributed extension.
    †Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda. A framework for composition,
    verification and real-time performance of multimedia interactive scenarios. In 15th International
    Conference on Application of Concurrency to System Design (ACSD’15), 2015. To appear
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 23/23
    23/23
    

    View Slide

88. Concluding Remarks
    Future Work
    • Integrate multimedia processes to FPGAs (e.g., FAUST).
    • Implement a Fast Ethernet module to provide a reliable and low-rate
    communication between a FPGA and external applications running
    on standard computers.
    • Compile the Timed Automata semantics of IS into FPGA.
    • Distributed extension†.
    †Frank Opitz, Edris Sahak, and Bernd Schwarz. Accelerating distributed computing with
    FPGAs. Xcell Journal, 79:20–27, Q2 2012
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 23/23
    23/23
    

    View Slide

89. Thank you for your attention!
    e-mail:jaime.arias@labri.fr
    

    View Slide

90. Exploiting Parallelism in FPGAs for the Real-Time
    Interpretation of Interactive Multimedia Scores*
    Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda
    Laboratoire Bordelais de Recherche en Informatique (LaBRI)
    Université de Bordeaux
    Journées d’Informatique Musicale (JIM 2015)
    Montreal - Canada, May 2015
    1
    
    *Supported by the ANR Project OSSIA and SCRIME
    

    View Slide

91. References
    Antoine Allombert. Aspects Temporels d’un Système de Partitions
    Musicales Interactives pour la Composition et l’Exécution. PhD thesis,
    Université de Bordeaux, 2009.
    Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda. A
    framework for composition, verification and real-time performance
    of multimedia interactive scenarios. In 15th International Conference
    on Application of Concurrency to System Design (ACSD’15), 2015. To
    appear.
    Rimas Aviziensis, Adrian Freed, Takahiko Suzuki, and David Wessel.
    Scalable Connectivity Processor for Computer Music Performance
    Systems. In Proceedings of the International Computer Music
    Conference, 2000.
    

    View Slide

92. References
    Jeff Bier and Jennifer Eyre. BDTI study certifies high-level synthesis
    flows for DSP-centric FPGA design. Xcell Journal, 71:12–17, Q2
    2010.
    Rahul Dubey. Introduction to Embedded System Design Using Field
    Programmable Gate Arrays. Springer London, London, 2009.
    Luc Langlois. Multirate digital signal processing for high-speed data
    converters for high-speed data converters. Xcell Journal, 73:50–53,
    Q4 2010.
    Edward A. Lee. The problem with threads. Computer, 39(5):33–42,
    May 2006.
    Frank Opitz, Edris Sahak, and Bernd Schwarz. Accelerating
    distributed computing with FPGAs. Xcell Journal, 79:20–27, Q2
    2012.
    

    View Slide

93. References
    J.J. Rodriguez-Andina, M.J. Moure, and M.D. Valdes. Features, Design
    Tools, and Application Domains of FPGAs. IEEE Transactions on
    Industrial Electronics, 54(4):1810–1823, August 2007.
    Robert Trausmuth, Christian Dusek, and Yann Orlarey. Using
    FAUST for FPGA Programming. In Proceedings of the 9th International
    Conference on Digital Audio Effects, pages 18–20, 2006.
    David Wessel and Matthew Wright. Problems and prospects for
    intimate musical control of computers. In Ivan Poupyrev, Michael J.
    Lyons, Sidney Fels, and Tina Blaine, editors, New Interfaces for
    Musical Expression, NIME-01, Proceedings, Seattle, April 1-2, 2001,
    pages 11–14, 2001.
    

    View Slide