Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Exploiting Parallelism in FPGAs for the Real-Time Interpretation of Interactive Multimedia Scores

Exploiting Parallelism in FPGAs for the Real-Time Interpretation of Interactive Multimedia Scores

Jaime Arias Almeida

May 01, 2015
Tweet

More Decks by Jaime Arias Almeida

Other Decks in Research

Transcript

  1. Exploiting Parallelism in FPGAs for the Real-Time
    Interpretation of Interactive Multimedia Scores*
    Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda
    Laboratoire Bordelais de Recherche en Informatique (LaBRI)
    Université de Bordeaux
    Journées d’Informatique Musicale (JIM 2015)
    Montreal - Canada, May 2015
    1

    *Supported by the ANR Project OSSIA and SCRIME

    View Slide

  2. Motivation
    Interactive Multimedia Scenarios
    • Interactive Scores† (IS) is a formalism for composing and executing
    interactive multimedia scenarios.
    (a) Live-art performances. (b) Interactive museum
    installations.
    (c) Plastic art installations.
    Figure: Some applications of IS
    †Antoine Allombert. Aspects Temporels d’un Système de Partitions Musicales Interactives pour la
    Composition et l’Exécution. PhD thesis, Université de Bordeaux, 2009
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 1/23
    1/23

    View Slide

  3. Motivation
    Interactive Multimedia Scenarios
    • Currently, IS is implemented in the software i-score†.
    Texture
    Structure
    Rigid Interval
    Semi-Flexible Interval
    Interaction Point
    OSC Messages Multimedia Processes
    Figure: Graphical interface of i-score
    †I-score website: http://i-score.org/
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 2/23
    2/23

    View Slide

  4. Motivation
    Interactive Multimedia Scenarios
    • Some multimedia applications perform:
    ◦ real-time tasks
    ◦ compute-intensive tasks
    ◦ data-intensive tasks
    • Sometimes the performance of standard computers is not sufficient.
    • Most of the multimedia applications are executed on architectures and
    operating systems that do not provide low-latency and real-time
    performance.
    • I-score is implemented using threads†: non-determinism and
    unreliability.
    †Edward A. Lee. The problem with threads. Computer, 39(5):33–42, May 2006
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 3/23
    3/23

    View Slide

  5. Motivation
    Field Programmable Gate Arrays (FPGAs)
    • FPGAs† have risen over the last years and at the same time their cost
    have been reduced.
    • FPGAs have already been used with success in many different
    industrial applications:
    ◦ Aerospace
    ◦ Automotive
    ◦ Medical
    ◦ Video and audio processing
    †Réseau de Portes Programmables in situ
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 4/23
    4/23

    View Slide

  6. Motivation
    Field Programmable Gate Arrays (FPGAs)
    • FPGAs have risen over the last years and at the same time their cost
    have been reduced.
    • FPGAs have already been used with success in many different
    industrial applications†:
    ◦ Aerospace
    ◦ Automotive
    ◦ Medical
    ◦ Video and audio processing
    †J.J. Rodriguez-Andina, M.J. Moure, and M.D. Valdes. Features, Design Tools, and Application
    Domains of FPGAs. IEEE Transactions on Industrial Electronics, 54(4):1810–1823, August 2007
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 4/23
    4/23

    View Slide

  7. Motivation
    Field Programmable Gate Arrays (FPGAs)
    • FPGAs offer the following benefits†:
    ◦ Reconfigurability
    ◦ High-level design
    ◦ Physical parallelism
    ◦ High-speed
    ◦ Reliability
    ◦ Re-use
    D Q
    Q
    Look-up Table
    (LUT)
    Flip Flop
    Mux
    Clock
    I0
    I1
    I2
    I3
    Out
    Configurable Logic Blocks (CLBs)
    Programmable Interconnection
    Network
    Configurable Input/Output Blocks (IOBs)
    †Rahul Dubey. Introduction to Embedded System Design Using Field Programmable Gate Arrays.
    Springer London, London, 2009
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 5/23
    5/23

    View Slide

  8. Motivation
    Field Programmable Gate Arrays (FPGAs)*
    *Jeff Bier and Jennifer Eyre. BDTI study certifies high-level synthesis flows for DSP-centric
    FPGA design. Xcell Journal, 71:12–17, Q2 2010
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 6/23
    6/23

    View Slide

  9. Motivation
    Field Programmable Gate Arrays (FPGAs)*
    *Luc Langlois. Multirate digital signal processing for high-speed data converters for high-speed
    data converters. Xcell Journal, 73:50–53, Q4 2010
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 7/23
    7/23

    View Slide

  10. This talk is about …*
    An approach for a true parallel implementation on FPGAs of the
    execution model of interactive multimedia scenarios.
    *David Wessel and Matthew Wright. Problems and prospects for intimate musical control of
    computers. In Ivan Poupyrev, Michael J. Lyons, Sidney Fels, and Tina Blaine, editors, New Interfaces
    for Musical Expression, NIME-01, Proceedings, Seattle, April 1-2, 2001, pages 11–14, 2001
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 8/23
    8/23

    View Slide

  11. Outline
    An Overview of IS
    Hardware Implementation
    Example
    Concluding Remarks

    View Slide

  12. Outline
    An Overview of IS
    Hardware Implementation
    Example
    Concluding Remarks

    View Slide

  13. An Overview of IS
    • An interactive scenario is composed of:
    ◦ Temporal objects
    • Textures → Multimedia processes
    • Structures → Hierarchy
    ◦ Temporal relations
    • Synchronization → ∆min
    = ∆max
    = 0
    • Rigid relations → ∆min
    = ∆max > 0
    • flexible relations → ∆min
    = 0 ∧ ∆max
    = ∞
    • Semi-flexible relations → ∆min ̸= ∆max ∧ ∆max ̸= ∞
    ◦ Interaction points
    • The system maintains the temporal relations during execution.
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 9/23
    9/23

    View Slide

  14. An Overview of IS
    time (s)
    0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
    Texture B
    Texture A
    Structure C
    Structure D
    Texture F
    Texture G
    Texture E
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 10/23
    10/23

    View Slide

  15. Outline
    An Overview of IS
    Hardware Implementation
    Example
    Concluding Remarks

    View Slide

  16. Hardware Implementation
    Building our clock
    • We need to generate our clock signal: frequency divider.
    cycles =
    Tclockscore
    TclockFPGA
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 11/23
    11/23

    View Slide

  17. Hardware Implementation
    Building our clock
    • We need to generate our clock signal: frequency divider.
    cycles =
    Tclockscore
    TclockFPGA
    =
    400ns
    10ns
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 11/23
    11/23

    View Slide

  18. Hardware Implementation
    Building our clock
    • We need to generate our clock signal: frequency divider.
    cycles =
    Tclockscore
    TclockFPGA
    =
    400ns
    10ns
    = 20 cycles
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 11/23
    11/23

    View Slide

  19. Hardware Implementation
    Building our clock
    • We need to generate our clock signal: frequency divider.
    cycles =
    Tclockscore
    TclockFPGA
    =
    400ns
    10ns
    = 20 cycles
    Figure: Frequency divider.
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 11/23
    11/23

    View Slide

  20. Hardware Implementation
    The main module: a temporal relation
    • A temporal relation:
    ◦ It is defined between two points.
    ◦ Defines a precedence relation.
    ◦ Defines an interval of time.
    • An interaction point allows for agogic modifications.
    Texture A Texture B
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 12/23
    12/23

    View Slide

  21. Hardware Implementation
    The main module: a temporal relation
    • A temporal relation:
    ◦ It is defined between two points.
    ◦ Defines a precedence relation.
    ◦ Defines an interval of time.
    • An interaction point allows for agogic modifications.
    Texture A Texture B
    end A start B
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 12/23
    12/23

    View Slide

  22. Hardware Implementation
    The main module: a temporal relation
    • A temporal relation:
    ◦ It is defined between two points.
    ◦ Defines a precedence relation.
    ◦ Defines an interval of time.
    • An interaction point allows for agogic modifications.
    Texture A Texture B
    end A start B
    B after A
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 12/23
    12/23

    View Slide

  23. Hardware Implementation
    The main module: a temporal relation
    • A temporal relation:
    ◦ It is defined between two points.
    ◦ Defines a precedence relation.
    ◦ Defines an interval of time.
    • An interaction point allows for agogic modifications.
    Texture A Texture B
    end A start B
    B after A
    min max
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 12/23
    12/23

    View Slide

  24. Hardware Implementation
    The main module: a temporal relation
    • A temporal relation:
    ◦ It is defined between two points.
    ◦ Defines a precedence relation.
    ◦ Defines an interval of time.
    • An interaction point allows for agogic modifications.
    Texture A Texture B
    end A start B
    B after A
    min max
    interaction point
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 12/23
    12/23

    View Slide

  25. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2


    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23

    View Slide

  26. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2


    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23

    View Slide

  27. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2


    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23

    View Slide

  28. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2


    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23

    View Slide

  29. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2


    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23

    View Slide

  30. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2


    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23

    View Slide

  31. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2


    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23

    View Slide

  32. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    8ms
    24ms
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23

    View Slide

  33. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    4ms
    20ms
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23

    View Slide

  34. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 2
    timer 1 0ms
    16ms
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23

    View Slide

  35. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    0ms
    12ms
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23

    View Slide

  36. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    0ms
    8ms
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23

    View Slide

  37. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    0ms
    4ms
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23

    View Slide

  38. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    0ms
    0ms
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23

    View Slide

  39. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    0ms
    0ms
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23

    View Slide

  40. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    0ms
    0ms
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23

    View Slide

  41. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    0ms
    0ms
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23

    View Slide

  42. Hardware Implementation
    The main module: a temporal relation
    time (s)
    0 1 2 3 4 5 6
    Texture A Texture B
    timer 1
    timer 2
    0ms
    0ms
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 13/23
    13/23

    View Slide

  43. Hardware Implementation
    Textures
    • A texture is the same as a temporal relation but it has an attached
    multimedia process.
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 14/23
    14/23

    View Slide

  44. Hardware Implementation
    Handling complex temporal properties
    • Composers can add one or more temporal relations in order to
    define the starting time of a temporal object.
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 15/23
    15/23

    View Slide

  45. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23

    View Slide

  46. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    – –
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23

    View Slide

  47. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    – –
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23

    View Slide

  48. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    – –
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23

    View Slide

  49. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    – –
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23

    View Slide

  50. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    – –
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23

    View Slide

  51. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    5ms 10ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23

    View Slide

  52. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    4ms 9ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23

    View Slide

  53. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    3ms 8ms
    timer 1 timer 2
    max(3, 5) = 3 min(8, 10) = 8
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23

    View Slide

  54. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    5ms 8ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23

    View Slide

  55. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    4ms 7ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23

    View Slide

  56. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    3ms 6ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23

    View Slide

  57. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    2ms 5ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23

    View Slide

  58. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    1ms 4ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23

    View Slide

  59. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    0ms 3ms
    timer 2
    timer 1
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23

    View Slide

  60. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    0ms 2ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23

    View Slide

  61. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    0ms 1ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23

    View Slide

  62. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    0ms 0ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23

    View Slide

  63. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    0ms 0ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23

    View Slide

  64. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    0ms 0ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23

    View Slide

  65. Hardware Implementation
    Handling complex temporal properties
    time
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
    min max
    TR 2
    min max
    TR 1
    min max
    interaction point
    idle new
    wait
    min.
    update
    wait
    max.
    end
    eventstart eventmin
    eventext /eventend
    eventmax /eventend
    eventstart
    0ms 0ms
    timer 1 timer 2
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 16/23
    16/23

    View Slide

  66. Hardware Implementation
    Hierarchical organization
    • A structure contains other temporal objects and defines its own
    temporal organization.
    • A multimedia scenario is a structure (root).
    • We can model a structure as a temporal relation.
    • The end of a structure defines the end of its children.
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 17/23
    17/23

    View Slide

  67. Hardware Implementation
    Hierarchical organization
    • A structure contains other temporal objects and defines its own
    temporal organization.
    • A multimedia scenario is a structure (root).
    • We can model a structure as a temporal relation.
    • The end of a structure defines the end of its children.
    idle
    new
    wait
    min.
    update
    wait
    max.
    end
    eventstart
    eventm
    in
    event
    ext /event
    end
    event
    m
    ax /event
    end
    eventstart
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 17/23
    17/23

    View Slide

  68. Hardware Implementation
    Hierarchical organization
    • A structure contains other temporal objects and defines its own
    temporal organization.
    • A multimedia scenario is a structure (root).
    • We can model a structure as a temporal relation.
    • The end of a structure defines the end of its children.
    idle
    new
    wait
    min.
    update
    wait
    max.
    end
    eventstart
    eventm
    in
    event
    ext /event
    end
    event
    m
    ax /event
    end
    eventstart
    kill/killed
    kill/killed
    kill/killed
    kill/killed
    kill/killed
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 17/23
    17/23

    View Slide

  69. Hardware Implementation
    Hierarchical organization
    • Conditions for stopping a structure:
    ◦ with IP:
    1. its maximum duration has elapsed.
    2. the interaction point has been triggered.
    ◦ normal:
    1. the maximum duration is infinity.
    2. its minimum duration has elapsed.
    3. its children have stopped.
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 18/23
    18/23

    View Slide

  70. Hardware Implementation
    Hierarchical organization
    • Conditions for stopping a structure:
    ◦ with IP:
    1. its maximum duration has elapsed.
    2. the interaction point has been triggered.
    ◦ normal:
    1. the maximum duration is infinity.
    2. its minimum duration has elapsed.
    3. its children have stopped.
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 18/23
    18/23

    View Slide

  71. Hardware Implementation
    Hierarchical organization
    • Conditions for stopping a structure:
    ◦ with IP:
    1. its maximum duration has elapsed.
    2. the interaction point has been triggered.
    ◦ normal:
    1. the maximum duration is infinity.
    2. its minimum duration has elapsed.
    3. its children have stopped.
    eventmin
    eventend1
    eventend2
    eventendn
    eventext
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 18/23
    18/23

    View Slide

  72. Hardware Implementation
    Code in SystemVerilog
    1 ...
    2 // FSM
    3 always_ff @(posedge clk_fsm, posedge rst_s) begin: FSM
    4 if ( rst_s ) State = idle;
    5 else begin: FSM_Sequencer
    6 unique case (State)
    7 ...
    8 wait_min: begin: wait_min_state
    9 rst_counter <= 0; enable_counter <= 1;
    10 if (kill_s || max_timeout) State = stop;
    11 else if (min_timeout) State = wait_end;
    12 else if (set_s) State = new_interval;
    13 end: wait_min_state
    14 endcase
    15 end: FSM_Sequencer
    16 end: FSM
    17
    18 // Output
    19 always_comb begin
    20 event_out = 1'b0; killed = 1'b0;
    21 unique case (State)
    22 ...
    23 wait_min : begin
    24 if (kill_s) killed = 1;
    25 else if (max_timeout) begin
    26 event_out = (clk_s & min_timeout); killed = event_out;
    27 end
    28 else event_out = 0;
    29 end
    30 endcase
    31 end
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 19/23
    19/23

    View Slide

  73. Outline
    An Overview of IS
    Hardware Implementation
    Example
    Concluding Remarks

    View Slide

  74. Example
    Texture B
    Texture A
    Structure C
    Structure D
    Texture F
    Texture G
    Texture E
    triggered at 21 s
    2 s
    1 s
    4 s
    ∆ = 6 s
    ∆ = 14 s
    ∆ = 10 s
    ∆ = 4 s
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 20/23
    20/23

    View Slide

  75. Example
    Texture B
    Texture A
    Structure C
    Structure D
    Texture F
    Texture G
    Texture E
    triggered at 21 s triggered at 31 s
    2 s
    1 s
    4 s
    ∆ = 6 s
    ∆ = 14 s
    ∆ = 10 s
    ∆ = 4 s
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 20/23
    20/23

    View Slide

  76. Example
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 21/23
    21/23

    View Slide

  77. Example
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 21/23
    21/23

    View Slide

  78. Example
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 21/23
    21/23

    View Slide

  79. Example
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 21/23
    21/23

    View Slide

  80. Example
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 21/23
    21/23

    View Slide

  81. Example
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 21/23
    21/23

    View Slide

  82. Example
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 21/23
    21/23

    View Slide

  83. Outline
    An Overview of IS
    Hardware Implementation
    Example
    Concluding Remarks

    View Slide

  84. Concluding Remarks
    Summary
    • A novel physical parallel implementation of IS.
    • Low-latency and real-time performance.
    • Reaction to events is almost instant.
    • It is not affected by the complex behavior of operating systems (e.g.,
    interrupt handling).
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 22/23
    22/23

    View Slide

  85. Concluding Remarks
    Future Work
    • Integrate multimedia processes to FPGAs (e.g., FAUST†).
    • Implement a Fast Ethernet module to provide a reliable and low-rate
    communication between a FPGA and external applications running
    on standard computers.
    • Compile the Timed Automata semantics of IS into FPGA.
    • Distributed extension.
    †Robert Trausmuth, Christian Dusek, and Yann Orlarey. Using FAUST for FPGA Programming.
    In Proceedings of the 9th International Conference on Digital Audio Effects, pages 18–20, 2006
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 23/23
    23/23

    View Slide

  86. Concluding Remarks
    Future Work
    • Integrate multimedia processes to FPGAs (e.g., FAUST).
    • Implement a Fast Ethernet module to provide a reliable and low-rate
    communication between a FPGA and external applications running
    on standard computers†.
    • Compile the Timed Automata semantics of IS into FPGA.
    • Distributed extension.
    †Rimas Aviziensis, Adrian Freed, Takahiko Suzuki, and David Wessel. Scalable Connectivity
    Processor for Computer Music Performance Systems. In Proceedings of the International Computer
    Music Conference, 2000
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 23/23
    23/23

    View Slide

  87. Concluding Remarks
    Future Work
    • Integrate multimedia processes to FPGAs (e.g., FAUST).
    • Implement a Fast Ethernet module to provide a reliable and low-rate
    communication between a FPGA and external applications running
    on standard computers.
    • Compile the Timed Automata semantics of IS into FPGA† [Done].
    • Distributed extension.
    †Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda. A framework for composition,
    verification and real-time performance of multimedia interactive scenarios. In 15th International
    Conference on Application of Concurrency to System Design (ACSD’15), 2015. To appear
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 23/23
    23/23

    View Slide

  88. Concluding Remarks
    Future Work
    • Integrate multimedia processes to FPGAs (e.g., FAUST).
    • Implement a Fast Ethernet module to provide a reliable and low-rate
    communication between a FPGA and external applications running
    on standard computers.
    • Compile the Timed Automata semantics of IS into FPGA.
    • Distributed extension†.
    †Frank Opitz, Edris Sahak, and Bernd Schwarz. Accelerating distributed computing with
    FPGAs. Xcell Journal, 79:20–27, Q2 2012
    Arias, Desainte-Catherine, and Rueda (LaBRI) Exploiting Parallelism in FPGAs for the Real-Time Interpretation of IS 23/23
    23/23

    View Slide

  89. Thank you for your attention!
    e-mail:[email protected]

    View Slide

  90. Exploiting Parallelism in FPGAs for the Real-Time
    Interpretation of Interactive Multimedia Scores*
    Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda
    Laboratoire Bordelais de Recherche en Informatique (LaBRI)
    Université de Bordeaux
    Journées d’Informatique Musicale (JIM 2015)
    Montreal - Canada, May 2015
    1

    *Supported by the ANR Project OSSIA and SCRIME

    View Slide

  91. References
    Antoine Allombert. Aspects Temporels d’un Système de Partitions
    Musicales Interactives pour la Composition et l’Exécution. PhD thesis,
    Université de Bordeaux, 2009.
    Jaime Arias, Myriam Desainte-Catherine, and Camilo Rueda. A
    framework for composition, verification and real-time performance
    of multimedia interactive scenarios. In 15th International Conference
    on Application of Concurrency to System Design (ACSD’15), 2015. To
    appear.
    Rimas Aviziensis, Adrian Freed, Takahiko Suzuki, and David Wessel.
    Scalable Connectivity Processor for Computer Music Performance
    Systems. In Proceedings of the International Computer Music
    Conference, 2000.

    View Slide

  92. References
    Jeff Bier and Jennifer Eyre. BDTI study certifies high-level synthesis
    flows for DSP-centric FPGA design. Xcell Journal, 71:12–17, Q2
    2010.
    Rahul Dubey. Introduction to Embedded System Design Using Field
    Programmable Gate Arrays. Springer London, London, 2009.
    Luc Langlois. Multirate digital signal processing for high-speed data
    converters for high-speed data converters. Xcell Journal, 73:50–53,
    Q4 2010.
    Edward A. Lee. The problem with threads. Computer, 39(5):33–42,
    May 2006.
    Frank Opitz, Edris Sahak, and Bernd Schwarz. Accelerating
    distributed computing with FPGAs. Xcell Journal, 79:20–27, Q2
    2012.

    View Slide

  93. References
    J.J. Rodriguez-Andina, M.J. Moure, and M.D. Valdes. Features, Design
    Tools, and Application Domains of FPGAs. IEEE Transactions on
    Industrial Electronics, 54(4):1810–1823, August 2007.
    Robert Trausmuth, Christian Dusek, and Yann Orlarey. Using
    FAUST for FPGA Programming. In Proceedings of the 9th International
    Conference on Digital Audio Effects, pages 18–20, 2006.
    David Wessel and Matthew Wright. Problems and prospects for
    intimate musical control of computers. In Ivan Poupyrev, Michael J.
    Lyons, Sidney Fels, and Tina Blaine, editors, New Interfaces for
    Musical Expression, NIME-01, Proceedings, Seattle, April 1-2, 2001,
    pages 11–14, 2001.

    View Slide