Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Making Machines that Make Music

Making Machines that Make Music

As presented in Euroclojure 2016.

Srihari Sriraman

October 26, 2016
Tweet

More Decks by Srihari Sriraman

Other Decks in Technology

Transcript

  1. Making Machines
    that Make Music
    Srihari Sriraman
    nilenso

    View Slide

  2. should we listen to some now?

    View Slide

  3. Why I do this
    I Sing, I do computers
    Bleeding edge research
    Dreamy ambitions
    Interesting By-products

    View Slide

  4. What this talk is about
    Melody
    Modelling
    Synthesis
    Generation
    Melody Modelling
    Synthesis
    Generation

    View Slide

  5. Melody
    ˈmɛlədi
    noun
    a sequence of single notes that is musically satisfying; a tune.

    View Slide

  6. Carnatic Music

    View Slide

  7. Carnatic Music
    Kalyani, Extempore
    MS Gopalakrishnan, Violin

    View Slide

  8. Khamas, Thillana
    Abhishek Raghuram

    View Slide

  9. Carnatic Music
    South Indian classical music
    Ragas, Gamakams
    Vocal Tradition
    Rich in compositions
    Extempore / Manodharma

    View Slide

  10. Tanpura
    Veena Mridangam

    View Slide

  11. The foundations of
    musical abstractions
    in Carnatic music

    View Slide

  12. Shruthi
    Tonic note
    Choice of artist
    Swarams are relative to this

    View Slide

  13. Laya
    Rhythm concepts
    Similar to time signatures
    A rather mature system

    View Slide

  14. Sa Ri Ga Ma Pa Da Sa
    Ni
    Swarams

    View Slide

  15. Sa Ri Ga Ma Pa Da Sa
    Ni
    S R G M P D S
    N
    R1 R2 R3
    G1 G2 G3 M1 M2
    D1 D2 D3
    N1 N2 N3
    Notation
    Pronunciation
    Variations

    View Slide

  16. Swarams
    The 12 semitones
    Elements of a raga
    Simples are sung
    Prescriptive notation

    View Slide

  17. Rāgā
    Kaapi, Extempore
    TM Krishna

    View Slide

  18. Raga
    Has a name
    Rule to ascend
    Rule to descend
    Not necessarily symmetric
    Not necessarily linear
    Grouped into families

    View Slide

  19. Raga
    Has a name
    Rule to ascend
    Rule to descend
    Not necessarily symmetric
    Not necessarily linear
    Grouped into families

    View Slide

  20. Demo
    of the fundamental abstractions.

    View Slide

  21. View Slide

  22. Tools &
    Libraries

    View Slide

  23. Fuzzy search
    For indic languages
    Needs to be fast
    Primary stitching mechanism
    Helps with multi-source data

    View Slide

  24. A quick recap
    Play the scale of a raga
    Fuzzy find a raga
    Play a phrase
    Play a phrase in the context of a raga
    Play some prescriptive notation

    View Slide

  25. But..
    ..that doesn’t sound like
    Carnatic music, does it?

    View Slide

  26. Synthesis

    View Slide

  27. Enter Melographs
    Me
    Machine

    View Slide

  28. another phrase
    Me
    Machine

    View Slide

  29. Prescriptive vs Descriptive

    View Slide

  30. Gamakams

    View Slide

  31. Sphuritam
    Orikai
    Jaaru
    Kampitam
    Sphuritam
    Nokku
    Ravai
    Kandippu
    Ullasitam
    Etra-jaru
    Iraka-jaru
    Odukkal
    Orikai
    Vali
    Kampitam

    View Slide

  32. Gamakams
    in SSP

    View Slide

  33. Gamakams
    in SSP

    View Slide

  34. Gaayaka
    | S, N D | N S R G |
    ((P S,,)) , ((S , S>>> S)) -((D. S. D))
    ((S , S>> S))- S R ((G<< G , ,))
    Subramanian, 2009
    Database of phrases
    Automatic Gamakam feature – guided

    View Slide

  35. Modelling Gamakams

    View Slide

  36. View Slide

  37. Me
    Machine
    Back to this…

    View Slide

  38. PASR
    Srikumar 2013
    Pitch, Attack, Sustain, Release
    Vector specifies the PASR vars
    for each prescriptive note

    View Slide

  39. Me
    Machine
    Rendering PASR…

    View Slide

  40. Rendering PASR…

    View Slide

  41. Generation

    View Slide

  42. Random | Within a raga

    View Slide

  43. Random | Within a raga

    View Slide

  44. Get data

    View Slide

  45. Kosha
    An Open Carnatic Music Database
    http://github.com/ssrihari/kosha

    View Slide

  46. View Slide

  47. View Slide

  48. View Slide

  49. Study data

    View Slide

  50. View Slide

  51. Melographs

    View Slide

  52. Melographs
    kalyANi-MS-Subbulakshmi-nidhi_cAla_sukhamA-tyAgarAja3.mpeg.wav.pitch.frequencies-pitch-histogram
    kalyANi-Kunnakudi-R-Vaidyanathan-nidhi_cAla_sukhamA-tyAgarAja49.mpeg.wav.pitch.frequencies-pitch-histogram

    View Slide

  53. Pitch Histograms

    View Slide

  54. Pitch Histograms
    kalyANi-MS-Subbulakshmi-nidhi_cAla_sukhamA-tyAgarAja3.mpeg

    View Slide

  55. Pitch Histograms
    Kalyani - Vocal Kalyani - Violin
    Mohana - Mandolin
    Mohana - Vocal
    Revati - Vocal Revati - Instrumental

    View Slide

  56. Extract Music Information

    View Slide

  57. Midi Histogram
    Normalised Midi Histogram

    View Slide

  58. Tonic note identification
    Bellur, A., V. Ishwar, X. Serra, and H. A. Murthy (2012)
    A knowledge based signal processing approach to tonic
    identification in indian classical music.

    Bellur, A., and H. A. Murthy (2013)
    Automatic tonic identification in classical music using
    melodic characteristics and tuning of the drone.
    Srihari, S. (2016) *
    Pick the most frequent note, it mostly just works.
    * not really, no

    View Slide

  59. Tonic note identification

    View Slide

  60. Tonic note identification

    View Slide

  61. Swaram Histogram
    Kalyani
    S, R2, G3, M2, P, D2, N3, S.
    S., N3, D2, P, M2, G3, R2, S

    View Slide

  62. Kalyani
    S, R2, G3, M2, P, D2, N3, S.
    S., N3, D2, P, M2, G3, R2, S
    Revati
    S, R1, M1, P, N2, S.
    S., N2, P, M1, R1, S
    Mohana
    S, R2, G3, P, D2, S.
    S., D2, P, G3, R2, S

    View Slide

  63. Generation with
    weighted probabilities

    View Slide

  64. View Slide

  65. In comparison with random
    Random
    Single
    Swaram
    Weighted

    View Slide

  66. Melody insights #1
    Tonic note is prominent
    Sa, and Pa have higher and sharper peaks
    Other note peaks are blunt
    Probabilities of all swarams in a raga are not the same
    Probabilities across octaves are not the same

    View Slide

  67. Two swaram probabilities

    View Slide

  68. Two swaram probabilities
    Prominence of adjacency
    Encoded rules of Arohanam and Avarohanam

    View Slide

  69. Two swaram probabilities

    View Slide

  70. Single swaram vs Two swarams
    Two
    Swaram
    Weighted
    Single
    Swaram
    Weighted

    View Slide

  71. Melody insights #2
    Swarams close to each other are more melodious
    The rules of Arohanam, Avarohanam are encoded
    We begin to see gamakams
    Sometimes, the in-between is worse than either extreme

    View Slide

  72. Three swaram probabilities
    and more
    A simple markov chain

    View Slide

  73. First Order Matrix
    https://en.wikipedia.org/wiki/Markov_chain#Music
    Markov Chains in Music
    https://github.com/rm-hull/markov-chains
    Second Order Matrix

    View Slide

  74. Markov Chains in Music

    View Slide

  75. View Slide

  76. Melody insights #3
    Generic markov chains don’t really work
    LSTMs also don’t work, probably

    View Slide

  77. By-products

    View Slide

  78. Automatic Transcription

    View Slide

  79. Automatic Transcription
    (:..n1 :..n1 :..m1 :..m1 :..d1 :..d1 :..d3 :.g1 :.m1 :.m1 :.m1 :.r3 :.r1
    :..n3 :..d3 :..p :..g3 :..m1 :..m2 :..g2 :..m1 :..g3 :.r1 :.s :.s :..r1 :
    ..p :.s :..n3 :.s :.s :.g1 :.s :..d3 :..n1 :..n1 :..n1 :..r1 :.s :.s :.g1
    :.r3 :.g1 :.r1 :..d3 :..d3 :..d3 :.g1 :.r1 :.s :.g1 :..r2 :..r1 :.s :.r3
    :..n3 :..d3 :..d3 :..n1 :..n1 :..n1 :..n1 :..n1 :..p :.s :..n1 :..g2 :..n
    3 :.r1 :.g3 :.g3 :.m1 :.r3 :.m1 :.g3 :.g3 :.p :.m1 :.m1 :..m1 :..g3 :.m1
    :..r2 :..r2 :..n3 :.s :.s :.g1 :.g3 :.m2 :.p :.d2 :.m2 :.m1 :.r3 :.r3 :.g
    1 :.g1 :.g1 :.r1 :..n1 :.r3 :.g3 :.s :.s :.r1 :.g1 :.r1 :..n3 :..n1 :..d3
    :..d3 :..n1 :..d3 :..n1 :..n1 :..n1 :..r1 :.s :..n1)

    View Slide

  80. Raga Identification

    View Slide

  81. Goodness of fit test

    View Slide

  82. :base mohanam-base
    :samples mohanam-files
    (12.39
    3.84
    11.14
    6.46
    9.88
    7.02
    9.41
    12.61
    13.22
    1.58)
    :base mohanam-base
    :samples kalyani-files
    (10.95
    28.66
    25.61
    15.26
    27.32
    21.53
    16.42
    18.58
    24.80
    23.80)
    :base mohanam-base
    :samples revati-files
    (46.56
    57.19
    65.69
    55.21
    38.61
    78.10
    56.27
    42.99
    70.92
    58.39)

    View Slide

  83. Raga Identification
    Revati sample
    vs
    Revati base
    Mohana sample
    vs
    Revati base

    View Slide

  84. What next
    Model insights as melodic abstractions
    Use synthesis models with generative music
    Experiment with Rhythm
    Synthesise Human Voice
    Deep learning (Recurrent variational auto encoders)

    View Slide

  85. Is this music though?

    View Slide

  86. Behag
    Dasarapada
    Abhishek Raghuram

    View Slide

  87. Making Machines
    that Make Music
    Srihari Sriraman
    nilenso

    View Slide