Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Making Machines that Make Music

Making Machines that Make Music

As presented in Euroclojure 2016.

Srihari Sriraman

October 26, 2016
Tweet

More Decks by Srihari Sriraman

Other Decks in Technology

Transcript

  1. Making Machines
    that Make Music
    Srihari Sriraman
    nilenso

    View full-size slide

  2. should we listen to some now?

    View full-size slide

  3. Why I do this
    I Sing, I do computers
    Bleeding edge research
    Dreamy ambitions
    Interesting By-products

    View full-size slide

  4. What this talk is about
    Melody
    Modelling
    Synthesis
    Generation
    Melody Modelling
    Synthesis
    Generation

    View full-size slide

  5. Melody
    ˈmɛlədi
    noun
    a sequence of single notes that is musically satisfying; a tune.

    View full-size slide

  6. Carnatic Music

    View full-size slide

  7. Carnatic Music
    Kalyani, Extempore
    MS Gopalakrishnan, Violin

    View full-size slide

  8. Khamas, Thillana
    Abhishek Raghuram

    View full-size slide

  9. Carnatic Music
    South Indian classical music
    Ragas, Gamakams
    Vocal Tradition
    Rich in compositions
    Extempore / Manodharma

    View full-size slide

  10. Tanpura
    Veena Mridangam

    View full-size slide

  11. The foundations of
    musical abstractions
    in Carnatic music

    View full-size slide

  12. Shruthi
    Tonic note
    Choice of artist
    Swarams are relative to this

    View full-size slide

  13. Laya
    Rhythm concepts
    Similar to time signatures
    A rather mature system

    View full-size slide

  14. Sa Ri Ga Ma Pa Da Sa
    Ni
    Swarams

    View full-size slide

  15. Sa Ri Ga Ma Pa Da Sa
    Ni
    S R G M P D S
    N
    R1 R2 R3
    G1 G2 G3 M1 M2
    D1 D2 D3
    N1 N2 N3
    Notation
    Pronunciation
    Variations

    View full-size slide

  16. Swarams
    The 12 semitones
    Elements of a raga
    Simples are sung
    Prescriptive notation

    View full-size slide

  17. Rāgā
    Kaapi, Extempore
    TM Krishna

    View full-size slide

  18. Raga
    Has a name
    Rule to ascend
    Rule to descend
    Not necessarily symmetric
    Not necessarily linear
    Grouped into families

    View full-size slide

  19. Raga
    Has a name
    Rule to ascend
    Rule to descend
    Not necessarily symmetric
    Not necessarily linear
    Grouped into families

    View full-size slide

  20. Demo
    of the fundamental abstractions.

    View full-size slide

  21. Tools &
    Libraries

    View full-size slide

  22. Fuzzy search
    For indic languages
    Needs to be fast
    Primary stitching mechanism
    Helps with multi-source data

    View full-size slide

  23. A quick recap
    Play the scale of a raga
    Fuzzy find a raga
    Play a phrase
    Play a phrase in the context of a raga
    Play some prescriptive notation

    View full-size slide

  24. But..
    ..that doesn’t sound like
    Carnatic music, does it?

    View full-size slide

  25. Enter Melographs
    Me
    Machine

    View full-size slide

  26. another phrase
    Me
    Machine

    View full-size slide

  27. Prescriptive vs Descriptive

    View full-size slide

  28. Sphuritam
    Orikai
    Jaaru
    Kampitam
    Sphuritam
    Nokku
    Ravai
    Kandippu
    Ullasitam
    Etra-jaru
    Iraka-jaru
    Odukkal
    Orikai
    Vali
    Kampitam

    View full-size slide

  29. Gamakams
    in SSP

    View full-size slide

  30. Gamakams
    in SSP

    View full-size slide

  31. Gaayaka
    | S, N D | N S R G |
    ((P S,,)) , ((S , S>>> S)) -((D. S. D))
    ((S , S>> S))- S R ((G<< G , ,))
    Subramanian, 2009
    Database of phrases
    Automatic Gamakam feature – guided

    View full-size slide

  32. Modelling Gamakams

    View full-size slide

  33. Me
    Machine
    Back to this…

    View full-size slide

  34. PASR
    Srikumar 2013
    Pitch, Attack, Sustain, Release
    Vector specifies the PASR vars
    for each prescriptive note

    View full-size slide

  35. Me
    Machine
    Rendering PASR…

    View full-size slide

  36. Rendering PASR…

    View full-size slide

  37. Random | Within a raga

    View full-size slide

  38. Random | Within a raga

    View full-size slide

  39. Kosha
    An Open Carnatic Music Database
    http://github.com/ssrihari/kosha

    View full-size slide

  40. Melographs
    kalyANi-MS-Subbulakshmi-nidhi_cAla_sukhamA-tyAgarAja3.mpeg.wav.pitch.frequencies-pitch-histogram
    kalyANi-Kunnakudi-R-Vaidyanathan-nidhi_cAla_sukhamA-tyAgarAja49.mpeg.wav.pitch.frequencies-pitch-histogram

    View full-size slide

  41. Pitch Histograms

    View full-size slide

  42. Pitch Histograms
    kalyANi-MS-Subbulakshmi-nidhi_cAla_sukhamA-tyAgarAja3.mpeg

    View full-size slide

  43. Pitch Histograms
    Kalyani - Vocal Kalyani - Violin
    Mohana - Mandolin
    Mohana - Vocal
    Revati - Vocal Revati - Instrumental

    View full-size slide

  44. Extract Music Information

    View full-size slide

  45. Midi Histogram
    Normalised Midi Histogram

    View full-size slide

  46. Tonic note identification
    Bellur, A., V. Ishwar, X. Serra, and H. A. Murthy (2012)
    A knowledge based signal processing approach to tonic
    identification in indian classical music.

    Bellur, A., and H. A. Murthy (2013)
    Automatic tonic identification in classical music using
    melodic characteristics and tuning of the drone.
    Srihari, S. (2016) *
    Pick the most frequent note, it mostly just works.
    * not really, no

    View full-size slide

  47. Tonic note identification

    View full-size slide

  48. Tonic note identification

    View full-size slide

  49. Swaram Histogram
    Kalyani
    S, R2, G3, M2, P, D2, N3, S.
    S., N3, D2, P, M2, G3, R2, S

    View full-size slide

  50. Kalyani
    S, R2, G3, M2, P, D2, N3, S.
    S., N3, D2, P, M2, G3, R2, S
    Revati
    S, R1, M1, P, N2, S.
    S., N2, P, M1, R1, S
    Mohana
    S, R2, G3, P, D2, S.
    S., D2, P, G3, R2, S

    View full-size slide

  51. Generation with
    weighted probabilities

    View full-size slide

  52. In comparison with random
    Random
    Single
    Swaram
    Weighted

    View full-size slide

  53. Melody insights #1
    Tonic note is prominent
    Sa, and Pa have higher and sharper peaks
    Other note peaks are blunt
    Probabilities of all swarams in a raga are not the same
    Probabilities across octaves are not the same

    View full-size slide

  54. Two swaram probabilities

    View full-size slide

  55. Two swaram probabilities
    Prominence of adjacency
    Encoded rules of Arohanam and Avarohanam

    View full-size slide

  56. Two swaram probabilities

    View full-size slide

  57. Single swaram vs Two swarams
    Two
    Swaram
    Weighted
    Single
    Swaram
    Weighted

    View full-size slide

  58. Melody insights #2
    Swarams close to each other are more melodious
    The rules of Arohanam, Avarohanam are encoded
    We begin to see gamakams
    Sometimes, the in-between is worse than either extreme

    View full-size slide

  59. Three swaram probabilities
    and more
    A simple markov chain

    View full-size slide

  60. First Order Matrix
    https://en.wikipedia.org/wiki/Markov_chain#Music
    Markov Chains in Music
    https://github.com/rm-hull/markov-chains
    Second Order Matrix

    View full-size slide

  61. Markov Chains in Music

    View full-size slide

  62. Melody insights #3
    Generic markov chains don’t really work
    LSTMs also don’t work, probably

    View full-size slide

  63. Automatic Transcription

    View full-size slide

  64. Automatic Transcription
    (:..n1 :..n1 :..m1 :..m1 :..d1 :..d1 :..d3 :.g1 :.m1 :.m1 :.m1 :.r3 :.r1
    :..n3 :..d3 :..p :..g3 :..m1 :..m2 :..g2 :..m1 :..g3 :.r1 :.s :.s :..r1 :
    ..p :.s :..n3 :.s :.s :.g1 :.s :..d3 :..n1 :..n1 :..n1 :..r1 :.s :.s :.g1
    :.r3 :.g1 :.r1 :..d3 :..d3 :..d3 :.g1 :.r1 :.s :.g1 :..r2 :..r1 :.s :.r3
    :..n3 :..d3 :..d3 :..n1 :..n1 :..n1 :..n1 :..n1 :..p :.s :..n1 :..g2 :..n
    3 :.r1 :.g3 :.g3 :.m1 :.r3 :.m1 :.g3 :.g3 :.p :.m1 :.m1 :..m1 :..g3 :.m1
    :..r2 :..r2 :..n3 :.s :.s :.g1 :.g3 :.m2 :.p :.d2 :.m2 :.m1 :.r3 :.r3 :.g
    1 :.g1 :.g1 :.r1 :..n1 :.r3 :.g3 :.s :.s :.r1 :.g1 :.r1 :..n3 :..n1 :..d3
    :..d3 :..n1 :..d3 :..n1 :..n1 :..n1 :..r1 :.s :..n1)

    View full-size slide

  65. Raga Identification

    View full-size slide

  66. Goodness of fit test

    View full-size slide

  67. :base mohanam-base
    :samples mohanam-files
    (12.39
    3.84
    11.14
    6.46
    9.88
    7.02
    9.41
    12.61
    13.22
    1.58)
    :base mohanam-base
    :samples kalyani-files
    (10.95
    28.66
    25.61
    15.26
    27.32
    21.53
    16.42
    18.58
    24.80
    23.80)
    :base mohanam-base
    :samples revati-files
    (46.56
    57.19
    65.69
    55.21
    38.61
    78.10
    56.27
    42.99
    70.92
    58.39)

    View full-size slide

  68. Raga Identification
    Revati sample
    vs
    Revati base
    Mohana sample
    vs
    Revati base

    View full-size slide

  69. What next
    Model insights as melodic abstractions
    Use synthesis models with generative music
    Experiment with Rhythm
    Synthesise Human Voice
    Deep learning (Recurrent variational auto encoders)

    View full-size slide

  70. Is this music though?

    View full-size slide

  71. Behag
    Dasarapada
    Abhishek Raghuram

    View full-size slide

  72. Making Machines
    that Make Music
    Srihari Sriraman
    nilenso

    View full-size slide