Making Machines that Make Music

As presented in Euroclojure 2016.

Transcript

1. Making Machines
   that Make Music
   Srihari Sriraman
   nilenso
   

   View Slide

2. should we listen to some now?
   

   View Slide

3. Why I do this
   I Sing, I do computers
   Bleeding edge research
   Dreamy ambitions
   Interesting By-products
   

   View Slide

4. What this talk is about
   Melody
   Modelling
   Synthesis
   Generation
   Melody Modelling
   Synthesis
   Generation
   

   View Slide

5. Melody
   ˈmɛlədi
   noun
   a sequence of single notes that is musically satisfying; a tune.
   

   View Slide

6. Carnatic Music
   

   View Slide

7. Carnatic Music
   Kalyani, Extempore
   MS Gopalakrishnan, Violin
   

   View Slide

8. Khamas, Thillana
   Abhishek Raghuram
   

   View Slide

9. Carnatic Music
   South Indian classical music
   Ragas, Gamakams
   Vocal Tradition
   Rich in compositions
   Extempore / Manodharma
   

   View Slide

10. Tanpura
    Veena Mridangam
    

    View Slide

11. The foundations of
    musical abstractions
    in Carnatic music
    

    View Slide

12. Shruthi
    Tonic note
    Choice of artist
    Swarams are relative to this
    

    View Slide

13. Laya
    Rhythm concepts
    Similar to time signatures
    A rather mature system
    

    View Slide

14. Sa Ri Ga Ma Pa Da Sa
    Ni
    Swarams
    

    View Slide

15. Sa Ri Ga Ma Pa Da Sa
    Ni
    S R G M P D S
    N
    R1 R2 R3
    G1 G2 G3 M1 M2
    D1 D2 D3
    N1 N2 N3
    Notation
    Pronunciation
    Variations
    

    View Slide

16. Swarams
    The 12 semitones
    Elements of a raga
    Simples are sung
    Prescriptive notation
    

    View Slide

17. Rāgā
    Kaapi, Extempore
    TM Krishna
    

    View Slide

18. Raga
    Has a name
    Rule to ascend
    Rule to descend
    Not necessarily symmetric
    Not necessarily linear
    Grouped into families
    

    View Slide

19. Raga
    Has a name
    Rule to ascend
    Rule to descend
    Not necessarily symmetric
    Not necessarily linear
    Grouped into families
    

    View Slide

20. Demo
    of the fundamental abstractions.
    

    View Slide

21. View Slide

22. Tools &
    Libraries
    

    View Slide

23. Fuzzy search
    For indic languages
    Needs to be fast
    Primary stitching mechanism
    Helps with multi-source data
    

    View Slide

24. A quick recap
    Play the scale of a raga
    Fuzzy find a raga
    Play a phrase
    Play a phrase in the context of a raga
    Play some prescriptive notation
    

    View Slide

25. But..
    ..that doesn’t sound like
    Carnatic music, does it?
    

    View Slide

26. Synthesis
    

    View Slide

27. Enter Melographs
    Me
    Machine
    

    View Slide

28. another phrase
    Me
    Machine
    

    View Slide

29. Prescriptive vs Descriptive
    

    View Slide

30. Gamakams
    

    View Slide

31. Sphuritam
    Orikai
    Jaaru
    Kampitam
    Sphuritam
    Nokku
    Ravai
    Kandippu
    Ullasitam
    Etra-jaru
    Iraka-jaru
    Odukkal
    Orikai
    Vali
    Kampitam
    

    View Slide

32. Gamakams
    in SSP
    

    View Slide

33. Gamakams
    in SSP
    

    View Slide

34. Gaayaka
    | S, N D | N S R G |
    ((P S,,)) , ((S , S>>> S)) -((D. S. D))
    ((S , S>> S))- S R ((G<< G , ,))
    Subramanian, 2009
    Database of phrases
    Automatic Gamakam feature – guided
    

    View Slide

35. Modelling Gamakams
    

    View Slide

36. View Slide

37. Me
    Machine
    Back to this…
    

    View Slide

38. PASR
    Srikumar 2013
    Pitch, Attack, Sustain, Release
    Vector specifies the PASR vars
    for each prescriptive note
    

    View Slide

39. Me
    Machine
    Rendering PASR…
    

    View Slide

40. Rendering PASR…
    

    View Slide

41. Generation
    

    View Slide

42. Random | Within a raga
    

    View Slide

43. Random | Within a raga
    

    View Slide

44. Get data
    

    View Slide

45. Kosha
    An Open Carnatic Music Database
    http://github.com/ssrihari/kosha
    

    View Slide

46. View Slide

47. View Slide

48. View Slide

49. Study data
    

    View Slide

50. View Slide

51. Melographs
    

    View Slide

52. Melographs
    kalyANi-MS-Subbulakshmi-nidhi_cAla_sukhamA-tyAgarAja3.mpeg.wav.pitch.frequencies-pitch-histogram
    kalyANi-Kunnakudi-R-Vaidyanathan-nidhi_cAla_sukhamA-tyAgarAja49.mpeg.wav.pitch.frequencies-pitch-histogram
    

    View Slide

53. Pitch Histograms
    

    View Slide

54. Pitch Histograms
    kalyANi-MS-Subbulakshmi-nidhi_cAla_sukhamA-tyAgarAja3.mpeg
    

    View Slide

55. Pitch Histograms
    Kalyani - Vocal Kalyani - Violin
    Mohana - Mandolin
    Mohana - Vocal
    Revati - Vocal Revati - Instrumental
    

    View Slide

56. Extract Music Information
    

    View Slide

57. Midi Histogram
    Normalised Midi Histogram
    

    View Slide

58. Tonic note identification
    Bellur, A., V. Ishwar, X. Serra, and H. A. Murthy (2012)
    A knowledge based signal processing approach to tonic
    identification in indian classical music.

    Bellur, A., and H. A. Murthy (2013)
    Automatic tonic identification in classical music using
    melodic characteristics and tuning of the drone.
    Srihari, S. (2016) *
    Pick the most frequent note, it mostly just works.
    * not really, no
    

    View Slide

59. Tonic note identification
    

    View Slide

60. Tonic note identification
    

    View Slide

61. Swaram Histogram
    Kalyani
    S, R2, G3, M2, P, D2, N3, S.
    S., N3, D2, P, M2, G3, R2, S
    

    View Slide

62. Kalyani
    S, R2, G3, M2, P, D2, N3, S.
    S., N3, D2, P, M2, G3, R2, S
    Revati
    S, R1, M1, P, N2, S.
    S., N2, P, M1, R1, S
    Mohana
    S, R2, G3, P, D2, S.
    S., D2, P, G3, R2, S
    

    View Slide

63. Generation with
    weighted probabilities
    

    View Slide

64. View Slide

65. In comparison with random
    Random
    Single
    Swaram
    Weighted
    

    View Slide

66. Melody insights #1
    Tonic note is prominent
    Sa, and Pa have higher and sharper peaks
    Other note peaks are blunt
    Probabilities of all swarams in a raga are not the same
    Probabilities across octaves are not the same
    

    View Slide

67. Two swaram probabilities
    

    View Slide

68. Two swaram probabilities
    Prominence of adjacency
    Encoded rules of Arohanam and Avarohanam
    

    View Slide

69. Two swaram probabilities
    

    View Slide

70. Single swaram vs Two swarams
    Two
    Swaram
    Weighted
    Single
    Swaram
    Weighted
    

    View Slide

71. Melody insights #2
    Swarams close to each other are more melodious
    The rules of Arohanam, Avarohanam are encoded
    We begin to see gamakams
    Sometimes, the in-between is worse than either extreme
    

    View Slide

72. Three swaram probabilities
    and more
    A simple markov chain
    

    View Slide

73. First Order Matrix
    https://en.wikipedia.org/wiki/Markov_chain#Music
    Markov Chains in Music
    https://github.com/rm-hull/markov-chains
    Second Order Matrix
    

    View Slide

74. Markov Chains in Music
    

    View Slide

75. View Slide

76. Melody insights #3
    Generic markov chains don’t really work
    LSTMs also don’t work, probably
    

    View Slide

77. By-products
    

    View Slide

78. Automatic Transcription
    

    View Slide

79. Automatic Transcription
    (:..n1 :..n1 :..m1 :..m1 :..d1 :..d1 :..d3 :.g1 :.m1 :.m1 :.m1 :.r3 :.r1
    :..n3 :..d3 :..p :..g3 :..m1 :..m2 :..g2 :..m1 :..g3 :.r1 :.s :.s :..r1 :
    ..p :.s :..n3 :.s :.s :.g1 :.s :..d3 :..n1 :..n1 :..n1 :..r1 :.s :.s :.g1
    :.r3 :.g1 :.r1 :..d3 :..d3 :..d3 :.g1 :.r1 :.s :.g1 :..r2 :..r1 :.s :.r3
    :..n3 :..d3 :..d3 :..n1 :..n1 :..n1 :..n1 :..n1 :..p :.s :..n1 :..g2 :..n
    3 :.r1 :.g3 :.g3 :.m1 :.r3 :.m1 :.g3 :.g3 :.p :.m1 :.m1 :..m1 :..g3 :.m1
    :..r2 :..r2 :..n3 :.s :.s :.g1 :.g3 :.m2 :.p :.d2 :.m2 :.m1 :.r3 :.r3 :.g
    1 :.g1 :.g1 :.r1 :..n1 :.r3 :.g3 :.s :.s :.r1 :.g1 :.r1 :..n3 :..n1 :..d3
    :..d3 :..n1 :..d3 :..n1 :..n1 :..n1 :..r1 :.s :..n1)
    

    View Slide

80. Raga Identification
    

    View Slide

81. Goodness of fit test
    

    View Slide

82. :base mohanam-base
    :samples mohanam-files
    (12.39
    3.84
    11.14
    6.46
    9.88
    7.02
    9.41
    12.61
    13.22
    1.58)
    :base mohanam-base
    :samples kalyani-files
    (10.95
    28.66
    25.61
    15.26
    27.32
    21.53
    16.42
    18.58
    24.80
    23.80)
    :base mohanam-base
    :samples revati-files
    (46.56
    57.19
    65.69
    55.21
    38.61
    78.10
    56.27
    42.99
    70.92
    58.39)
    

    View Slide

83. Raga Identification
    Revati sample
    vs
    Revati base
    Mohana sample
    vs
    Revati base
    

    View Slide

84. What next
    Model insights as melodic abstractions
    Use synthesis models with generative music
    Experiment with Rhythm
    Synthesise Human Voice
    Deep learning (Recurrent variational auto encoders)
    

    View Slide

85. Is this music though?
    

    View Slide

86. Behag
    Dasarapada
    Abhishek Raghuram
    

    View Slide

87. Making Machines
    that Make Music
    Srihari Sriraman
    nilenso
    

    View Slide