Making Machines that Make Music

As presented in Euroclojure 2016.

Transcript

1. Making Machines that Make Music Srihari Sriraman nilenso
2. should we listen to some now?
3. Why I do this I Sing, I do computers Bleeding edge research Dreamy ambitions Interesting By-products
4. What this talk is about Melody Modelling Synthesis Generation Melody Modelling Synthesis Generation
5. Melody ˈmɛlədi noun a sequence of single notes that is musically satisfying; a tune.
6. Carnatic Music
7. Carnatic Music Kalyani, Extempore MS Gopalakrishnan, Violin
8. Khamas, Thillana Abhishek Raghuram
9. Carnatic Music South Indian classical music Ragas, Gamakams Vocal Tradition Rich in compositions Extempore / Manodharma
10. Tanpura Veena Mridangam
11. The foundations of musical abstractions in Carnatic music
12. Shruthi Tonic note Choice of artist Swarams are relative to this
13. Laya Rhythm concepts Similar to time signatures A rather mature system
14. Sa Ri Ga Ma Pa Da Sa Ni Swarams
15. Sa Ri Ga Ma Pa Da Sa Ni S R G M P D S N R1 R2 R3 G1 G2 G3 M1 M2 D1 D2 D3 N1 N2 N3 Notation Pronunciation Variations
16. Swarams The 12 semitones Elements of a raga Simples are sung Prescriptive notation
17. Rāgā Kaapi, Extempore TM Krishna
18. Raga Has a name Rule to ascend Rule to descend Not necessarily symmetric Not necessarily linear Grouped into families
19. Raga Has a name Rule to ascend Rule to descend Not necessarily symmetric Not necessarily linear Grouped into families
20. Demo of the fundamental abstractions.
22. Tools & Libraries
23. Fuzzy search For indic languages Needs to be fast Primary stitching mechanism Helps with multi-source data
24. A quick recap Play the scale of a raga Fuzzy find a raga Play a phrase Play a phrase in the context of a raga Play some prescriptive notation
25. But.. ..that doesn’t sound like Carnatic music, does it?
26. Synthesis
27. Enter Melographs Me Machine
28. another phrase Me Machine
29. Prescriptive vs Descriptive
30. Gamakams
31. Sphuritam Orikai Jaaru Kampitam Sphuritam Nokku Ravai Kandippu Ullasitam Etra-jaru Iraka-jaru Odukkal Orikai Vali Kampitam
32. Gamakams in SSP
33. Gamakams in SSP
34. Gaayaka | S, N D | N S R G | ((P S,,)) , ((S , S>>> S)) -((D. S. D)) ((S , S>> S))- S R ((G<< G , ,)) Subramanian, 2009 Database of phrases Automatic Gamakam feature – guided
35. Modelling Gamakams
37. Me Machine Back to this…
38. PASR Srikumar 2013 Pitch, Attack, Sustain, Release Vector specifies the PASR vars for each prescriptive note
39. Me Machine Rendering PASR…
40. Rendering PASR…
41. Generation
42. Random | Within a raga
43. Random | Within a raga
44. Get data
45. Kosha An Open Carnatic Music Database http://github.com/ssrihari/kosha
49. Study data
51. Melographs
52. Melographs kalyANi-MS-Subbulakshmi-nidhi_cAla_sukhamA-tyAgarAja3.mpeg.wav.pitch.frequencies-pitch-histogram kalyANi-Kunnakudi-R-Vaidyanathan-nidhi_cAla_sukhamA-tyAgarAja49.mpeg.wav.pitch.frequencies-pitch-histogram
53. Pitch Histograms
54. Pitch Histograms kalyANi-MS-Subbulakshmi-nidhi_cAla_sukhamA-tyAgarAja3.mpeg
55. Pitch Histograms Kalyani - Vocal Kalyani - Violin Mohana - Mandolin Mohana - Vocal Revati - Vocal Revati - Instrumental
56. Extract Music Information
57. Midi Histogram Normalised Midi Histogram
58. Tonic note identification Bellur, A., V. Ishwar, X. Serra, and H. A. Murthy (2012) A knowledge based signal processing approach to tonic identification in indian classical music.
 Bellur, A., and H. A. Murthy (2013) Automatic tonic identification in classical music using melodic characteristics and tuning of the drone. Srihari, S. (2016) * Pick the most frequent note, it mostly just works. * not really, no
59. Tonic note identification
60. Tonic note identification
61. Swaram Histogram Kalyani S, R2, G3, M2, P, D2, N3, S. S., N3, D2, P, M2, G3, R2, S
62. Kalyani S, R2, G3, M2, P, D2, N3, S. S., N3, D2, P, M2, G3, R2, S Revati S, R1, M1, P, N2, S. S., N2, P, M1, R1, S Mohana S, R2, G3, P, D2, S. S., D2, P, G3, R2, S
63. Generation with weighted probabilities
65. In comparison with random Random Single Swaram Weighted
66. Melody insights #1 Tonic note is prominent Sa, and Pa have higher and sharper peaks Other note peaks are blunt Probabilities of all swarams in a raga are not the same Probabilities across octaves are not the same
67. Two swaram probabilities
68. Two swaram probabilities Prominence of adjacency Encoded rules of Arohanam and Avarohanam
69. Two swaram probabilities
70. Single swaram vs Two swarams Two Swaram Weighted Single Swaram Weighted
71. Melody insights #2 Swarams close to each other are more melodious The rules of Arohanam, Avarohanam are encoded We begin to see gamakams Sometimes, the in-between is worse than either extreme
72. Three swaram probabilities and more A simple markov chain
73. First Order Matrix https://en.wikipedia.org/wiki/Markov_chain#Music Markov Chains in Music https://github.com/rm-hull/markov-chains Second Order Matrix
74. Markov Chains in Music
76. Melody insights #3 Generic markov chains don’t really work LSTMs also don’t work, probably
77. By-products
78. Automatic Transcription
79. Automatic Transcription (:..n1 :..n1 :..m1 :..m1 :..d1 :..d1 :..d3 :.g1 :.m1 :.m1 :.m1 :.r3 :.r1 :..n3 :..d3 :..p :..g3 :..m1 :..m2 :..g2 :..m1 :..g3 :.r1 :.s :.s :..r1 : ..p :.s :..n3 :.s :.s :.g1 :.s :..d3 :..n1 :..n1 :..n1 :..r1 :.s :.s :.g1 :.r3 :.g1 :.r1 :..d3 :..d3 :..d3 :.g1 :.r1 :.s :.g1 :..r2 :..r1 :.s :.r3 :..n3 :..d3 :..d3 :..n1 :..n1 :..n1 :..n1 :..n1 :..p :.s :..n1 :..g2 :..n 3 :.r1 :.g3 :.g3 :.m1 :.r3 :.m1 :.g3 :.g3 :.p :.m1 :.m1 :..m1 :..g3 :.m1 :..r2 :..r2 :..n3 :.s :.s :.g1 :.g3 :.m2 :.p :.d2 :.m2 :.m1 :.r3 :.r3 :.g 1 :.g1 :.g1 :.r1 :..n1 :.r3 :.g3 :.s :.s :.r1 :.g1 :.r1 :..n3 :..n1 :..d3 :..d3 :..n1 :..d3 :..n1 :..n1 :..n1 :..r1 :.s :..n1)
80. Raga Identification
81. Goodness of fit test
82. :base mohanam-base :samples mohanam-files (12.39 3.84 11.14 6.46 9.88 7.02 9.41 12.61 13.22 1.58) :base mohanam-base :samples kalyani-files (10.95 28.66 25.61 15.26 27.32 21.53 16.42 18.58 24.80 23.80) :base mohanam-base :samples revati-files (46.56 57.19 65.69 55.21 38.61 78.10 56.27 42.99 70.92 58.39)
83. Raga Identification Revati sample vs Revati base Mohana sample vs Revati base
84. What next Model insights as melodic abstractions Use synthesis models with generative music Experiment with Rhythm Synthesise Human Voice Deep learning (Recurrent variational auto encoders)
85. Is this music though?
86. Behag Dasarapada Abhishek Raghuram
87. Making Machines that Make Music Srihari Sriraman nilenso