Slide 1

Slide 1 text

Making Machines that Make Music Srihari Sriraman nilenso

Slide 2

Slide 2 text

should we listen to some now?

Slide 3

Slide 3 text

Why I do this I Sing, I do computers Bleeding edge research Dreamy ambitions Interesting By-products

Slide 4

Slide 4 text

What this talk is about Melody Modelling Synthesis Generation Melody Modelling Synthesis Generation

Slide 5

Slide 5 text

Melody ˈmɛlədi noun a sequence of single notes that is musically satisfying; a tune.

Slide 6

Slide 6 text

Carnatic Music

Slide 7

Slide 7 text

Carnatic Music Kalyani, Extempore MS Gopalakrishnan, Violin

Slide 8

Slide 8 text

Khamas, Thillana Abhishek Raghuram

Slide 9

Slide 9 text

Carnatic Music South Indian classical music Ragas, Gamakams Vocal Tradition Rich in compositions Extempore / Manodharma

Slide 10

Slide 10 text

Tanpura Veena Mridangam

Slide 11

Slide 11 text

The foundations of musical abstractions in Carnatic music

Slide 12

Slide 12 text

Shruthi Tonic note Choice of artist Swarams are relative to this

Slide 13

Slide 13 text

Laya Rhythm concepts Similar to time signatures A rather mature system

Slide 14

Slide 14 text

Sa Ri Ga Ma Pa Da Sa Ni Swarams

Slide 15

Slide 15 text

Sa Ri Ga Ma Pa Da Sa Ni S R G M P D S N R1 R2 R3 G1 G2 G3 M1 M2 D1 D2 D3 N1 N2 N3 Notation Pronunciation Variations

Slide 16

Slide 16 text

Swarams The 12 semitones Elements of a raga Simples are sung Prescriptive notation

Slide 17

Slide 17 text

Rāgā Kaapi, Extempore TM Krishna

Slide 18

Slide 18 text

Raga Has a name Rule to ascend Rule to descend Not necessarily symmetric Not necessarily linear Grouped into families

Slide 19

Slide 19 text

Raga Has a name Rule to ascend Rule to descend Not necessarily symmetric Not necessarily linear Grouped into families

Slide 20

Slide 20 text

Demo of the fundamental abstractions.

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

Tools & Libraries

Slide 23

Slide 23 text

Fuzzy search For indic languages Needs to be fast Primary stitching mechanism Helps with multi-source data

Slide 24

Slide 24 text

A quick recap Play the scale of a raga Fuzzy find a raga Play a phrase Play a phrase in the context of a raga Play some prescriptive notation

Slide 25

Slide 25 text

But.. ..that doesn’t sound like Carnatic music, does it?

Slide 26

Slide 26 text

Synthesis

Slide 27

Slide 27 text

Enter Melographs Me Machine

Slide 28

Slide 28 text

another phrase Me Machine

Slide 29

Slide 29 text

Prescriptive vs Descriptive

Slide 30

Slide 30 text

Gamakams

Slide 31

Slide 31 text

Sphuritam Orikai Jaaru Kampitam Sphuritam Nokku Ravai Kandippu Ullasitam Etra-jaru Iraka-jaru Odukkal Orikai Vali Kampitam

Slide 32

Slide 32 text

Gamakams in SSP

Slide 33

Slide 33 text

Gamakams in SSP

Slide 34

Slide 34 text

Gaayaka | S, N D | N S R G | ((P S,,)) , ((S , S>>> S)) -((D. S. D)) ((S , S>> S))- S R ((G<< G , ,)) Subramanian, 2009 Database of phrases Automatic Gamakam feature – guided

Slide 35

Slide 35 text

Modelling Gamakams

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

Me Machine Back to this…

Slide 38

Slide 38 text

PASR Srikumar 2013 Pitch, Attack, Sustain, Release Vector specifies the PASR vars for each prescriptive note

Slide 39

Slide 39 text

Me Machine Rendering PASR…

Slide 40

Slide 40 text

Rendering PASR…

Slide 41

Slide 41 text

Generation

Slide 42

Slide 42 text

Random | Within a raga

Slide 43

Slide 43 text

Random | Within a raga

Slide 44

Slide 44 text

Get data

Slide 45

Slide 45 text

Kosha An Open Carnatic Music Database http://github.com/ssrihari/kosha

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

Study data

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

Melographs

Slide 52

Slide 52 text

Melographs kalyANi-MS-Subbulakshmi-nidhi_cAla_sukhamA-tyAgarAja3.mpeg.wav.pitch.frequencies-pitch-histogram kalyANi-Kunnakudi-R-Vaidyanathan-nidhi_cAla_sukhamA-tyAgarAja49.mpeg.wav.pitch.frequencies-pitch-histogram

Slide 53

Slide 53 text

Pitch Histograms

Slide 54

Slide 54 text

Pitch Histograms kalyANi-MS-Subbulakshmi-nidhi_cAla_sukhamA-tyAgarAja3.mpeg

Slide 55

Slide 55 text

Pitch Histograms Kalyani - Vocal Kalyani - Violin Mohana - Mandolin Mohana - Vocal Revati - Vocal Revati - Instrumental

Slide 56

Slide 56 text

Extract Music Information

Slide 57

Slide 57 text

Midi Histogram Normalised Midi Histogram

Slide 58

Slide 58 text

Tonic note identification Bellur, A., V. Ishwar, X. Serra, and H. A. Murthy (2012) A knowledge based signal processing approach to tonic identification in indian classical music.
 Bellur, A., and H. A. Murthy (2013) Automatic tonic identification in classical music using melodic characteristics and tuning of the drone. Srihari, S. (2016) * Pick the most frequent note, it mostly just works. * not really, no

Slide 59

Slide 59 text

Tonic note identification

Slide 60

Slide 60 text

Tonic note identification

Slide 61

Slide 61 text

Swaram Histogram Kalyani S, R2, G3, M2, P, D2, N3, S. S., N3, D2, P, M2, G3, R2, S

Slide 62

Slide 62 text

Kalyani S, R2, G3, M2, P, D2, N3, S. S., N3, D2, P, M2, G3, R2, S Revati S, R1, M1, P, N2, S. S., N2, P, M1, R1, S Mohana S, R2, G3, P, D2, S. S., D2, P, G3, R2, S

Slide 63

Slide 63 text

Generation with weighted probabilities

Slide 64

Slide 64 text

No content

Slide 65

Slide 65 text

In comparison with random Random Single Swaram Weighted

Slide 66

Slide 66 text

Melody insights #1 Tonic note is prominent Sa, and Pa have higher and sharper peaks Other note peaks are blunt Probabilities of all swarams in a raga are not the same Probabilities across octaves are not the same

Slide 67

Slide 67 text

Two swaram probabilities

Slide 68

Slide 68 text

Two swaram probabilities Prominence of adjacency Encoded rules of Arohanam and Avarohanam

Slide 69

Slide 69 text

Two swaram probabilities

Slide 70

Slide 70 text

Single swaram vs Two swarams Two Swaram Weighted Single Swaram Weighted

Slide 71

Slide 71 text

Melody insights #2 Swarams close to each other are more melodious The rules of Arohanam, Avarohanam are encoded We begin to see gamakams Sometimes, the in-between is worse than either extreme

Slide 72

Slide 72 text

Three swaram probabilities and more A simple markov chain

Slide 73

Slide 73 text

First Order Matrix https://en.wikipedia.org/wiki/Markov_chain#Music Markov Chains in Music https://github.com/rm-hull/markov-chains Second Order Matrix

Slide 74

Slide 74 text

Markov Chains in Music

Slide 75

Slide 75 text

No content

Slide 76

Slide 76 text

Melody insights #3 Generic markov chains don’t really work LSTMs also don’t work, probably

Slide 77

Slide 77 text

By-products

Slide 78

Slide 78 text

Automatic Transcription

Slide 79

Slide 79 text

Automatic Transcription (:..n1 :..n1 :..m1 :..m1 :..d1 :..d1 :..d3 :.g1 :.m1 :.m1 :.m1 :.r3 :.r1 :..n3 :..d3 :..p :..g3 :..m1 :..m2 :..g2 :..m1 :..g3 :.r1 :.s :.s :..r1 : ..p :.s :..n3 :.s :.s :.g1 :.s :..d3 :..n1 :..n1 :..n1 :..r1 :.s :.s :.g1 :.r3 :.g1 :.r1 :..d3 :..d3 :..d3 :.g1 :.r1 :.s :.g1 :..r2 :..r1 :.s :.r3 :..n3 :..d3 :..d3 :..n1 :..n1 :..n1 :..n1 :..n1 :..p :.s :..n1 :..g2 :..n 3 :.r1 :.g3 :.g3 :.m1 :.r3 :.m1 :.g3 :.g3 :.p :.m1 :.m1 :..m1 :..g3 :.m1 :..r2 :..r2 :..n3 :.s :.s :.g1 :.g3 :.m2 :.p :.d2 :.m2 :.m1 :.r3 :.r3 :.g 1 :.g1 :.g1 :.r1 :..n1 :.r3 :.g3 :.s :.s :.r1 :.g1 :.r1 :..n3 :..n1 :..d3 :..d3 :..n1 :..d3 :..n1 :..n1 :..n1 :..r1 :.s :..n1)

Slide 80

Slide 80 text

Raga Identification

Slide 81

Slide 81 text

Goodness of fit test

Slide 82

Slide 82 text

:base mohanam-base :samples mohanam-files (12.39 3.84 11.14 6.46 9.88 7.02 9.41 12.61 13.22 1.58) :base mohanam-base :samples kalyani-files (10.95 28.66 25.61 15.26 27.32 21.53 16.42 18.58 24.80 23.80) :base mohanam-base :samples revati-files (46.56 57.19 65.69 55.21 38.61 78.10 56.27 42.99 70.92 58.39)

Slide 83

Slide 83 text

Raga Identification Revati sample vs Revati base Mohana sample vs Revati base

Slide 84

Slide 84 text

What next Model insights as melodic abstractions Use synthesis models with generative music Experiment with Rhythm Synthesise Human Voice Deep learning (Recurrent variational auto encoders)

Slide 85

Slide 85 text

Is this music though?

Slide 86

Slide 86 text

Behag Dasarapada Abhishek Raghuram

Slide 87

Slide 87 text

Making Machines that Make Music Srihari Sriraman nilenso