Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Making machines that make music – Strangeloop 2018

Srihari Sriraman
September 27, 2018

Making machines that make music – Strangeloop 2018

Another iteration of the talk, for presentation at Strangeloop 2018.

Srihari Sriraman

September 27, 2018
Tweet

More Decks by Srihari Sriraman

Other Decks in Research

Transcript

  1. Making machines
    that make music
    ीहर ीरामन्
    नलेो
    Srihari Sriraman
    nilenso

    View Slide

  2. Why I do this
    I sing, I do computers
    Bleeding edge research
    A long time hobby
    Re-discovery

    View Slide

  3. Structure of this talk
    Universality, Carnatic music
    A timeline of 4 phases
    Bridging the gap between machines and humans
    Understand domain, create abstractions
    Synthesise ⟷ Generate ⟷ Transcribe
    Reflect on melody insights

    View Slide

  4. This is what it sounds like
    [play synthesised music]

    View Slide

  5. Prominent tonic
    Resolves tension from musical movement
    Pervasive pentatonic scales
    Common in a lot of cultures: celtic, chinese, african, english, indian, etc
    7 note scales, 12 semitones
    Latin, Chinese, Indian cultures have similar solfege
    Emotive melodies
    Lullabies, sing-alongs, march songs, festival songs, hymns
    Universality
    Natural phenomena observed around the world

    View Slide

  6. Music is
    Math / Science / Art

    View Slide

  7. Universality

    View Slide

  8. Universality
    Natural phenomenons
    Prominent tonic
    Pervasive pentatonic scales
    7 note scales, 12 semitones
    Emotive melodies
    Tonic | आधार षज
    - Resolves tension from musical
    movement.
    - Everyone knows when a piece of
    music returns to the tonic.

    View Slide

  9. Pentatonic | औडुवम्
    Common in a lot of cultures: celtic,
    chinese, african, english, indian, etc
    hps://en.wikipedia.org/wiki/Pentatonic_scale#Pervasiveness
    Popular Bobby McFerrin demonstration
    of the power of the pentatonic scale
    hps://www.youtube.com/watch?v=ne6tB2KiZuk
    Universality
    Natural phenomenons
    Prominent tonic
    Pervasive pentatonic scales
    7 note scales, 12 semitones
    Emotive melodies

    View Slide

  10. Universality
    Natural phenomenons
    Prominent tonic
    Pervasive pentatonic scales
    7 note scales, 12 semitones
    Emotive melodies
    7 notes | स राः
    Latin: do re mi fa sol la si
    Chinese: shàng chě gōng fán liù wù yi
    Indian: sa ri ga ma pa da ni

    View Slide

  11. Universality
    Natural phenomenons
    Prominent tonic
    Pervasive pentatonic scales
    7 note scales, 12 semitones
    Emotive melodies
    Emotive melodies | रस
    Happy sing-alongs, lullabies, march songs,
    harvest songs, festival songs, hymns,
    funeral songs sound similar across
    cultures.

    View Slide

  12. Carnatic music
    शाीय संगीतम्

    View Slide

  13. Kalyani, Extempore
    MS Gopalakrishnan, Violin

    View Slide

  14. Tanpura
    Veena Mridangam

    View Slide

  15. Khamas, Thillana
    Abhishek Raghuram

    View Slide

  16. Classical, art music
    - Not Tribal | Folk | Religious | Popular
    - Intent is artistic
    - Music itself is the sole focus
    Carnatic Music
    South Indian Classical Music
    Classical, art music
    Ancient, traditional music
    Melodic music
    Extempore, and compositions

    View Slide

  17. Ancient, traditional music
    – Earliest treatise ना शा (Natya Shastra)
    at 500BC
    – लण (Lakshana Grantha): Rich
    tradition of Sanskrit grammar texts
    – Oral tradition, trained by exposure
    Carnatic Music
    South Indian Classical Music
    Classical, art music
    Ancient, traditional music
    Melodic music
    Extempore, and compositions

    View Slide

  18. Carnatic Music
    South Indian Classical Music
    Classical, art music
    Ancient, traditional music
    Melodic music
    Extempore, and compositions
    Digitisation hype cycle

    View Slide

  19. Carnatic Music
    South Indian Classical Music
    Classical, art music
    Ancient, traditional music
    Melodic music
    Extempore, and compositions
    Carnatic music
    Digitisation hype cycle

    View Slide

  20. Melodic music
    - Has lile place for harmony
    - Emphasis on notes in succession
    - Explores melody through ragas and gamakas
    Carnatic Music
    South Indian Classical Music
    Classical, art music
    Ancient, traditional music
    Melodic music
    Extempore, and compositions

    View Slide

  21. Extempore, compositions
    - Similar to jazz music
    - Renditions of century old compositions
    - Improvisations are generally around
    compositions
    Carnatic Music
    South Indian Classical Music
    Classical, art music
    Ancient, traditional music
    Melodic music
    Extempore, and compositions

    View Slide

  22. Model fundamental abstractions
    Render prescriptive notation
    Model and synthesise gamaka
    Phase #1

    View Slide

  23. Fundamental abstractions
    आधार षज, र, राग

    View Slide

  24. Fundamentals
    आधार षज, र, राग
    Adhara Shadja / Tonic
    Swara / Note
    Raga / Melodic Framework
    Demo
    Adhara Shadja
    आधार षज | Tonic, pitch
    - Choice of the artist based on voice
    - Does not change for every song
    - Swaras are relative to this

    View Slide

  25. (def adhaara-shadja
    {:.a 57 :.a# 58 :.b 59
    :c 60 :c# 61 :db 61
    :d 62 :d# 63 :eb 63
    :e 64 :f 65 :f# 66
    :gb 66 :g 67 :g# 68
    :ab 68 :a 69 :a# 70
    :bb 70 :b 71 :c. 72})
    Fundamentals
    आधार षज, र, राग
    Adhara Shadja / Tonic
    Swara / Note
    Raga / Melodic Framework
    Demo

    View Slide

  26. - The 12 semitones in an octave
    - Solfege is sung, and used as notation
    - Elements of a raga
    Swara
    र | Note
    Fundamentals
    आधार षज, र, राग
    Adhara Shadja / Tonic
    Swara / Note
    Raga / Melodic Framework
    Demo

    View Slide

  27. Sa Ri Ga Ma Pa Da Sa
    Ni

    View Slide

  28. Sa Ri Ga Ma Pa Da Sa
    Ni
    S R G M P D S
    N
    R1 R2 R3
    G1 G2 G3 M1 M2
    D1 D2 D3
    N1 N2 N3
    Notation
    Pronunciation
    Variations

    View Slide

  29. (def madhya-sthayi-sthanams
    {:s 0
    :r1 1 :r2 2 :r3 3
    :g1 2 :g2 3 :g3 4
    :m1 5 :m2 6
    :p 7
    :d1 8 :d2 9
    :d3 10
    :n1 9 :n2 10 :n3 11})
    Fundamentals
    आधार षज, र, राग
    Adhara Shadja / Tonic
    Swara / Note
    Raga / Melodic Framework
    Demo

    View Slide

  30. (def sthayis
    {:mandra {:position :before
    :dots "."
    :difference -12}
    :madhya {:position :none
    :dots ""
    :difference 0}
    :thara {:position :after
    :dots "."
    :difference 12}})
    Fundamentals
    आधार षज, र, राग
    Adhara Shadja / Tonic
    Swara / Note
    Raga / Melodic Framework
    Demo

    View Slide

  31. Prescriptive Notation
    s,,r g,p, d,s., n,d,
    p,dp mgrs rsnd s,,,
    Fundamentals
    आधार षज, र, राग
    Adhara Shadja / Tonic
    Swara / Note
    Raga / Melodic Framework
    Demo

    View Slide

  32. Raga
    राग | Melodic framework
    Kaapi, Extempore
    TM Krishna

    View Slide

  33. Raga
    राग / Melodic framework
    - Melodic modes with added specialities
    - Grammar that a composition adheres to
    - Evokes a certain set of emotions
    Fundamentals
    आधार षज, र, राग
    Adhara Shadja / Tonic
    Swara / Note
    Raga / Melodic Framework
    Demo

    View Slide

  34. ragavardhini.ragams> (count ragams)
    5280
    ragavardhini.ragams> (into {} (take 2 ragams))
    {:mayamalavagowla
    {:num 15
    :arohanam [:s :r1 :g3 :m1 :p :d1 :n3 :s.]
    :avarohanam [:s. :n3 :d1 :p :m1 :g3 :r1 :s]}
    :gaula
    {:arohanam [:s :r1 :m1 :p :n3 :s.]
    :avarohanam [:s. :n3 :p :m1 :r1 :g3 :m1 :r1 :s]
    :parent-mela-name :mayamalavagowla
    :parent-mela-num 15}}
    Fundamentals
    आधार षज, र, राग
    Adhara Shadja / Tonic
    Swara / Note
    Raga / Melodic Framework
    Demo

    View Slide

  35. Scale | आरोह, अवरोह
    – Example:
    – Aarohanam: 1 2 3 5 6 8
    – Avarohanam: 8 7 6 5 4 3 2 1
    – They form a skeleton of the raga
    – Can be asymmetric, and non-linear
    Fundamentals
    आधार षज, र, राग
    Adhara Shadja / Tonic
    Swara / Note
    Raga / Melodic Framework
    Demo

    View Slide

  36. (:ragam (search/search-ragam “goula"))
    {:arohanam (:s :r1 :m1 :p :n3 :s.)
    :avarohanam (:s. :n3 :p :m1 :r1 :g3 :m1 :r1 :s)
    :name :gaula
    :parent-mela-name :mayamalavagowla
    :parent-mela-num 15}
    (play-arohanam-and-avarohanam
    (:ragam (search/search-ragam “goula")))
    Lookup and play the scale
    Fundamentals
    आधार षज, र, राग
    Adhara Shadja / Tonic
    Swara / Note
    Raga / Melodic Framework
    Demo

    View Slide

  37. (play-phrase (phrase [:s :r2 :g3 :p :m1 :g3 :r2 :s]
    [ 1 1 1 1 1 1 2 4]
    (:lower kalams)))
    (play-phrase
    (phrase (:mechakalyani r/ragams)
    [:m :d :n :g :m :d :r :g :m :g :m :d :n :s.]
    [ 1 1 2 1 1 2 1 1 4 1 1 1 1 4]
    (:middle kalams)))
    Play a phrase in a raga
    Fundamentals
    आधार षज, र, राग
    Adhara Shadja / Tonic
    Swara / Note
    Raga / Melodic Framework
    Demo

    View Slide

  38. (play-notation (:bilahari r/ragams)
    "s,,r g,p, d,s., n,d,
    p,dp mgrs rs .n .d s,,,")
    (play-notation (:mayamalavagowla r/ragams)
    "s,,r g,p, d,s., n,d,
    p,dp mgrs rs .n .d s,,,")
    Play prescriptive notation
    Fundamentals
    आधार षज, र, राग
    Adhara Shadja / Tonic
    Swara / Note
    Raga / Melodic Framework
    Demo

    View Slide

  39. Etymology in Sanskrit
    सक
    ् क
    ृ तम् (correctly done) = सं
    ृ तम्
    Specifies grammar for creating new words
    Words can be composed (समास / Samāsa)
    Elegant solution to the naming problem in soware

    View Slide

  40. मेव राजते इत र
    - that which shines on its own
    - In the context of grammar, “swara”
    means vowel. A consonant in Sanskrit
    cannot be pronounced without a vowel.
    Etymology
    Independent
    Resonant
    Swara
    र | Note

    View Slide

  41. ः: अनुरणनाक: तो रयत
    - that which shines on its own
    - resonant, unlike spoken words
    Sangita Ratnakara, Sarngadeva, 12 CE
    Swara
    र | Note

    View Slide

  42. Raga / राग
    – colour: hue, tint, dye
    – emotion: desire, interest, anger, melancholy, joy,
    delight, yearning
    – More than a scale: phraseology / रागवाचक
    Raga
    राग / Melodic framework

    View Slide

  43. More than a scale
    – A class of ragas has a phraseology / रागवाचक
    – Learning a raga is akin to learning a
    language by exposure (say a mother tongue)
    – A raga is absorbed
    Etymology
    Color, Emotion
    More than a scale
    Raga
    राग / Melodic framework

    View Slide

  44. So far..
    ..we’ve managed to play some prescriptive
    notation within the context of a raga

    View Slide

  45. But..
    ..that doesn’t sound like Carnatic music

    View Slide

  46. Enter Melographs
    Me
    Machine

    View Slide

  47. Enter Melographs
    Me
    Machine

    View Slide

  48. Gamaka

    View Slide

  49. छाया ुरा याम् रो य गमये
    What is it?
    Kinds of gamaka
    Prescriptive vs Descriptive
    Prior art
    Synthesis
    Rendering PASR
    Gamaka
    गमक / Ornamentation
    Movement of a note from it’s pitch towards another
    so that the second passes like shadow over it.
    Sangita samayasaara, Parsvadeva, 12CE

    View Slide

  50. - Embellishment, ornamentation
    - Shake, push, glide, flick
    - Pervasive in Carnatic music
    - Every swaram is a continuum
    What is it?
    Gamaka
    गमक / Ornamentation
    What is it?
    Kinds of gamaka
    Prescriptive vs Descriptive
    Prior art
    Synthesis
    Rendering PASR

    View Slide

  51. Sphuritam
    Jaaru
    What is it?
    Kinds of gamaka
    Prescriptive vs Descriptive
    Prior art
    Synthesis
    Rendering PASR
    Gamaka
    गमक / Ornamentation

    View Slide

  52. Kampitam
    Orikai
    What is it?
    Kinds of gamaka
    Prescriptive vs Descriptive
    Prior art
    Synthesis
    Rendering PASR
    Gamaka
    गमक / Ornamentation

    View Slide

  53. Prescriptive vs Descriptive
    What is it?
    Kinds of gamaka
    Prescriptive vs Descriptive
    Prior art
    Synthesis
    Rendering PASR
    Gamaka
    गमक / Ornamentation

    View Slide

  54. Prior art
    Gaayaka | Subramanian, 2009
    | S, N D | N S R G |
    ((P S,,)) , ((S , S>>> S)) -((D. S. D))
    ((S , S>> S))- S R ((G<< G , ,))
    - Database of phrases
    - < and > increase and decrease pitch
    - ( and ) are speed factors; deeper is faster
    What is it?
    Kinds of gamaka
    Prescriptive vs Descriptive
    Prior art
    Synthesis
    Rendering PASR
    Gamaka
    गमक / Ornamentation

    View Slide

  55. PASR | Srikumar, 2013
    - Pitch, Aack, Sustain, Release
    - Vector specifies the PASR vars for each note
    [["^pa:2" [[4 0 0 0] [2 2 0 0] [7 2 4 0]]]
    ["^ma1:2" [[5 0 8 0]]]
    ["^ga3:2" [[5 0 6 0] [4 0.5 0 0.5] [5 1 0 0]]]
    ["^ga3" [[5 0 0 0.5] [4 1 1 1] [5 0.5 0 0]]]
    ["ma1" [[5 0 0 0.5] [4 1 1 1] [5 0.5 0 0]]]
    ["^ri2:2" [[5 0 0 0.5] [4 1 1 1] [5 0.5 0 0.5]
    [2 1 1 1] [4 0.5 0 0]]]
    ["^ga3" [[4 0 0 0.5] [2 1 1 1] [[5 0.2] 0.5 0 0]]]
    ["ri2" [[[5 0.2] 0 0 0.5] [2 1 1 1] [4 0.5 0 0]]]
    ["^sa:4" [[0 0 16 0]]]]
    -
    Prior art
    Gamaka
    गमक / Ornamentation
    What is it?
    Kinds of gamaka
    Prescriptive vs Descriptive
    Prior art
    Synthesis
    Rendering PASR

    View Slide

  56. SSP | Sangita Samradaya Pradarshini
    संगीत सादाय दषन
    What is it?
    Kinds of gamaka
    Prescriptive vs Descriptive
    Prior art
    Synthesis
    Rendering PASR
    Gamaka
    गमक / Ornamentation

    View Slide

  57. Gamaka
    गमक / Ornamentation
    What is it?
    Kinds of gamaka
    Prescriptive vs Descriptive
    Prior art
    Synthesis
    Rendering PASR

    View Slide

  58. - Frequency envelopes
    (g-inst sphuritam-inst
    (envelope [f f lf f f]
    [ d4 d1 d1 d4]
    :welch))
    (g-inst jaru-inst
    (envelope [pf f f]
    [ d9 d1]
    :welch))
    Synthesis
    Gamaka
    गमक / Ornamentation
    What is it?
    Kinds of gamaka
    Prescriptive vs Descriptive
    Prior art
    Synthesis
    Rendering PASR

    View Slide

  59. Back to this…
    Me
    Machine
    (play-phrase
    (string->phrase (:mohana r/ragams)
    "^g, ~r, ^s ^r ^g ^r"
    1))

    View Slide

  60. Rendering PASR…
    Gamaka
    गमक / Ornamentation
    What is it?
    Kinds of gamaka
    Prescriptive vs Descriptive
    Prior art
    Synthesis
    Rendering PASR

    View Slide

  61. Melody Insights #1
    Prescriptive notation is insuicient for reproduction
    Raga as a framework for creating melodies is powerful
    Melographs make melodies tangible for observation
    Gamaka synthesis can be mathematically modelled as a curve

    View Slide

  62. Generate a stream of Carnatic music
    Try to convert existing music into music data
    Study music data, look for paerns
    Infer characteristics of raga from music data
    Phase #2

    View Slide

  63. Generation
    A naive approach

    View Slide

  64. (defn gen-durations []
    (let [m 16
    n 8
    sum (- m n)]
    (->> (repeatedly n #(rand-nth (range (inc sum))))
    (cons sum)
    sort
    adjacent-differences
    (map inc))))
    (defn random-swarams [swarams num]
    (repeatedly num #(rand-nth swarams)))
    (defn random-durations [jathi num]
    ;; [1, 2, 1, 1, 4, 1, 1, 2, 2, 4]
    (take num (flatten (repeatedly gen-durations))))
    Generate random durations
    Pick swarams from a raga
    Generation
    A naive approach

    View Slide

  65. Generate random durations
    Pick swarams from a raga
    Generation
    A naive approach
    (play-completely-random-phrase :mohana 4 100)

    View Slide

  66. Transcription

    View Slide

  67. Step 1.
    Build a massive database by scraping the internet
    Ragas, lyrics
    Ragas
    Compositions, Renditions
    Get data, study data
    Get data, study data
    Pitch histograms
    MIDI histograms
    Tonic identification
    Swaram histograms
    Transcription
    Audio files to notes

    View Slide

  68. Get data, study data
    Step 2.
    Build a user interface to search songs and ragas,
    so that I can see what’s in the database.
    Get data, study data
    Pitch histograms
    MIDI histograms
    Tonic identification
    Swaram histograms
    Transcription
    Audio files to notes

    View Slide

  69. Get data, study data
    Step 3.
    Download renditions of a given raga
    (->> "bhairavi"
    (db-search/renditions-in-ragam)
    (take 50)
    (download-renditions)
    hp://github.com/ssrihari/kosha
    Get data, study data
    Pitch histograms
    MIDI histograms
    Tonic identification
    Swaram histograms
    Transcription
    Audio files to notes

    View Slide

  70. Get data, study data
    Step 4.
    Transcribe the fundamental frequencies of vocals
    Get data, study data
    Pitch histograms
    MIDI histograms
    Tonic identification
    Swaram histograms
    Transcription
    Audio files to notes

    View Slide

  71. Get data, study data
    Step 5.
    Look at melographs, spectrograms, and try to find
    usable insights about melody.
    kalyANi-MS-Subbulakshmi-nidhi_cAla_sukhamA-tyAgarAja3.mpeg.wav.pitch.frequencies
    Get data, study data
    Pitch histograms
    MIDI histograms
    Tonic identification
    Swaram histograms
    Transcription
    Audio files to notes

    View Slide

  72. Pitch histograms
    (defn pitch-histogram [filename]
    (let [frequencies (f/freqs-from-file filename)
    data-set (to-dataset frequencies)
    title (str "pitch histogram for " filename)]
    (with-data data-set
    (save (histogram (or plot-axis :freq)
    :density true
    :nbins 1200
    :title title
    :x-label "frequency"
    :y-label "ocurrances")
    (str filename "-pitch-histogram.png")
    :width 1200
    :height 300))))
    Get data, study data
    Pitch histograms
    MIDI histograms
    Tonic identification
    Swaram histograms
    Transcription
    Audio files to notes

    View Slide

  73. Pitch histograms
    (defn pitch-histogram [filename]
    (let [frequencies (f/freqs-from-file filename)
    data-set (to-dataset frequencies)
    title (str "pitch histogram for " filename)]
    (with-data data-set
    (save (histogram (or plot-axis :freq)
    :density true
    :nbins 1200
    :title title
    :x-label "frequency"
    :y-label "ocurrances")
    (str filename "-pitch-histogram.png")
    :width 1200
    :height 300))))
    Get data, study data
    Pitch histograms
    MIDI histograms
    Tonic identification
    Swaram histograms
    Transcription
    Audio files to notes

    View Slide

  74. kalyANi-MS-Subbulakshmi-nidhi_cAla_sukhamA-tyAgarAja3.mpeg
    Pitch histograms

    View Slide

  75. Pitch histograms

    View Slide

  76. MIDI Histogram
    Normalised MIDI Histogram
    (overtone.core/hz->midi frequency)
    (map #(mod % 12) frequencies)

    View Slide

  77. Tonic identification
    Bellur, A., V. Ishwar, X. Serra, and H. A. Murthy (2012)
    A knowledge based signal processing approach to tonic identification in
    indian classical music.
    Bellur, A., and H. A. Murthy (2013)
    Automatic tonic identification in classical music using melodic
    characteristics and tuning of the drone.
    Srihari, S. (2016) *
    Pick the highest one, it mostly just works.
    * not really, no
    Get data, study data
    Pitch histograms
    MIDI histograms
    Tonic identification
    Swaram histograms
    Transcription
    Audio files to notes

    View Slide

  78. Tonic identification
    (defn find-tonic-midi [freqs prominent-note]
    (let [midi-occurrances (midi-histogram freqs)
    prominent-midi-diff
    (->> midi-occurrances
    normalize-octaves
    (sort-by second >)
    ffirst)]
    (cond
    (= :s prominent-note) (+ 60 prominent-midi-diff)
    (= :p prominent-note) (+ 65 prominent-midi-diff)
    :else (+ 60 prominent-midi-diff))))
    Get data, study data
    Pitch histograms
    MIDI histograms
    Tonic identification
    Swaram histograms
    Transcription
    Audio files to notes

    View Slide

  79. Get data, study data
    Pitch histograms
    MIDI histograms
    Tonic identification
    Swaram histograms
    Transcription
    Audio files to notes
    Tonic identification
    (defn find-tonic-midi [freqs prominent-note]
    (let [midi-occurrances (midi-histogram freqs)
    prominent-midi-diff
    (->> midi-occurrances
    normalize-octaves
    (sort-by second >)
    ffirst)]
    (cond
    (= :s prominent-note) (+ 60 prominent-midi-diff)
    (= :p prominent-note) (+ 65 prominent-midi-diff)
    :else (+ 60 prominent-midi-diff))))

    View Slide

  80. Swaram Histogram
    Kalyani
    S, R2, G3, M2, P, D2, N3, S.
    S., N3, D2, P, M2, G3, R2, S

    View Slide

  81. Kalyani
    S, R2, G3, M2, P, D2, N3, S.
    S., N3, D2, P, M2, G3, R2, S
    Revati
    S, R1, M1, P, N2, S.
    S., N2, P, M1, R1, S
    Mohana
    S, R2, G3, P, D2, S.
    S., D2, P, G3, R2, S

    View Slide

  82. Generation
    Not so naive now

    View Slide

  83. (defn get-next-swaram [swaram-allocations]
    (let [r (rand 100)]
    (->> swaram-allocations
    (filter (fn [[swaram [b e]]] (< b r e)))
    first
    first)))
    One swaram probabilities
    One swaram probabilities
    Two-swaram probabilities
    One vs two swarams
    Markov chains
    Generation

    View Slide

  84. Random
    Swaram Weighted
    One swaram probabilities
    One swaram probabilities
    Two-swaram probabilities
    One vs two swarams
    Markov chains
    Generation

    View Slide

  85. Melody Insights #2
    Tonic is prominent
    Sa (tonic), and Pa (5th) have higher and sharper peaks
    Other note peaks are blunt
    Probabilities of all swarams in a raga are not the same
    Probabilities across octaves are not the same

    View Slide

  86. Make generative music sound continuous
    Extract raga grammar, use it in generation
    How do we detect melodic paerns?
    Phase #3

    View Slide

  87. Generation
    Continued…

    View Slide

  88. Two swaram probabilities
    {:.s
    ([:.s 57.35530546623794]
    [:.r2 6.109324758842444]
    [:s 5.868167202572347]
    [:.d2 5.466237942122186]
    [:.p 5.42604501607717]
    [:.g3 3.215434083601286]…)
    :.r2
    ([:.s 29.53020134228188]
    [:.r2 21.0738255033557]
    [:.g3 14.49664429530201]
    [:.p 6.577181208053691]…)
    :.p
    ([:.p 44.131214761660694]
    [:.d2 16.04305484366991]
    [:.g3 8.14966683751922]…)
    …}
    One swaram probabilities
    Two swaram probabilities
    One vs two swarams
    Markov chains
    Generation

    View Slide

  89. Two swaram probabilities
    One swaram probabilities
    Two swaram probabilities
    One vs two swarams
    Markov chains
    Generation

    View Slide

  90. One swaram probabilities
    Two swaram probabilities
    One vs two swarams
    Markov chains
    Generation
    (defn swaram-generator [swaram allocations]
    (lazy-seq
    (cons swaram
    (swaram-generator
    (get-next-swaram swaram allocations)
    allocations))))
    Two swaram probabilities

    View Slide

  91. One swaram probabilities
    Two swaram probabilities
    One vs two swarams
    Markov chains
    Generation
    One vs two swarams

    View Slide

  92. One swaram probabilities
    Two swaram probabilities
    One vs two swarams
    Markov chains
    Generation
    Markov chains
    https://en.wikipedia.org/wiki/Markov_chain#Music
    https://github.com/rm-hull/markov-chains

    View Slide

  93. One swaram probabilities
    Two swaram probabilities
    One vs two swarams
    Markov chains
    Generation
    Markov chains
    https://en.wikipedia.org/wiki/Markov_chain#Music
    https://github.com/rm-hull/markov-chains

    View Slide

  94. Melody Insights #3
    Adjacent swarams seem more melodious
    Sequencing per raga rules makes it sound beer
    Gamakas at this level give a Carnatic texture
    Sometimes, the in-between is worse than either extreme
    Higher order Markov chains do not make it sound beer

    View Slide

  95. Back to the drawing board
    Explore rhythm concepts
    Explore repetition
    Try to generate melodies
    Phase #4

    View Slide

  96. Laya, and Tala
    Temporal discipline

    View Slide

  97. ुत: माता लय: पता
    Shruti is the mother, Laya is the father
    - Shruti: Pitch fidelity
    - Laya: Temporal discipline, or Tempo
    Laya, Tala
    Temporal discipline
    Etymology, Meaning
    Avartana
    Matra, Akshara, Gati

    View Slide

  98. - Laya organizes music in time
    - A change in tempo has communicative weight
    Laya | लय
    Laya, Tala
    Temporal discipline
    Etymology, Meaning
    Avartana
    Matra, Akshara, Gati

    View Slide

  99. TAla is the fundamental principle that
    binds Gita (song), Vadya (instrument)
    and Nria (dance).
    ताळ: तल थठयामत गीतम्
    वाम् तथानृं यथाालेततम्
    Sangita Ratnakara, Sarngadeva, 12 CE
    Laya, Tala
    Temporal discipline
    Etymology, Meaning
    Avartana
    Matra, Akshara, Gati

    View Slide

  100. Tala | ताळ
    - A recurrent time frame, a cycle of beats
    - Derived from poetic meter, rather than dance
    - Hand gestures for conduction
    Laya, Tala
    Temporal discipline
    Etymology, Meaning
    Avartana
    Matra, Akshara, Gati

    View Slide

  101. Tala Systems
    - Navasandhi talas (108)
    - Chapu talas (4)
    - Suladi Sapta talas (175)
    Laya, Tala
    Temporal discipline
    Etymology, Meaning
    Avartanam
    Matra, Akshara, Gati

    View Slide

  102. Avartana | आवतन
    - avartana: 1 cycle
    - signifies number of beats in 1 cycle
    - simhanandana tala: 128 beats per cycle
    - sharaba nandana tala: 79 beats per cycle
    Laya, Tala
    Temporal discipline
    Etymology, Meaning
    Avartana
    Matra, Akshara, Gati

    View Slide

  103. Kriya, Anga, Jati
    - Anga: A subdivision of the tala
    - Kriya: The gesture used to refer to an anga
    - Jati: The number of matras in a laghu
    Laya, Tala
    Temporal discipline
    Etymology, Meaning
    Avartanam
    Kriya, Anga, Jati
    Matra, Akshara, Gati

    View Slide

  104. Matra, Akshara, Gati
    - Matra: beat (within an avartana)
    - Akshara: division of beat (within a matra)
    - Gati: gait; factor dividing matra into akshara
    Laya, Tala
    Temporal discipline
    Etymology, Meaning
    Avartana
    Matra, Akshara, Gati

    View Slide

  105. Transcription
    based on laya and gati

    View Slide

  106. Original Record
    Gati: 1
    Gati: 2
    Gati: 4
    Gati: 8
    Transcription
    Based on laya and gati
    A clinical process
    Detail vs structure trade-os
    Varna and Alankara
    Removing outliers

    View Slide

  107. Gati: 2
    Gati: 8
    Melodic structure is visible at 1/2 a matra / beat
    Embellishments / Gamakas are visible at 1/8th
    Transcription
    Based on laya and gati
    A clinical process
    Detail vs structure trade-os
    Varna and Alankara
    Removing outliers

    View Slide

  108. Transcription
    Based on laya and gati
    A clinical process
    Detail vs structure trade-os
    Varna and Alankara
    Removing outliers
    Varna and Alankara
    - Tonal paerns at matra (beat) level
    is called varna
    - Embellishments at akshara (sub-beat)
    level is called alankara

    View Slide

  109. Removing outliers
    - Short-lived frequencies (< akshara/2)
    - Non prominent swaras in raga
    - Low and high cutos for unnatural frequencies
    - Octave shis, and continuous tonics
    - Detecting rests
    Transcription
    Based on laya and gati
    A clinical process
    Detail vs structure trade-os
    Varna and Alankara
    Removing outliers

    View Slide

  110. Generation
    Last time, promise

    View Slide

  111. Revisiting Markov Chains
    - Transcribing with input BPM improves quality
    - Markov chains at multiple levels of granularity
    - So many knobs/toggles to play with
    Revisiting Markov chains
    At matra, akshara levels
    Embellishment chains
    Aavrii
    Generation

    View Slide

  112. (markov/generate-and-play
    mohanam-swarams
    {:order 12
    :duration-seconds 100
    :playback-gati 2
    :generation-gati 4
    :mode :discrete
    :bpm 80})
    (markov/generate-and-play
    mohanam-swarams
    {:order 22
    :duration-seconds 100
    :playback-gati 10
    :generation-gati 8
    :mode :continuous
    :bpm 80})
    Revisiting Markov chains
    At matra, akshara levels
    Embellishment chains
    Aavrii
    Generation

    View Slide

  113. Embellishment chains
    - Filling matra chains with alankara chains fails
    - Gamakas are contextual, and have their own laya
    Revisiting Markov chains
    At matra, akshara levels
    Embellishment chains
    Aavrii
    Generation

    View Slide

  114. Revisiting Markov chains
    At matra, akshara levels
    Embellishment chains
    Aavrii
    Generation Repetition / Avartana
    Organic repetition seems hard to model, I’m on it.

    View Slide

  115. Melody Insights #4
    Time is fundamental to melody, and inseparable from frequency
    Humans perceive more than a beat when hearing a beat
    Machines understand tonal structures at 1/2 a beat
    Markov chains don’t work for embellishments
    Simplest form of repetition can be powerful

    View Slide

  116. By-products

    View Slide

  117. Using a simple
    goodness of fit test
    By-product #1
    Raga Identification
    (defn raga-diff [base-osp sample-osp]
    (let [ordered-vals (fn [osp]
    (vals (sort-by first < osp)))
    probs (ordered-vals base-osp)
    table (ordered-vals sample-osp)]
    (chisq-test :table table
    :probs (map #(* 0.01 %) probs))))

    View Slide

  118. (defn raga-diff [base-osp sample-osp]
    (let [ordered-vals (fn [osp]
    (vals (sort-by first < osp)))
    probs (ordered-vals base-osp)
    table (ordered-vals sample-osp)]
    (chisq-test :table table
    :probs (map #(* 0.01 %) probs))))
    Using a simple
    goodness of fit test
    By-product #1
    Raga Identification

    View Slide

  119. By-product #2
    Useful in education, and
    preserving music
    Automatic notation, playback
    (let [props {:bpm 87 :gati 2 :offset 60}
    swaras (bpm-transcribe
    (first samples/research-files))]
    (prescriptive-notation
    (get swaras (:gati props)) 16))
    (“.s .r .g .p .d , s , .d .p .g , "
    ".r .s , .r , .g .p , .d , s , .d"
    " .p , .g , .r , .s .r .g .p , .d"
    " s , .d .p , .g .r .s .r .g .p ,"
    " .d s , , .d .p , .r .s .r .d s "
    ".d .p .g .r .g s .p .g .p s .p .”)

    View Slide

  120. By-product #3
    An electronic rhythm accompanist
    Electronic tala assistant
    (def talas
    {:eka [:I]
    :rupaka [:O :I]
    :jhampa [:I :U :O]
    :triputa [:I :O :O]
    :matya [:I :O :I]
    :ata [:I :I :O :O]
    :dhruva [:I :O :I :I]})
    (play-avartanams 60 4
    (->> (tala-accents :chatusra :rupaka)
    #(mapcat combo)
    (repeatedly 10)))

    View Slide

  121. Is this music though?
    Is it art?

    View Slide

  122. Behag, Dasarapada
    Abhishek Raghuram

    View Slide

  123. References
    Sangita Ratnakara of Sarngadeva – RK Shringy, Prem Lata Sharma
    Sangita Sampradaya Pradarsini – SRJ, NR, RSJ
    Brhaddesi of Sri Matanga Muni – Prem Lata Sharma
    Appreciating Carnatic Music – Lakshmi Sreeram
    South Indian Music – P. Sambamurthy
    Compmusic – MTG, Barcelona, Xavier Serra
    Gayaka - Subramanian
    Srikumar’s PASR thesis

    View Slide

  124. Making machines
    that make music
    ीहर ीरामन्
    नलेो
    Srihari Sriraman
    nilenso

    View Slide

  125. Shruti
    ुत
    Etymology
    Intervals in indian music
    ूयते इत ुत:
    - that which should be heard
    - the vedas, tanpura
    - that which can be heard
    - the word shruti probably means an
    interval here

    View Slide