JamSketch Deep α: A CNN-based Improvisation System in Accordance with User's Melodic Outline Drawing

ACM Multimedia Asia 2022 (demo)

Transcript

1. JamSketch Deep α: JamSketch Deep α: A CNN-based Improvisation System in A CNN-based Improvisation System in Accordance with User's Melodic Outline Drawing Accordance with User's Melodic Outline Drawing Tetsuro Kitahara and Akio Yonamine (Nihon Univ., Japan Tetsuro Kitahara and Akio Yonamine (Nihon Univ., Japan) Goal To enable everyone to enjoy musical improvisation Technical details including non-musicians Why improvisation is difficult Cmaj7 Am7 Chord progression Harmony theory Musical scale Music knowledge Do-Re-Mi Learned musical phrases Create a melody Play the melody at the same time Related works ism [Ishida et al. 2004] MIDI Tone Generator If the N-gram probability for this note is lower than a threshold, the note is replaced to another one Thermoscore [Miyashita et al. 2004] System corrects musically unavailable notes System makes keys for musically unavailable notes hot to avoid playing such notes long coJIVE [Buchholz et al. 2007] System controls key width in the virtual keyboard according to musical availability Our proposal JamSketch Deep α Melodic outline User draws: Melody System generates: Intuitive, easy to draw Data representation x1 x2 x3 x60 x61 x62 x63 x73 (C2) (C#2) (D2) (B6) (C) (C#) (B) Pitch in outline (60-dim) Rest in outline (1-dim) Chroma in chord tones (12-dim) ... Input matrix ... ... ... y1 y2 y3 y60 y61 y62 y63 y120 ... ... y121 (C2) (C#2) (D2) (B6) (C2) (C#2) (D2) (B6) Pitch (new) in melody Pitch (cont.) in melody Rest in melody 12 frames 12 frames Output matrix (60-dim) (60-dim) (1-dim) note starting at that time note continued from the previous time Example Example Melodic outline time ch pitch Chord time conv. conv. ch deconv. time Pitch time pitch deconv. Pitch (cont) ch filter size inv. filter size Model Data for training • Output: 96 Blues melodies from Weimar Jazz Database. • Input: Melodic outlines made by smoothing the melodies