Share
Embed
Transcript
31 slides
-
ࣗݾհ • ࢁຊ ༤ʢ·ͱ Ώ͏ʣ • ஜେֶେֶӃ ࣉᖒݚڀࣨ ത࢜3ʢࠓमྃ༧ఆʣ •
ՎএςΫχοΫͷੳ͓Αͼݕग़ʹ͍ͭͯݚڀ • ใΛ·ͱΊΔͷ͕͖ • https://github.com/yamathcy/awesome-music-informatics • https://github.com/yamathcy/ISMIR-2023-Papers • … • ݪઌੜʹ͓༠͍͍͖ͨͩɼܰʹൃදΛਃࠐ͠·ͨ͠ 2 -
ࠓͷ༰ • ࣗવݴޠॲཧ x Իָใॲཧͳݚڀʹ͍ͭͯͷαʔϕΠใࠂ • Իָ -> ςΩετ (Captioning,
Question & Answering) • Իָ + ςΩετͷϚϧνϞʔμϧදݱֶश • ՎࢺੜɼςΩετϕʔεͷԻָੜʹ͍ͭͯࠓճର֎ 3 -
ͭ·Βͳ͍ʁ • ࡢࠓͷେنݴޠϞσϧʢLLMʣʹ͍ͭͯͲ͏ࢥ͍·͔͢ʁ • Ϣʔβʔͱͯ͠ศརʹͳͬͯॿ͔ͬͨ͜ͱଟ͍ • Ұํݚڀऀͱͯ͠Ϧιʔε࣋ͬͨΜউͪͷੈքʹͳΓ໘ന͘ͳ͘ͳͬͨɾ ΕΔ͜ͱ͕গͳ͘ͳͬͨʁ • Իָʹ͍ͭͯใΛநग़ͯ͠؍͍ͨ͠ਓʢat
least ࢲʣʹͱͬͯɼɹɹɹɹɹɹ ͦͷͨΊͷࣗಈࣝผͷੑೳ্͕͕Δ͜ͱΉ͠Ζ͍෩ʁ • ͳͲͳͲ… 4 -
Music and Language (:= M&L) • ԻָใॲཧͱࣗવݴޠॲཧΛ݁ͼͭ ͚Δ • Իָ͔ΒςΩετͰใΛநग़͢Δ
ʢCaptioningɼ Q&Aʣ • ݴ༿ͰԻָΛ୳͢ʢMusic searchʣ etc… • Vision & Languageͱಉ͘͡ɼɹɹɹ ݴޠͱͷϚϧνϞʔμϧֶशͷҰ 5 It’s a jazz music piece with slow tempo. Piano, muted-trumpet, tenor sax, bass, and drums are appeared. Captioning Question: Is this music a vocal song? Answer: No. It’s an instrumental song. Q&A ϛϡʔττϥϯϖοτͱ ςφʔαοΫε͕ϝϩσΟʔΛ ԋ͢Δδϟζָۂ Music search είΞ: 0.91 -
ͳͥࣗવݴޠ͔ʁ • 1. ΦʔϓϯϘΩϟϒϥϦʔͳใهड़͕Մೳ • ↔︎ λά͚ͷࣝผͰσʔληοτ ͷΫϥεʹറΓΛड͚Δ • ʢཧ্ʣࡉ͔ͳχϡΞϯεΛهड़Ͱ
͖Δ • 2. ҉ʹ͞·͟·ͳΛಉ࣌ʹղ͍͍ͯΔ • ↔︎ ࣝผͰɼλεΫ͝ͱʹϞσϧΛ࡞ Βͳ͍ͱ͍͚ͳ͍ 6 ϩοΫ Ϊλʔ ܹ͍͠ 80’ … ͜ͷۂ80’ͷ ϩοΫιϯάͰ͋Γɼ ͍ςϯϙͰ ΪλʔͷԻͱͱʹ ܹ͘͠υϥϜ͕ ໐Γڹ͖·͢ɽ ɽɽɽ λά͚ Ωϟϓγϣχϯά ͜ͷۂ80’ͷ ϩοΫιϯάͰ͋Γɼ ΪλʔͷԻͱͱʹ ܹ͘͠υϥϜ͕ ໐Γڹ͖·͢ɽ ɽɽɽ Ϝʔυࣝผ δϟϯϧࣝผ ָثࣝผ -
M&L͕ࠓΞπ͍ • ࠃࡍձٞISMIRʹ͓͍ͯ2021͝ΖΛڥʹM&Lܥݚڀ͕૿͑ͨ • ϕετϖʔύʔϊϛωʔτʹຖग़ݱ • 2023ॳΊʹMusicLM [Agostinelli 23] ͕ൃදɼݚڀࣄྫैͬͯ૿Ճ
• จͰMusicCapsͱ͍͏Իָ-ΩϟϓγϣϯͷϖΞσʔληοτൃද • AudioυϝΠϯ͚ͩͰͳ͘ɼSymbolicυϝΠϯ·Ͱʹਐग़ • CLaMP [SWu 23] 7 <"HPTUJOFMMJ>.VTJD-.(FOFSBUJOH.VTJD'SPN5FYU "OESFB"HPTUJOFMMJFUBM"S9JW <48V>$-B.1$POUSBTUJWF-BOHVBHF.VTJD1SFUSBJOJOHGPS$SPTT.PEBM4ZNCPMJD.VTJD*OGPSNBUJPO3FUSJFWBM 4IBOHEB8VFUBM*4.*3 -
ࠓͷτϐοΫओʹ2ͭ • Music to Text • Իָʹର͢Δ༰هड़จΛੜ • Music-Language ϚϧνϞʔμϧදݱֶश
• ԻָͱࣗવݴޠؒΛඥ͚ͨಛϕΫτϧͷ֫ಘ 8 -
ͦͷલʹ… • جૅతͳ༻ޠͨͪ • Encoder: ೖྗͱͳΔϝσΟΞใΛΑΓ࣍ݩͳϕΫτϧʹม • Ex. BERT, RoBERTa
• Embedding: ϝσΟΞใͷີϕΫτϧදݱɽ্ۭؒͰͷϕΫτϧͷڑ ͕ҙຯతͳۙ͞Λࣔ͢ • Decoder: EmbeddingΛݩʹग़ྗͱͳΔϝσΟΞใͷܥྻΛਪ • Ex. GPT, LLaMA 10 -
Music -> Text ᶃ Captioningܕ "VEJP &ODPEFS "VEJP&NCFEEJOH 5FYU%FDPEFS <4UBSU>
3PDL TPOH XJUI ʜ ೖྗԻڹ৴߸ %FDPEFSೖྗ લͷεςοϓͷ୯ޠτʔΫϯ ग़ྗςΩετ 3PDL TPOH XJUI QJBOP ʜ ྆Ϟʔμϧ ༥߹ͷͨΊͷ ԿΒ͔ • ೖྗ: ԻָԻڹ৴߸ • Audio Encoderʹ௨͠ɼຒΊࠐΈϕΫτϧ ʢEmbeddingʣΛಘΔ • Text Decoderͷ෦ͰɼAudio embeddingͱ Text Decoderͷதؒग़ྗΛ༥߹ • Text DecoderςΩετΛॱ࣍ੜ͢Δ • Muscaps [Manco 21] , LP-MusicCaps [Doh 23] etc. 11 <.BODP>.VT$BQT(FOFSBUJOH$BQUJPOTGPS.VTJD"VEJP*MBSJB.BODPFUBM *+$// <%PI>-1.VTJD$BQT--.#BTFE1TFVEP.VTJD$BQUJPOJOH 4FVOHIFPO%PIFUBM*4.*3 -
Ϟʔμϧ༥߹ͷͨΊͷςΫχοΫ • Concat: ୯ʹ྆ϞʔμϧͷϕΫτϧΛ݁߹ • FiLM [Perez 18]: ҟͳΔϞʔμϧʹର͢Δ ΞϑΟϯมͷΛܾΊΔ
• Cross-attention: TransformerϕʔεͳΒ Ͱͷํ๏ɽ M&LͰQueryͱKey&Value ΛͦΕͧΕҟͳΔϞʔμϧ͔ΒಘΔ 12 [Perez 18] FiLM Visual reasoning with a General Conditioning Layer. Perez et al. AAAI2018 https://vaclavkosar.com/ml/cross-attention-in- transformer-architecture -
σʔληοτ: MusicCaps [Agostinelli 23] • ԻָͱΩϟϓγϣϯͷϖΞσʔληοτ • Իָ-ΩϟϓγϣϯͷσʔληοτͰ࠷ॳͷΦʔ ϓϯσʔλ •
YouTubeϏσΦͷԻָΫϦοϓ5521݅ʹϓϩͷ ԻָՈ͕هड़จΛΞϊςʔγϣϯ, ܭ15࣌ؒ΄Ͳ • https://www.kaggle.com/datasets/googleai/ musiccaps • Music captioningͷֶशɾධՁʹ༻͍ΒΕΔ 13 "CBHQJQFFOTFNCMFQMBZJOHB RVJDLNFMPEZJOVOJTPO FNQMPZJOH USJMMTUPPSOBNFOUJU YoutubeϏσΦͷ ԻΫϦοϓ Ωϟϓγϣϯ <"HPTUJOFMMJ>.VTJD-.(FOFSBUJOH.VTJD'SPN5FYU "OESFB"HPTUJOFMMJFUBM"S9JW σʔλ গ͠Γͳ͍ -
طଘͷλά͚༻σʔλͷλά͔ΒٖΩϟϓγϣϯΛੜ͠σʔλΛ֦ு 14 • GPT-3.5 TurboʹΑͬͯɼλάͷΈͷ MagnaTagATune [Law 09], Million-song Dataset
[Bertin-Mahieux 11]͔Βٖࣅతͳ ΩϟϓγϣϯΛੜ • ্هͷૢ࡞ʹΑͬͯɼMusicCapsͷ200ഒҎ ্ͷ4000࣌ؒͷϖΞσʔλΛߏங • ٖΩϟϓγϣϯͰ͋ͬͯੑೳΛ্͛ΒΕͨ LP-MusicCaps [Doh 23] <%PI>-1.VTJD$BQT--.#BTFE1TFVEP.VTJD$BQUJPOJOH 4FVOHIFPO%PIFUBM*4.*3 <-BX>&WBMVBUJPOPGBMHPSJUINTVTJOHHBNFT5IFDBTFPGNVTJDUBHHJOH&-BXFUBM*4.*3 <#FSUJO.BIJFVY>5IFNJMMJPOTPOHEBUBTFU5#FSUJO.BIJFVYFUBM*4.*3 -
Music -> Textܕ ᶄ Q&Aܕ • ೖྗ: ԻָԻڹ৴߸, ςΩετ •
Իڹ৴߸ΛAudio Encoderʹ௨͠Audio embeddingΛಘΔ • ్தͰAudio EmbeddingͱTextଆͷதؒग़ྗͱ༥߹͢Δ • ೖྗςΩετ • ᶃText Encoder Ͱॲཧ or • ᶄΠϯετϥΫγϣϯͱͯ͠Text Decoderʹ༩͑Δ • Text DecoderͰςΩετΛग़ྗ • ᶃBART-fusion [Zhang 22], ᶄMu-LLaMa [Liu 24] etc. 15 <;IBOH>*OUFSQSFUJOH4POH-ZSJDTXJUIBO"VEJP*OGPSNFE1SFUSBJOFE-BOHVBHF.PEFM*4.*3 <-JV>-1.VTJD$BQT--.#BTFE1TFVEP.VTJD$BQUJPOJOH FUBM*$"441 "DDFQUFE -
Music Understanding LLaMA [Liu 24] • Իָ৴߸ʹର͢ΔQ&AΛ࣮ݱ • ֶशࡁΈϞσϧΛ׆༻ •
Audio Encoder: MERT [Li 24] • Text Decoder: LLaMA-2 [Touvron 23] • MERT͔Βͷग़ྗʹɼAdapterͱ ͍͏ػߏΛ௨ͯ͠ɼͦΕΛ Decoderͷ࠷ऴʹೖྗ 16 <5PVWSPO>-MBNB0QFO'PVOEBUJPOBOE'JOF5VOFE$IBU.PEFMT5PVWSPOFUBM"SYJW <-J>.&35"DPVTUJD.VTJD6OEFSTUBOEJOH.PEFMXJUI-BSHF4DBMF4FMGTVQFSWJTFE5SBJOJOH :-JFUBM *$-3 -
σʔληοτɿMusicQA [Liu 24] • LLMʹΑͬͯQ&AϖΞΛMusicCaps (Ωϟϓγϣϯ)MagnaTagATune (λ ά)͔Βੜ • 2छྨͷํ๏Ͱ࣭จΛ࡞
• Closed-end: ΠϯετϥΫγϣϯͰ ࣭Λ༩͑ͯɼͦͷ͑Λੜ͢Δ • Open-end: Ϟσϧࣗʹ࣭Λੜ ͤͯ͞ɼͦͷ࣭ͷ͑ੜ͢Δ 17 <-JV>-1.VTJD$BQT--.#BTFE1TFVEP.VTJD$BQUJPOJOH FUBM*$"441 "DDFQUFE Closed Open -
M&L ϚϧνϞʔμϧදݱֶश • ϚϧνϞʔμϧදݱֶश • EncoderʹΑͬͯಘΒΕΔEmbedding ͕ɼಉ֓͡೦༝དྷͰ͋Εڑ͕ۙ͘ ͳΔΑ͏ʹஔ͢ΔۭؒΛಘΔͨΊͷ ํ๏ •
Audio EncoderͱText EncoderΛ͍ɼ ྆ऀͷEmbeddingʹର͢Δڑʹؔ͠ ֶͯश͢Δ 19 -
ϚϧνϞʔμϧදݱֶश͕Մೳʹ͢Δ͜ͱ • จষʹΑΔԻָݕࡧ • จষͷText embeddingͱϥΠϒϥϦʹ͋ΔԻָͷ Audio embeddingؒͷ݁ͼ͚͕ͭͰ͖ΔΑ͏ʹ • จ຺ใΛͭԻָಛྔͷ֫ಘ
• ͢Ͱʹdiscriminableͳಛ͕֫ಘͰ͖͓ͯΓɼɹ θϩγϣοτֶशɼϦιʔελεΫͷసҠ ֶश͕ظ • ୯ମͰੜͰ͖ͳ͍͕ɼMusicLMͰɹɹɹ ςΩετϓϩϯϓτͱԻָͷඥ͚ͷϞδϡʔϧ ͱͯ͠Ұ෦ʹར༻͞Ε͍ͯΔ [Agostinelli 23] 20 ϛϡʔττϥϯϖοτͱ ςφʔαοΫε͕ϝϩσΟʔΛ ԋ͢Δδϟζָۂ -
21 • τϦϓϨοτֶश • ΞϯΧʔɼਖ਼ྫʢࣅ͍ͯΔͷʣɼෛྫʢҟͳΔͷʣΛ༻ҙ͠ɼ ΞϯΧʔ-ਖ਼ྫͷڑʹ ରͯ͠ΞϯΧʔ-ෛྫͷڑΛେ͖͘͢Δ • θϩγϣοτԻָλά͚ [Choi
19], λάʹΑΔԻָݕࡧ [Won 21] • ରরֶश • ΞϯΧʔͱਖ਼ྫͷڑΛ࠷খԽ͠ɼΞϯΧʔͱෛྫͷڑΛ࠷େԽ • ը૾-ςΩετؒͷϞσϧCLIP [Radford 21]Ͱޭͨ͠Έ • ྆ऀͷੑೳΛൺֱͨ͠ࣄྫ͋Γ; ରরֶशͷํ͕Ұʹ࠷దԽͰ͖Δαϯϓϧ͕ଟ͍ [Doh 23] ࣮ݱͷͨΊͷख๏͍͔ͭ͘ <$IPJ>;FSPTIPU-FBSOJOHGPS"VEJPCBTFE.VTJD$MBTTJ fi DBUJPOBOE5BHHJOH+$IPJFUBM*4.*3 <8PO>.VMUJNPEBM.FUSJD-FBSOJOHGPS5BHCBTFE.VTJD3FUSJFWBM.XPOFUBM*$"441 <3BEGPSE>-FBSOJOH5SBOTGFSBCMF7JTVBM.PEFMT'SPN/BUVSBM-BOHVBHF4VQFSWJTJPO 3BEGPSEFUBM"S9JW <%PI>508"3%6/*7&34"-5&9550.64*$3&53*&7"- %PIFUBM *$"441 -
τϦϓϨοτֶश 22 • 3ͭΈͷαϯϓϧΛ༻ҙ͠ɼɹɹɹ ͦΕͧΕAnchor, Positive sample, Negative sampleͱ͢Δ •
Anchorʹର͠ྨࣅ͕ΑΓ͍ۙͷ ΛPositive sampleɼԕ͍ͷΛ Negative sampleͱ͠ɼͦΕͧΕͷɹ ڑͷࠩΛɼ͋ΔδʢϚʔδϯʣʹ ͳΔΑ͏ʹ࠷దԽ͢Δɽ -
࣮ݱํ๏ɿରরֶश • 2ͭͷҟͳΔϞʔμϧͷEmbeddingʹ ର͠ɼಉ͡σʔλϖΞͰ͋ΕੵΛ େ͖͘ɼͦ͏Ͱͳ͚Εখ͘͢͞Δ • ෳͷσʔλϖΞʹରͯ͠Ұڍʹֶश ͕Մೳ • ଛࣦؔ:
InfoNCE Loss 23 I൪ͷαϯϓϧͷaudio embedding z^a_i, text embedding z^t_i ΞϯΧʔͷྨࣅ͕େ͖͘ͳΕ0ʹۙͮ͘ -
M&LͷରরֶशϞσϧ: MuLan [Huang 22] / MusCall [Manco 22] • Իָ
(Audio) ͱςΩετͷϚϧν ϞʔμϧֶशϞσϧ • ͱʹରরֶश͕ϕʔε • (ISMIR 2022Ͱωλ͕ඃΔͱ͍͏ح) • θϩγϣοτྨʹΑΔԻָλά͚ɼసҠ ֶशʹΑΔԻָλά͚ɼԻָݕࡧʹ͓͍ ͯɼੑೳΛ֬ೝ 25 <.BODP>$0/53"45*7&"6%*0-"/(6"(&-&"3/*/('03.64*$*.BODPFUBM*4.*3 <)VBOH>.6-"/"+0*/5&.#&%%*/(0'.64*$"6%*0"/%/"563"- -"/(6"(&2)VBOHFUBM*4.*3 -
M&LͷରরֶशϞσϧ: CLaMP [SWu 23] • Իָ (Symbolic) ͱςΩετͷϚϧνϞʔμ ϧֶशϞσϧ •
ରরֶश͕ϕʔε • M3ͱ͍͏ABC notationϕʔεͰָේͷࣗݾڭࢣ͋Γ ֶशϞσϧซͤͯఏҊ • WikiMusicDatasetͱ͍͏ָේʹର͢ΔΩϟϓγϣϯ σʔληοτΛఏҊ • ςΩετָේݕࡧΛՄೳʹ • θϩγϣοτྨ/సஔֶशͰྑੑೳ 26 <48V>$-B.1$POUSBTUJWF-BOHVBHF.VTJD1SFUSBJOJOHGPS$SPTT.PEBM 4ZNCPMJD.VTJD*OGPSNBUJPO3FUSJFWBM 4IBOHEB8VFUBM*4.*3 -
27 • Captioning • MusicCaps https://www.kaggle.com/datasets/googleai/musiccaps • LP-MusicCaps-MC https://huggingface.co/datasets/seungheondoh/LP-MusicCaps-MC •
LP-MusicCaps-MSD https://huggingface.co/datasets/seungheondoh/LP-MusicCaps-MSD • LP-MusicCaps-MTT https://huggingface.co/datasets/seungheondoh/LP-MusicCaps-MTT • MusicBench https://huggingface.co/datasets/amaai-lab/MusicBench • The song describer dataset https://github.com/mulab-mir/song-describer-dataset • WikiMuTe https://huggingface.co/datasets/davanstrien/WikiMuTe • Question-answering • Music-AVQA https://gewu-lab.github.io/MUSIC-AVQA/ • MusicQA https://crypto-code.github.io/MU-LLaMA-Demo/#dataget ͑Δσʔληοτͷ·ͱΊ -
ײ a.k.a. ϞϠϞϠᶃ: ͳͥࣗવݴޠ͔ʁ • ԻָͱࣗવݴޠΛ݁ͼ͚ͭΔࢫΈɼԻը૾ͷͦΕͱҟͳΔʁ • ٕज़తʹɼΓ্͕͍ͬͯΔࣗવݴޠॲཧͱ༥߹͢Δͷ໘ന͍ • ʮԻָͱʯ༥߹ͯ͠خ͍͜͠ͱ͋Δʁ
• Իָͱࣗવݴޠͦͦ݁ͼ͚Ͱ͖Δͷ͔ • ࡢͷฏాઌੜͷ͓ͷதʹ͋ͬͨ௨Γ྆ऀඇՄٯ • ໘ന͍, ͱ͍͏Ҏ্ʹԿΛͨΒͩ͢Ζ͏͔ 28 -
ײ a.k.a. ϞϠϞϠᶄ: ࣗͷλεΫʹಛԽ͍ͤͨ͞߹ • ΩϟϓγϣϯʹೖΔ͖ཁ݅Կʁ • Իָʹ͞·͟·ͳଐੑΛ͍࣋ͬͯΔ • Ͳ͜·ͰΛهड़͢Δ͖͔ʁ
• Իݯ͔ΒΘ͔Δใ (Acoustics)ɼͲͷΑ͏ʹফඅ͞Εͨʢ͞ΕΔʣ͔ͷ ใ (Cultural)ɼϝλσʔλใ (Editorial)ʹେผͰ͖Δ [Pachet 05] • Ͳ͏ΩϟϓγϣϯΛΉ͖͔ • ΩϟϓγϣϯΛهड़͢Δ୯ҐۂશମͰOK? • ࠁҰࠁͰঢ়ଶมԽ͢Δ 29 [Pachet 05]Pachet, F., 2005. Knowledge management and musical metadata. Idea Group, 12. -
ԻָͱࣗવݴޠॲཧͷϚϧνϞʔμϧֶशͷαʔϕΠ 30 • ಈΛհ • Իָ -> ςΩετ (Captioning, Question
& Answering) • Իָ + ςΩετͷϚϧνϞʔμϧදݱֶश • ૉʹࢥ͍ͬͯΔٙΛͿͪ·͚ • ͦͦతͳଟ͍ͱࢥ͏ͷͰɼٞͷՐछͱͳΕ͍Ͱ͢ ·ͱΊ