【輪講資料】SIMLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval

ACL2023読み会@名大で発表した資料です。

Transcript

1. SIMLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval 9݄28೔ ACLಡΈձ ෢ాɾ࡫໺ݚڀࣨɹM2 ໼໺ઍߛ Liang Wang, Nan Yang, Xiaolong Huang, 
 Binxing Jiao Linjun Yang, Daxin Jiang, Rangan Majumder, Furu Wei ACL 2023
2. SimLM • ີϕΫτϧݕࡧͷͨΊͷࣄલֶशख๏SimLMͷఏҊ 2
3. બΜͩཧ༝ • LLMͱ૬ੑͷྑ͍ݕࡧϞσϧʹֶ͍ͭͯͼ͍ͨ • LLMΛ௥Ճֶश͠ͳ͘ͱ΋஌ࣝΛૠೖͰ͖Δ • HallucinationΛ๷͙͜ͱ͕Ͱ͖Δ 3 --. 3FUSJFWBM
   "VHNFOUFE
   (FOFSBUJPO
   3"(
   ͷྫ ʁ ࠓ೥ͷηɾϦʔά 
 ༏উٿஂ͸Ͳ͜ʁ
4. બΜͩཧ༝ • LLMͱ૬ੑͷྑ͍ݕࡧϞσϧʹֶ͍ͭͯͼ͍ͨ • LLMΛ௥Ճֶश͠ͳ͘ͱ΋஌ࣝΛૠೖͰ͖Δ • HallucinationΛ๷͙͜ͱ͕Ͱ͖Δ 4 --. 3FUSJFWFS ֎෦ 
 %BUB 3FUSJFWBM
   "VHNFOUFE
   (FOFSBUJPO
   3"(
   ͷྫ ʁ 📙 🔎 ࠓ೥ͷηɾϦʔά 
 ༏উٿஂ͸Ͳ͜ʁ ࡕਆλΠΨʔε͕18೥ͿΓ 6౓໨ͷηɾϦʔά༏উ
5. બΜͩཧ༝ • LLMͱ૬ੑͷྑ͍ݕࡧϞσϧʹֶ͍ͭͯͼ͍ͨ • LLMΛ௥Ճֶश͠ͳ͘ͱ΋஌ࣝΛૠೖͰ͖Δ • HallucinationΛ๷͙͜ͱ͕Ͱ͖Δ 5 --. 3FUSJFWFS ֎෦ 
 %BUB 3FUSJFWBM
   "VHNFOUFE
   (FOFSBUJPO
   3"(
   ͷྫ --. ʁ ࠓ೥ͷηɾϦʔά༏উٿஂ͸Ͳ͜ʁ 
 📙ࢀߟจݙɿ 
 ࡕਆλΠΨʔε͕೥ͿΓ౓໨ͷʜ 📙 🔎 ࠓ೥ͷηɾϦʔά 
 ༏উٿஂ͸Ͳ͜ʁ ݕࡧͨ͠ࢀߟจݙΛ 
 --.ʹ౉͢
6. બΜͩཧ༝ • LLMͱ૬ੑͷྑ͍ݕࡧϞσϧʹֶ͍ͭͯͼ͍ͨ • LLMΛ௥Ճֶश͠ͳ͘ͱ΋஌ࣝΛૠೖͰ͖Δ • HallucinationΛ๷͙͜ͱ͕Ͱ͖Δ 6 --. 3FUSJFWFS ֎෦ 
 %BUB 3FUSJFWBM
   "VHNFOUFE
   (FOFSBUJPO
   3"(
   ͷྫ --. ʁ ࠓ೥ͷηɾϦʔά༏উٿஂ͸Ͳ͜ʁ 
 📙ࢀߟจݙɿ 
 ࡕਆλΠΨʔε͕೥ͿΓ౓໨ͷʜ 📙 🔎 ࠓ೥ͷηɾϦʔά 
 ༏উٿஂ͸Ͳ͜ʁ ࡕਆλΠΨʔε ݕࡧͨ͠ࢀߟจݙΛ 
 --.ʹ౉͢
7. Passage retrieval • େྔͷύοηʔδ܈͔ΒΫΤϦʹؔ࿈͢ΔύοηʔδΛݕࡧͯ͠ ͘Δ ✦ ૄϕΫτϧݕࡧ • Ωʔϫʔυݕࡧ • (ྫ)ɿTF-IDF, BM25 ✦ ີϕΫτϧݕࡧ • ϕΫτϧʹม׵͠ɺͦͷྨࣅ౓Ͱݕࡧ • (ྫ)ɿOpenAIͷEmbedding APIͷίαΠϯྨࣅ౓Ͱݕࡧ͢Δ 7
8. Passage retrieval ✦ ີϕΫτϧݕࡧ • DPR • ΫΤϦ༻ͱจষ༻ͷ2ͭͷΤϯίʔμΛར༻͢Δ (Bi/Dual-encoder) • FaissʹΑΔۙ๣୳ࡧΛߦ͏͜ͱͰΫΤϦจষϖΞΛݟ͚ͭΔ • ColBERT • ΫΤϦɺจষશମͰ͸ͳ͘ɺτʔΫϯϨϕϧͷϕΫτϧΛར༻ͯ͠ΑΓߴਫ਼ ౓ͳྨࣅ౓Λܭࢉ͢Δ (multi vector) • อଘ͢Δ΂͖৘ใ͕ଟ͘ɺ֨ೲίετ͕ߴ͍ʢSimLMͱൺֱͯ͠6ഒఔ౓ʣ • Condenser: • ΫΤϦɺจষશମΛϕΫτϧͱ͢ΔࡍͷදݱϘτϧωοΫ ([CLS]ͳͲ) ͕Α Γҙຯ৘ใΛѹॖͨ͠දݱʹͳΔΑ͏ʹࣄલֶशΛߦ͏ 8
9. SimLM • ີϕΫτϧݕࡧͷͨΊͷࣄલֶशख๏SimLMͷఏҊ • Τϯίʔμͷग़ྗ͢ΔදݱϘτϧωοΫ͕ɺશͯͷҙຯ৘ใΛѹॖ ͢Δ͜ͱΑ͏ʹֶशͤ͞ɺྑ޷ͳੑೳΛࣔͨ͠ • ରরֶशͱ஌ࣝৠཹʹΑΔFine-tuningΛߦ͏͜ͱͰɺColBERTͷ Α͏ͳmulti vectorͳख๏ʹউΔੑೳΛࣔͨ͠ 9
10. SimLMɿࣄલֶश 10 You never know what you’re going to get in life. You never [MASK] what you’re going to [MASK] in life. ੜ੒Ϟσϧ ( fi x) You never expect what you’re going to be in life. [CLS] You never tell what you’re going to learn in life. Τϯίʔμ ઙ͍σίʔμ [CLS]දݱ ஔ׵લτʔΫϯͷ༧ଌ Lenc = − 1 |x| |x| ∑ i=1 log p(x[i]|xenc ) ஔ׵લτʔΫϯͷ༧ଌ Ldec = − 1 |x| |x| ∑ i=1 log p(x[i]|xdec ) x xenc xdec
11. 1. ֬཰pͰจͷҰ෦ΛϚεΫ͢Δ 2. ੜ੒ϞσϧʹΑΓ[MASK]Λ
 ͦΕͧΕஔ׵͠ɺΤϯίʔμɺ σίʔμ΁ͷೖྗͱ͢Δ SimLMɿࣄલֶश You never know what you’re going to get in life. You never [MASK] what you’re going to [MASK] in life. ੜ੒Ϟσϧ ( fi x) You never expect what you’re going to be in life. [CLS] You never tell what you’re going to learn in life. [CLS]දݱ Τϯίʔμ ઙ͍σίʔμ 11
12. 3. Τϯίʔμ͸จશମΛड͚औΓɺ
 શͯͷτʔΫϯʹ͍ͭͯஔ׵લΛ ༧ଌ͢Δ SimLMɿࣄલֶश 4. σίʔμ͸Τϯίʔμͷ
 [CLS]දݱͱจશମΛड͚औΓɺ
 શͯͷτʔΫϯʹ͍ͭͯஔ׵લΛ ༧ଌ͢Δ You never know what you’re going to get in life. You never expect what you’re going to be in life. [CLS] You never tell what you’re going to learn in life. Τϯίʔμ ઙ͍σίʔμ [CLS]දݱ ஔ׵લτʔΫϯͷ༧ଌ L = − 1 |x| |x| ∑ i=1 log p(x[i]|xinput ) [MASK]ஔ׵ͱੜ੒ 12
13. 3. Τϯίʔμ͸จશମΛड͚औΓɺ
 શͯͷτʔΫϯʹ͍ͭͯஔ׵લΛ ༧ଌ͢Δ SimLMɿࣄલֶश 4. σίʔμ͸Τϯίʔμͷ
 [CLS]දݱͱจશମΛड͚औΓɺ
 શͯͷτʔΫϯʹ͍ͭͯஔ׵લΛ ༧ଌ͢Δ You never know what you’re going to get in life. You never expect what you’re going to be in life. [CLS] You never tell what you’re going to learn in life. Τϯίʔμ ઙ͍σίʔμ [CLS]දݱ [MASK]ஔ׵ͱੜ੒ ஔ׵લτʔΫϯͷ༧ଌ L = − 1 |x| |x| ∑ i=1 log p(x[i]|xinput ) 2૚ͷΈͷTransformerͳͷͰ දݱྗʹ๡͘͠ɺΤϯίʔμͷ [CLS]දݱʹཔΔ͔͠ͳ͍ 13
14. 3. Τϯίʔμ͸จશମΛड͚औΓɺ
 શͯͷτʔΫϯʹ͍ͭͯஔ׵લΛ ༧ଌ͢Δ SimLMɿࣄલֶश 4. σίʔμ͸Τϯίʔμͷ
 [CLS]දݱͱจશମΛड͚औΓɺ
 શͯͷτʔΫϯʹ͍ͭͯஔ׵લΛ ༧ଌ͢Δ You never know what you’re going to get in life. You never expect what you’re going to be in life. [CLS] You never tell what you’re going to learn in life. Τϯίʔμ ઙ͍σίʔμ [CLS]දݱ [MASK]ஔ׵ͱੜ੒ ஔ׵લτʔΫϯͷ༧ଌ L = − 1 |x| |x| ∑ i=1 log p(x[i]|xinput ) 2૚ͷΈͷTransformerͳͷͰ දݱྗʹ๡͘͠ɺΤϯίʔμͷ [CLS]දݱʹཔΔ͔͠ͳ͍ ΑΓจҙΛѹॖͨ͠[CLS]Λ
 ग़ྗ͢ΔΑ͏ʹͳΔ 14
15. 3. Τϯίʔμ͸จશମΛड͚औΓɺ
 શͯͷτʔΫϯʹ͍ͭͯஔ׵લΛ ༧ଌ͢Δ SimLMɿࣄલֶश 4. σίʔμ͸Τϯίʔμͷ
 [CLS]දݱͱจશମΛड͚औΓɺ
 શͯͷτʔΫϯʹ͍ͭͯஔ׵લΛ ༧ଌ͢Δ You never know what you’re going to get in life. You never expect what you’re going to be in life. [CLS] You never tell what you’re going to learn in life. ઙ͍σίʔμ [CLS]දݱ [MASK]ஔ׵ͱੜ੒ ஔ׵લτʔΫϯͷ༧ଌ L = − 1 |x| |x| ∑ i=1 log p(x[i]|xinput ) Τϯίʔμ ࠓޙͷֶशɺਪ࿦Ͱ͸
 ΤϯίʔμͷΈར༻ 15
16. SimLMɿ Fine-tuning • ಉ͡σʔλͰෳ਺ճֶश • લεςʔδͷϞσϧग़ྗΛֶशʹར༻͠ɺͲΜͲΜੑೳΛ্͛Δ 16 Retriever 1 Retriever 2 Retriever distill Re-ranker
 (cross encoder) ࣄલֶशࡁΈΤϯίʔμͰॳظԽ #.Ͱෛྫநग़ ෛྫநग़ ෛྫநग़ ৠཹ %BUB
17. SimLMɿ Fine-tuning 17 ✓ Retriever1 BM25ʢૄϕΫτϧݕࡧͷख๏ʣͰநग़͞Εͨhard negativeΛར༻͠ɺରর ֶशΛߦ͏ • ΫΤϦʹରͯ͠ྨࣅ౓ͷߴ͍n͔݅Βਖ਼ղҎ֎ͷ΋ͷΛhard negativeͱ͢Δ Retriever 1 Retriever 2 Retriever distill Re-ranker ࣄલֶशࡁΈΤϯίʔμͰॳظԽ #.Ͱෛྫநग़ ෛྫநग़ ෛྫநग़ ৠཹ %BUB
18. ରরֶश 18 positive passage query negative passage (queryͱྨࣅ౓͕ߴ͍͕) ෆਖ਼ղ ਖ਼ղ QPTJUJWF IBSE
    OFHBUJWF • positiveϖΞͷྨࣅ౓͕ߴ͘ɺnegativeϖΞͷྨࣅ౓͕௿͘ͳΔΑ͏ʹֶ श
19. ରরֶश 19 positive passage query negative passage ਖ਼ղ QPTJUJWF (queryͱྨࣅ౓͕ߴ͍͕) ෆਖ਼ղ positive passage query negative passage ਖ਼ղ (queryͱྨࣅ౓͕ߴ͍͕) ෆਖ਼ղ positive passage query negative passage ਖ਼ղ (queryͱྨࣅ౓͕ߴ͍͕) ෆਖ਼ղ όον 
 αΠζ ɾɾɾ IBSE
    OFHBUJWF JOCBUDI 
 OFHBUJWFT
20. ରরֶश • : ਖ਼ղͷquery, passageϖΞ • : ྨࣅ౓ؔ਺ ίαΠϯྨࣅ౓ʹԹ౓ύϥϝʔλΛಋೖͨ͠΋ͷ • : શෛྫͷू߹ (q+, d+) ϕ ℕ 20 Lcnt = − log ϕ(q+, d+) ϕ(q+, d+) + ∑ ni ∈ℕ (ϕ(q+, ni ) + ϕ(d+, ni ))
21. SimLMɿ Fine-tuning 21 ✓ Retriever1 BM25ʢૄϕΫτϧݕࡧͷख๏ʣͰநग़͞Εͨhard negativeΛར༻͠ɺରর ֶशΛߦ͏ • ΫΤϦʹରͯ͠ྨࣅ౓ͷߴ͍n͔݅Βਖ਼ղҎ֎ͷ΋ͷΛhard negativeͱ͢Δ Retriever 1 Retriever 2 Retriever distill Re-ranker ࣄલֶशࡁΈΤϯίʔμͰॳظԽ #.Ͱෛྫநग़ ෛྫநग़ ෛྫநग़ ৠཹ %BUB
22. SimLMɿ Fine-tuning 22 ✓ Retriever 2 Retriever 1 Ͱநग़͞Εͨhard negativeΛར༻͠ɺରরֶशΛߦ͏ Retriever 1 Retriever 2 Retriever distill Re-ranker ࣄલֶशࡁΈΤϯίʔμͰॳظԽ #.Ͱෛྫநग़ ෛྫநग़ ෛྫநग़ ৠཹ %BUB
23. SimLMɿ Fine-tuning 23 ✓ Re-ranker Retriever 2 Ͱநग़͞Εͨhard negativeΛར༻͠ɺਖ਼ྫͷείΞ͕ߴ͘ɺෛྫ ͷείΞ͕௿͘ͳΔ༷ʹֶश͢Δ Retriever 1 Retriever 2 Retriever distill Re-ranker ࣄલֶशࡁΈΤϯίʔμͰॳظԽ #.Ͱෛྫநग़ ෛྫநग़ ෛྫநग़ ৠཹ %BUB
24. SimLMɿ Fine-tuning 24 ✓ Retriever distill Re-ranker͔Β஌ࣝৠཹ • cross-encoderϕʔεͷϞσϧ͸ڧྗ͕ͩɺඇ࣮༻త • ݕࡧ͢ΔͨͼʹશͯͷจষΛϕΫτϧԽ͠ͳ͍ͱ͍͚ͳ͍ Retriever 1 Retriever 2 Retriever distill Re-ranker ࣄલֶशࡁΈΤϯίʔμͰॳظԽ #.Ͱෛྫநग़ ෛྫநग़ ෛྫநग़ ৠཹ %BUB
25. SimLMɿ Fine-tuning 25 ✓ Retriever distill ;ͨͭͷଛࣦͷઢܗิ׬࿨Λଛࣦͱͯ͠ར༻ֶ͠श • Retriever 2 Ͱநग़͞Εͨhard negativeΛར༻ͨ͠Contrastive Loss • Re-rankerͷ෇༩͢ΔείΞ෼෍ͱͷKL divergence • Re-rankerͱRetrieverͷ෇༩͢ΔείΞ෼෍͕ۙ͘ͳΔΑ͏ʹֶश Lkl = n ∑ i=1 pi ranker log pi ranker pi ret L = Lcnt + αLkl
26. σʔληοτ • MS MARCO • ͱͯ΋େن໛ͳ࣭໰Ԡ౴σʔληοτ • 50ສ݅ͷϥϕϧ෇͖ΫΤϦͱ880ສ݅ͷจষ͔Βߏ੒ • NQ • 8ສͷ࣭໰ɺճ౴ϖΞͱWikipediaதͷ210ສ݅ͷจষ͔Βߏ੒͞Ε Δ࣭໰Ԡ౴σʔληοτ 26
27. ධՁࢦඪ (ྫ) MRR@3 27 1 3 × ( 1 3 + 1 2 ) = 0.278 • Recall • ༧ଌʹਖ਼ղؚ͕·ΕΔ֬཰ (ྫ) R@3 2 3 = 0.667 ༧ଌ είΞ߱ॱ ਖ਼ղ ॱҐ EPH
    DBU
    JOV JOV  NJLBO
    NBLF
    NBO NBLF  POF
    UXP
    UFO [FSP  • Mean reciprocal rank (MRR) • ਖ਼ղͨ͠༧ଌʹ͍ͭͯͦͷॱҐͷٯ਺ͷฏۉ
28. ੑೳධՁ࣮ݧ 28 ɾɾɾ
29. ੑೳධՁ࣮ݧ • multi-vector (จষͰ͸ͳ͘τʔΫϯ͝ͱʹͻͱͭͷϕΫτϧΛ֨ೲ͢Δ) ख๏Ͱ͋ΔColBERTΛ྇կ͢Δੑೳ 29 ɾɾɾ
30. ֶशύΠϓϥΠϯͷӨڹ෼ੳ • ෛྫͷநग़΍Re-rankerͷৠཹͰੑೳ޲্͍ͯ͠Δ • ෛྫΛநग़͢Δஈ֊Λ૿΍ͯ͠΋༗ҙͳੑೳͷมԽ͸ͳ͔ͬͨ 30 .4."3$0ͰධՁ
31. ࣄલֶश࣌ͷ໨తؔ਺ʹ͍ͭͯ • Enc-Dec MLM: ಉ͡ΞʔΩςΫνϟͰੜ੒ثΛ࢖༻ͤͣɺ[MASK]෦෼ͷ༧ଌΛߦ͏ • Condenser: දݱϘτϧωοΫʹจҙΛѹॖ͢ΔΑ͏ͳࣄલֶशΞʔΩςΫνϟ (ઙ ͍૚ͷจશମͷදݱͱਂ͍૚ͷ[CLS]දݱΛ࢖ͬͯMLMͰֶश) • MLM: ΤϯίʔμͷΈ[MASK]෦෼ͷ༧ଌΛߦ͏ • Enc-Dec RTD: ಉ͡ΞʔΩςΫνϟͰஔ׵τʔΫϯͷݕग़Λߦ͏ • AutoEncoder: Τϯίʔμͷ[CLS]͔Βσίʔμ͕ೖྗΛ෮ݩ͢Δ 31 .4."3$0ͰධՁ
32. ࣄલֶश࣌ͷ໨తؔ਺ʹ͍ͭͯ • ಉ͡ΞʔΩςΫνϟͰൺֱ͢ΔͱɺSimLM > Enc-Dec MLM >> Enc-Dec RTD • ݕࡧλεΫʹ͓͍ͯ͸ MLM > RTD Ͱ͋Δ • (ࢲ৴) SimLMͷ໨తؔ਺͸ஔ׵લͷ༧ଌͰ͋ΓɺMLMͱੑೳ͕ۙ ͍͜ͱ͔Βࣝผ͢ΔΑ͏ͳλεΫΑΓτʔΫϯΛ༧ଌ͢ΔΑ͏ͳ λεΫͷํ͕ྑ͍ʁ 32 .4."3$0ͰධՁ
33. ࣄલֶशίʔύεʹΑΔӨڹ • MS-MARCOͰ͸BM25ʹΑΔෛྫɺNQͰ͸நग़͞ΕͨෛྫΛར ༻ͨ͠Fine-tuningΛߦ͔ͬͯΒධՁ • ࣄલֶशͱFine-tuningɺධՁσʔλ͕ಉ࣌͡ߴ͍ੑೳ • טΈ߹͍ͬͯͳ͍࣌Ͱ΋ੑೳͷ޲্͸େ͖͘ɺࣄલֶशʹΑͬͯ ৘ใΛ[CLS]දݱʹѹॖ͢Δೳྗ͕֫ಘ͞Ε͍ͯΔ 33
34. ·ͱΊ • ີϕΫτϧݕࡧͷͨΊͷࣄલֶशख๏SimLMͷఏҊ • Τϯίʔμͷग़ྗͷ͏ͪɺදݱϘτϧωοΫ (ࠓճ͸[CLS]) Λઙ͍σίʔμ ʹ౉͢͜ͱͰɺશͯͷҙຯ৘ใΛ
 දݱϘτϧωοΫʹѹॖ͢Δ͜ͱ
 Λֶशͤ͞ɺྑ޷ͳੑೳΛࣔͨ͠ 34 You never know what you’re going to get in life. You never [MASK] what you’re going to [MASK] in life. ੜ੒Ϟσϧ ( fi x) You never expect what you’re going to be in life. [CLS] You never tell what you’re going to learn in life. Τϯίʔμ ઙ͍σίʔμ [CLS]දݱ
35. ײ૝ • ͱͯ΋ίετ͕͔͔ͬͯେมͦ͏ • ର৅σʔλ͝ͱʹࣄલֶश͕ඞཁͳͷ͸ਏ͍ • දݱϘτϧωοΫΛྑ͘͢ΔͨΊͷࣄલֶशΞʔΩςΫνϟ͕໘ന͍ • ݕࡧք۾ʹ͸৭ʑ͋Γͦ͏ʁ • CondenserͳͲ… 35