Share
Embed
【輪講資料】SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval
ACL2023読み会@名大で発表した資料です。
Yano
September 28, 2023
Transcript
35 slides
-
SIMLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval 9݄28
ACLಡΈձ ాɾݚڀࣨɹM2 ઍߛ Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao Linjun Yang, Daxin Jiang, Rangan Majumder, Furu Wei ACL 2023 -
બΜͩཧ༝ • LLMͱ૬ੑͷྑ͍ݕࡧϞσϧʹֶ͍ͭͯͼ͍ͨ • LLMΛՃֶश͠ͳ͘ͱࣝΛૠೖͰ͖Δ • HallucinationΛ͙͜ͱ͕Ͱ͖Δ 4 --. 3FUSJFWFS
֎෦ %BUB 3FUSJFWBM"VHNFOUFE(FOFSBUJPO 3"( ͷྫ ʁ 📙 🔎 ࠓͷηɾϦʔά ༏উٿஂͲ͜ʁ ࡕਆλΠΨʔε͕18ͿΓ 6ͷηɾϦʔά༏উ -
બΜͩཧ༝ • LLMͱ૬ੑͷྑ͍ݕࡧϞσϧʹֶ͍ͭͯͼ͍ͨ • LLMΛՃֶश͠ͳ͘ͱࣝΛૠೖͰ͖Δ • HallucinationΛ͙͜ͱ͕Ͱ͖Δ 5 --. 3FUSJFWFS
֎෦ %BUB 3FUSJFWBM"VHNFOUFE(FOFSBUJPO 3"( ͷྫ --. ʁ ࠓͷηɾϦʔά༏উٿஂͲ͜ʁ 📙ࢀߟจݙɿ ࡕਆλΠΨʔε͕ͿΓͷʜ 📙 🔎 ࠓͷηɾϦʔά ༏উٿஂͲ͜ʁ ݕࡧͨ͠ࢀߟจݙΛ --.ʹ͢ -
બΜͩཧ༝ • LLMͱ૬ੑͷྑ͍ݕࡧϞσϧʹֶ͍ͭͯͼ͍ͨ • LLMΛՃֶश͠ͳ͘ͱࣝΛૠೖͰ͖Δ • HallucinationΛ͙͜ͱ͕Ͱ͖Δ 6 --. 3FUSJFWFS
֎෦ %BUB 3FUSJFWBM"VHNFOUFE(FOFSBUJPO 3"( ͷྫ --. ʁ ࠓͷηɾϦʔά༏উٿஂͲ͜ʁ 📙ࢀߟจݙɿ ࡕਆλΠΨʔε͕ͿΓͷʜ 📙 🔎 ࠓͷηɾϦʔά ༏উٿஂͲ͜ʁ ࡕਆλΠΨʔε ݕࡧͨ͠ࢀߟจݙΛ --.ʹ͢ -
Passage retrieval • େྔͷύοηʔδ܈͔ΒΫΤϦʹؔ࿈͢ΔύοηʔδΛݕࡧͯ͠ ͘Δ ✦ ૄϕΫτϧݕࡧ • Ωʔϫʔυݕࡧ •
(ྫ)ɿTF-IDF, BM25 ✦ ີϕΫτϧݕࡧ • ϕΫτϧʹม͠ɺͦͷྨࣅͰݕࡧ • (ྫ)ɿOpenAIͷEmbedding APIͷίαΠϯྨࣅͰݕࡧ͢Δ 7 -
Passage retrieval ✦ ີϕΫτϧݕࡧ • DPR • ΫΤϦ༻ͱจষ༻ͷ2ͭͷΤϯίʔμΛར༻͢Δ (Bi/Dual-encoder) •
FaissʹΑΔۙ୳ࡧΛߦ͏͜ͱͰΫΤϦจষϖΞΛݟ͚ͭΔ • ColBERT • ΫΤϦɺจষશମͰͳ͘ɺτʔΫϯϨϕϧͷϕΫτϧΛར༻ͯ͠ΑΓߴਫ਼ ͳྨࣅΛܭࢉ͢Δ (multi vector) • อଘ͢Δ͖ใ͕ଟ͘ɺ֨ೲίετ͕ߴ͍ʢSimLMͱൺֱͯ͠6ഒఔʣ • Condenser: • ΫΤϦɺจষશମΛϕΫτϧͱ͢ΔࡍͷදݱϘτϧωοΫ ([CLS]ͳͲ) ͕Α ΓҙຯใΛѹॖͨ͠දݱʹͳΔΑ͏ʹࣄલֶशΛߦ͏ 8 -
SimLMɿࣄલֶश 10 You never know what you’re going to get
in life. You never [MASK] what you’re going to [MASK] in life. ੜϞσϧ ( fi x) You never expect what you’re going to be in life. [CLS] You never tell what you’re going to learn in life. Τϯίʔμ ઙ͍σίʔμ [CLS]දݱ ஔલτʔΫϯͷ༧ଌ Lenc = − 1 |x| |x| ∑ i=1 log p(x[i]|xenc ) ஔલτʔΫϯͷ༧ଌ Ldec = − 1 |x| |x| ∑ i=1 log p(x[i]|xdec ) x xenc xdec -
1. ֬pͰจͷҰ෦ΛϚεΫ͢Δ 2. ੜϞσϧʹΑΓ[MASK]Λ ͦΕͧΕஔ͠ɺΤϯίʔμɺ σίʔμͷೖྗͱ͢Δ SimLMɿࣄલֶश You never know
what you’re going to get in life. You never [MASK] what you’re going to [MASK] in life. ੜϞσϧ ( fi x) You never expect what you’re going to be in life. [CLS] You never tell what you’re going to learn in life. [CLS]දݱ Τϯίʔμ ઙ͍σίʔμ 11 -
3. ΤϯίʔμจશମΛड͚औΓɺ શͯͷτʔΫϯʹ͍ͭͯஔલΛ ༧ଌ͢Δ SimLMɿࣄલֶश 4. σίʔμΤϯίʔμͷ [CLS]දݱͱจશମΛड͚औΓɺ શͯͷτʔΫϯʹ͍ͭͯஔલΛ ༧ଌ͢Δ
You never know what you’re going to get in life. You never expect what you’re going to be in life. [CLS] You never tell what you’re going to learn in life. Τϯίʔμ ઙ͍σίʔμ [CLS]දݱ ஔલτʔΫϯͷ༧ଌ L = − 1 |x| |x| ∑ i=1 log p(x[i]|xinput ) [MASK]ஔͱੜ 12 -
3. ΤϯίʔμจશମΛड͚औΓɺ શͯͷτʔΫϯʹ͍ͭͯஔલΛ ༧ଌ͢Δ SimLMɿࣄલֶश 4. σίʔμΤϯίʔμͷ [CLS]දݱͱจશମΛड͚औΓɺ શͯͷτʔΫϯʹ͍ͭͯஔલΛ ༧ଌ͢Δ
You never know what you’re going to get in life. You never expect what you’re going to be in life. [CLS] You never tell what you’re going to learn in life. Τϯίʔμ ઙ͍σίʔμ [CLS]දݱ [MASK]ஔͱੜ ஔલτʔΫϯͷ༧ଌ L = − 1 |x| |x| ∑ i=1 log p(x[i]|xinput ) 2ͷΈͷTransformerͳͷͰ දݱྗʹ͘͠ɺΤϯίʔμͷ [CLS]දݱʹཔΔ͔͠ͳ͍ 13 -
3. ΤϯίʔμจશମΛड͚औΓɺ શͯͷτʔΫϯʹ͍ͭͯஔલΛ ༧ଌ͢Δ SimLMɿࣄલֶश 4. σίʔμΤϯίʔμͷ [CLS]දݱͱจશମΛड͚औΓɺ શͯͷτʔΫϯʹ͍ͭͯஔલΛ ༧ଌ͢Δ
You never know what you’re going to get in life. You never expect what you’re going to be in life. [CLS] You never tell what you’re going to learn in life. Τϯίʔμ ઙ͍σίʔμ [CLS]දݱ [MASK]ஔͱੜ ஔલτʔΫϯͷ༧ଌ L = − 1 |x| |x| ∑ i=1 log p(x[i]|xinput ) 2ͷΈͷTransformerͳͷͰ දݱྗʹ͘͠ɺΤϯίʔμͷ [CLS]දݱʹཔΔ͔͠ͳ͍ ΑΓจҙΛѹॖͨ͠[CLS]Λ ग़ྗ͢ΔΑ͏ʹͳΔ 14 -
3. ΤϯίʔμจશମΛड͚औΓɺ શͯͷτʔΫϯʹ͍ͭͯஔલΛ ༧ଌ͢Δ SimLMɿࣄલֶश 4. σίʔμΤϯίʔμͷ [CLS]දݱͱจશମΛड͚औΓɺ શͯͷτʔΫϯʹ͍ͭͯஔલΛ ༧ଌ͢Δ
You never know what you’re going to get in life. You never expect what you’re going to be in life. [CLS] You never tell what you’re going to learn in life. ઙ͍σίʔμ [CLS]දݱ [MASK]ஔͱੜ ஔલτʔΫϯͷ༧ଌ L = − 1 |x| |x| ∑ i=1 log p(x[i]|xinput ) Τϯίʔμ ࠓޙͷֶशɺਪͰ ΤϯίʔμͷΈར༻ 15 -
SimLMɿ Fine-tuning • ಉ͡σʔλͰෳճֶश • લεςʔδͷϞσϧग़ྗΛֶशʹར༻͠ɺͲΜͲΜੑೳΛ্͛Δ 16 Retriever 1 Retriever
2 Retriever distill Re-ranker (cross encoder) ࣄલֶशࡁΈΤϯίʔμͰॳظԽ #.Ͱෛྫநग़ ෛྫநग़ ෛྫநग़ ৠཹ %BUB -
SimLMɿ Fine-tuning 17 ✓ Retriever1 BM25ʢૄϕΫτϧݕࡧͷख๏ʣͰநग़͞Εͨhard negativeΛར༻͠ɺରর ֶशΛߦ͏ • ΫΤϦʹରͯ͠ྨࣅͷߴ͍n͔݅Βਖ਼ղҎ֎ͷͷΛhard
negativeͱ͢Δ Retriever 1 Retriever 2 Retriever distill Re-ranker ࣄલֶशࡁΈΤϯίʔμͰॳظԽ #.Ͱෛྫநग़ ෛྫநग़ ෛྫநग़ ৠཹ %BUB -
ରরֶश 18 positive passage query negative passage (queryͱྨࣅ͕ߴ͍͕) ෆਖ਼ղ ਖ਼ղ
QPTJUJWF IBSEOFHBUJWF • positiveϖΞͷྨࣅ͕ߴ͘ɺnegativeϖΞͷྨࣅ͕͘ͳΔΑ͏ʹֶ श -
ରরֶश 19 positive passage query negative passage ਖ਼ղ QPTJUJWF (queryͱྨࣅ͕ߴ͍͕)
ෆਖ਼ղ positive passage query negative passage ਖ਼ղ (queryͱྨࣅ͕ߴ͍͕) ෆਖ਼ղ positive passage query negative passage ਖ਼ղ (queryͱྨࣅ͕ߴ͍͕) ෆਖ਼ղ όον αΠζ ɾɾɾ IBSEOFHBUJWF JOCBUDI OFHBUJWFT -
ରরֶश • : ਖ਼ղͷquery, passageϖΞ • : ྨࣅؔ ίαΠϯྨࣅʹԹύϥϝʔλΛಋೖͨ͠ͷ •
: શෛྫͷू߹ (q+, d+) ϕ ℕ 20 Lcnt = − log ϕ(q+, d+) ϕ(q+, d+) + ∑ ni ∈ℕ (ϕ(q+, ni ) + ϕ(d+, ni )) -
SimLMɿ Fine-tuning 21 ✓ Retriever1 BM25ʢૄϕΫτϧݕࡧͷख๏ʣͰநग़͞Εͨhard negativeΛར༻͠ɺରর ֶशΛߦ͏ • ΫΤϦʹରͯ͠ྨࣅͷߴ͍n͔݅Βਖ਼ղҎ֎ͷͷΛhard
negativeͱ͢Δ Retriever 1 Retriever 2 Retriever distill Re-ranker ࣄલֶशࡁΈΤϯίʔμͰॳظԽ #.Ͱෛྫநग़ ෛྫநग़ ෛྫநग़ ৠཹ %BUB -
SimLMɿ Fine-tuning 22 ✓ Retriever 2 Retriever 1 Ͱநग़͞Εͨhard negativeΛར༻͠ɺରরֶशΛߦ͏
Retriever 1 Retriever 2 Retriever distill Re-ranker ࣄલֶशࡁΈΤϯίʔμͰॳظԽ #.Ͱෛྫநग़ ෛྫநग़ ෛྫநग़ ৠཹ %BUB -
SimLMɿ Fine-tuning 23 ✓ Re-ranker Retriever 2 Ͱநग़͞Εͨhard negativeΛར༻͠ɺਖ਼ྫͷείΞ͕ߴ͘ɺෛྫ ͷείΞ͕͘ͳΔ༷ʹֶश͢Δ
Retriever 1 Retriever 2 Retriever distill Re-ranker ࣄલֶशࡁΈΤϯίʔμͰॳظԽ #.Ͱෛྫநग़ ෛྫநग़ ෛྫநग़ ৠཹ %BUB -
SimLMɿ Fine-tuning 24 ✓ Retriever distill Re-ranker͔Βࣝৠཹ • cross-encoderϕʔεͷϞσϧڧྗ͕ͩɺඇ࣮༻త •
ݕࡧ͢ΔͨͼʹશͯͷจষΛϕΫτϧԽ͠ͳ͍ͱ͍͚ͳ͍ Retriever 1 Retriever 2 Retriever distill Re-ranker ࣄલֶशࡁΈΤϯίʔμͰॳظԽ #.Ͱෛྫநग़ ෛྫநग़ ෛྫநग़ ৠཹ %BUB -
SimLMɿ Fine-tuning 25 ✓ Retriever distill ;ͨͭͷଛࣦͷઢܗิΛଛࣦͱͯ͠ར༻ֶ͠श • Retriever 2
Ͱநग़͞Εͨhard negativeΛར༻ͨ͠Contrastive Loss • Re-rankerͷ༩͢ΔείΞͱͷKL divergence • Re-rankerͱRetrieverͷ༩͢ΔείΞ͕ۙ͘ͳΔΑ͏ʹֶश Lkl = n ∑ i=1 pi ranker log pi ranker pi ret L = Lcnt + αLkl -
σʔληοτ • MS MARCO • ͱͯେنͳ࣭Ԡσʔληοτ • 50ສ݅ͷϥϕϧ͖ΫΤϦͱ880ສ݅ͷจষ͔Βߏ • NQ
• 8ສͷ࣭ɺճϖΞͱWikipediaதͷ210ສ݅ͷจষ͔Βߏ͞Ε Δ࣭Ԡσʔληοτ 26 -
ධՁࢦඪ (ྫ) MRR@3 27 1 3 × ( 1 3
+ 1 2 ) = 0.278 • Recall • ༧ଌʹਖ਼ղؚ͕·ΕΔ֬ (ྫ) R@3 2 3 = 0.667 ༧ଌ είΞ߱ॱ ਖ਼ղ ॱҐ EPH DBU JOV JOV NJLBO NBLF NBO NBLF POF UXP UFO [FSP • Mean reciprocal rank (MRR) • ਖ਼ղͨ͠༧ଌʹ͍ͭͯͦͷॱҐͷٯͷฏۉ -
ࣄલֶश࣌ͷతؔʹ͍ͭͯ • Enc-Dec MLM: ಉ͡ΞʔΩςΫνϟͰੜثΛ༻ͤͣɺ[MASK]෦ͷ༧ଌΛߦ͏ • Condenser: දݱϘτϧωοΫʹจҙΛѹॖ͢ΔΑ͏ͳࣄલֶशΞʔΩςΫνϟ (ઙ ͍ͷจશମͷදݱͱਂ͍ͷ[CLS]දݱΛͬͯMLMͰֶश)
• MLM: ΤϯίʔμͷΈ[MASK]෦ͷ༧ଌΛߦ͏ • Enc-Dec RTD: ಉ͡ΞʔΩςΫνϟͰஔτʔΫϯͷݕग़Λߦ͏ • AutoEncoder: Τϯίʔμͷ[CLS]͔Βσίʔμ͕ೖྗΛ෮ݩ͢Δ 31 .4."3$0ͰධՁ -
ࣄલֶश࣌ͷతؔʹ͍ͭͯ • ಉ͡ΞʔΩςΫνϟͰൺֱ͢ΔͱɺSimLM > Enc-Dec MLM >> Enc-Dec RTD •
ݕࡧλεΫʹ͓͍ͯ MLM > RTD Ͱ͋Δ • (ࢲ৴) SimLMͷతؔஔલͷ༧ଌͰ͋ΓɺMLMͱੑೳ͕ۙ ͍͜ͱ͔Βࣝผ͢ΔΑ͏ͳλεΫΑΓτʔΫϯΛ༧ଌ͢ΔΑ͏ͳ λεΫͷํ͕ྑ͍ʁ 32 .4."3$0ͰධՁ -
·ͱΊ • ີϕΫτϧݕࡧͷͨΊͷࣄલֶशख๏SimLMͷఏҊ • Τϯίʔμͷग़ྗͷ͏ͪɺදݱϘτϧωοΫ (ࠓճ[CLS]) Λઙ͍σίʔμ ʹ͢͜ͱͰɺશͯͷҙຯใΛ දݱϘτϧωοΫʹѹॖ͢Δ͜ͱ Λֶशͤ͞ɺྑͳੑೳΛࣔͨ͠
34 You never know what you’re going to get in life. You never [MASK] what you’re going to [MASK] in life. ੜϞσϧ ( fi x) You never expect what you’re going to be in life. [CLS] You never tell what you’re going to learn in life. Τϯίʔμ ઙ͍σίʔμ [CLS]දݱ