SIMLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval 9݄28೔ ACLಡΈձ ෢ాɾ࡫໺ݚڀࣨɹM2 ໼໺ઍߛ Liang Wang, Nan Yang, Xiaolong Huang, 
 Binxing Jiao Linjun Yang, Daxin Jiang, Rangan Majumder, Furu Wei ACL 2023

SimLM ● ີϕΫτϧݕࡧͷͨΊͷࣄલֶशख๏SimLMͷఏҊ 2

બΜͩཧ༝ ● LLMͱ૬ੑͷྑ͍ݕࡧϞσϧʹֶ͍ͭͯͼ͍ͨ ● LLMΛ௥Ճֶश͠ͳ͘ͱ΋஌ࣝΛૠೖͰ͖Δ ● HallucinationΛ๷͙͜ͱ͕Ͱ͖Δ 3 --. 3FUSJFWBM"VHNFOUFE(FOFSBUJPO 3"( ͷྫ ʁ ࠓ೥ͷηɾϦʔά 
 ༏উٿஂ͸Ͳ͜ʁ

બΜͩཧ༝ ● LLMͱ૬ੑͷྑ͍ݕࡧϞσϧʹֶ͍ͭͯͼ͍ͨ ● LLMΛ௥Ճֶश͠ͳ͘ͱ΋஌ࣝΛૠೖͰ͖Δ ● HallucinationΛ๷͙͜ͱ͕Ͱ͖Δ 4 --. 3FUSJFWFS ֎෦ 
 %BUB 3FUSJFWBM"VHNFOUFE(FOFSBUJPO 3"( ͷྫ ʁ 📙 🔎 ࠓ೥ͷηɾϦʔά 
 ༏উٿஂ͸Ͳ͜ʁ ࡕਆλΠΨʔε͕18೥ͿΓ 6౓໨ͷηɾϦʔά༏উ

બΜͩཧ༝ ● LLMͱ૬ੑͷྑ͍ݕࡧϞσϧʹֶ͍ͭͯͼ͍ͨ ● LLMΛ௥Ճֶश͠ͳ͘ͱ΋஌ࣝΛૠೖͰ͖Δ ● HallucinationΛ๷͙͜ͱ͕Ͱ͖Δ 5 --. 3FUSJFWFS ֎෦ 
 %BUB 3FUSJFWBM"VHNFOUFE(FOFSBUJPO 3"( ͷྫ --. ʁ ࠓ೥ͷηɾϦʔά༏উٿஂ͸Ͳ͜ʁ 
 📙ࢀߟจݙɿ 
 ࡕਆλΠΨʔε͕೥ͿΓ౓໨ͷʜ 📙 🔎 ࠓ೥ͷηɾϦʔά 
 ༏উٿஂ͸Ͳ͜ʁ ݕࡧͨ͠ࢀߟจݙΛ 
 --.ʹ౉͢

બΜͩཧ༝ ● LLMͱ૬ੑͷྑ͍ݕࡧϞσϧʹֶ͍ͭͯͼ͍ͨ ● LLMΛ௥Ճֶश͠ͳ͘ͱ΋஌ࣝΛૠೖͰ͖Δ ● HallucinationΛ๷͙͜ͱ͕Ͱ͖Δ 6 --. 3FUSJFWFS ֎෦ 
 %BUB 3FUSJFWBM"VHNFOUFE(FOFSBUJPO 3"( ͷྫ --. ʁ ࠓ೥ͷηɾϦʔά༏উٿஂ͸Ͳ͜ʁ 
 📙ࢀߟจݙɿ 
 ࡕਆλΠΨʔε͕೥ͿΓ౓໨ͷʜ 📙 🔎 ࠓ೥ͷηɾϦʔά 
 ༏উٿஂ͸Ͳ͜ʁ ࡕਆλΠΨʔε ݕࡧͨ͠ࢀߟจݙΛ 
 --.ʹ౉͢

Passage retrieval ● େྔͷύοηʔδ܈͔ΒΫΤϦʹؔ࿈͢ΔύοηʔδΛݕࡧͯ͠ ͘Δ ✦ ૄϕΫτϧݕࡧ ● Ωʔϫʔυݕࡧ ● (ྫ)ɿTF-IDF, BM25 ✦ ີϕΫτϧݕࡧ ● ϕΫτϧʹม׵͠ɺͦͷྨࣅ౓Ͱݕࡧ ● (ྫ)ɿOpenAIͷEmbedding APIͷίαΠϯྨࣅ౓Ͱݕࡧ͢Δ 7

Passage retrieval ✦ ີϕΫτϧݕࡧ ● DPR ● ΫΤϦ༻ͱจষ༻ͷ2ͭͷΤϯίʔμΛར༻͢Δ (Bi/Dual-encoder) ● FaissʹΑΔۙ๣୳ࡧΛߦ͏͜ͱͰΫΤϦจষϖΞΛݟ͚ͭΔ ● ColBERT ● ΫΤϦɺจষશମͰ͸ͳ͘ɺτʔΫϯϨϕϧͷϕΫτϧΛར༻ͯ͠ΑΓߴਫ਼ ౓ͳྨࣅ౓Λܭࢉ͢Δ (multi vector) ● อଘ͢Δ΂͖৘ใ͕ଟ͘ɺ֨ೲίετ͕ߴ͍ʢSimLMͱൺֱͯ͠6ഒఔ౓ʣ ● Condenser: ● ΫΤϦɺจষશମΛϕΫτϧͱ͢ΔࡍͷදݱϘτϧωοΫ ([CLS]ͳͲ) ͕Α Γҙຯ৘ใΛѹॖͨ͠දݱʹͳΔΑ͏ʹࣄલֶशΛߦ͏ 8

SimLM ● ີϕΫτϧݕࡧͷͨΊͷࣄલֶशख๏SimLMͷఏҊ ● Τϯίʔμͷग़ྗ͢ΔදݱϘτϧωοΫ͕ɺશͯͷҙຯ৘ใΛѹॖ ͢Δ͜ͱΑ͏ʹֶशͤ͞ɺྑ޷ͳੑೳΛࣔͨ͠ ● ରরֶशͱ஌ࣝৠཹʹΑΔFine-tuningΛߦ͏͜ͱͰɺColBERTͷ Α͏ͳmulti vectorͳख๏ʹউΔੑೳΛࣔͨ͠ 9

SimLMɿࣄલֶश 10 You never know what you’re going to get in life. You never [MASK] what you’re going to [MASK] in life. ੜ੒Ϟσϧ ( fi x) You never expect what you’re going to be in life. [CLS] You never tell what you’re going to learn in life. Τϯίʔμ ઙ͍σίʔμ [CLS]දݱ ஔ׵લτʔΫϯͷ༧ଌ Lenc = − 1 |x| |x| ∑ i=1 log p(x[i]|xenc ) ஔ׵લτʔΫϯͷ༧ଌ Ldec = − 1 |x| |x| ∑ i=1 log p(x[i]|xdec ) x xenc xdec

1. ֬཰pͰจͷҰ෦ΛϚεΫ͢Δ 2. ੜ੒ϞσϧʹΑΓ[MASK]Λ
 ͦΕͧΕஔ׵͠ɺΤϯίʔμɺ σίʔμ΁ͷೖྗͱ͢Δ SimLMɿࣄલֶश You never know what you’re going to get in life. You never [MASK] what you’re going to [MASK] in life. ੜ੒Ϟσϧ ( fi x) You never expect what you’re going to be in life. [CLS] You never tell what you’re going to learn in life. [CLS]දݱ Τϯίʔμ ઙ͍σίʔμ 11

3. Τϯίʔμ͸จશମΛड͚औΓɺ
 શͯͷτʔΫϯʹ͍ͭͯஔ׵લΛ ༧ଌ͢Δ SimLMɿࣄલֶश 4. σίʔμ͸Τϯίʔμͷ
 [CLS]දݱͱจશମΛड͚औΓɺ
 શͯͷτʔΫϯʹ͍ͭͯஔ׵લΛ ༧ଌ͢Δ You never know what you’re going to get in life. You never expect what you’re going to be in life. [CLS] You never tell what you’re going to learn in life. Τϯίʔμ ઙ͍σίʔμ [CLS]දݱ ஔ׵લτʔΫϯͷ༧ଌ L = − 1 |x| |x| ∑ i=1 log p(x[i]|xinput ) [MASK]ஔ׵ͱੜ੒ 12

3. Τϯίʔμ͸จશମΛड͚औΓɺ
 શͯͷτʔΫϯʹ͍ͭͯஔ׵લΛ ༧ଌ͢Δ SimLMɿࣄલֶश 4. σίʔμ͸Τϯίʔμͷ
 [CLS]දݱͱจશମΛड͚औΓɺ
 શͯͷτʔΫϯʹ͍ͭͯஔ׵લΛ ༧ଌ͢Δ You never know what you’re going to get in life. You never expect what you’re going to be in life. [CLS] You never tell what you’re going to learn in life. Τϯίʔμ ઙ͍σίʔμ [CLS]දݱ [MASK]ஔ׵ͱੜ੒ ஔ׵લτʔΫϯͷ༧ଌ L = − 1 |x| |x| ∑ i=1 log p(x[i]|xinput ) 2૚ͷΈͷTransformerͳͷͰ දݱྗʹ๡͘͠ɺΤϯίʔμͷ [CLS]දݱʹཔΔ͔͠ͳ͍ 13

3. Τϯίʔμ͸จશମΛड͚औΓɺ
 શͯͷτʔΫϯʹ͍ͭͯஔ׵લΛ ༧ଌ͢Δ SimLMɿࣄલֶश 4. σίʔμ͸Τϯίʔμͷ
 [CLS]දݱͱจશମΛड͚औΓɺ
 શͯͷτʔΫϯʹ͍ͭͯஔ׵લΛ ༧ଌ͢Δ You never know what you’re going to get in life. You never expect what you’re going to be in life. [CLS] You never tell what you’re going to learn in life. Τϯίʔμ ઙ͍σίʔμ [CLS]දݱ [MASK]ஔ׵ͱੜ੒ ஔ׵લτʔΫϯͷ༧ଌ L = − 1 |x| |x| ∑ i=1 log p(x[i]|xinput ) 2૚ͷΈͷTransformerͳͷͰ දݱྗʹ๡͘͠ɺΤϯίʔμͷ [CLS]දݱʹཔΔ͔͠ͳ͍ ΑΓจҙΛѹॖͨ͠[CLS]Λ
 ग़ྗ͢ΔΑ͏ʹͳΔ 14

3. Τϯίʔμ͸จશମΛड͚औΓɺ
 શͯͷτʔΫϯʹ͍ͭͯஔ׵લΛ ༧ଌ͢Δ SimLMɿࣄલֶश 4. σίʔμ͸Τϯίʔμͷ
 [CLS]දݱͱจશମΛड͚औΓɺ
 શͯͷτʔΫϯʹ͍ͭͯஔ׵લΛ ༧ଌ͢Δ You never know what you’re going to get in life. You never expect what you’re going to be in life. [CLS] You never tell what you’re going to learn in life. ઙ͍σίʔμ [CLS]දݱ [MASK]ஔ׵ͱੜ੒ ஔ׵લτʔΫϯͷ༧ଌ L = − 1 |x| |x| ∑ i=1 log p(x[i]|xinput ) Τϯίʔμ ࠓޙͷֶशɺਪ࿦Ͱ͸
 ΤϯίʔμͷΈར༻ 15

SimLMɿ Fine-tuning ● ಉ͡σʔλͰෳ਺ճֶश ● લεςʔδͷϞσϧग़ྗΛֶशʹར༻͠ɺͲΜͲΜੑೳΛ্͛Δ 16 Retriever 1 Retriever 2 Retriever distill Re-ranker
 (cross encoder) ࣄલֶशࡁΈΤϯίʔμͰॳظԽ #.Ͱෛྫநग़ ෛྫநग़ ෛྫநग़ ৠཹ %BUB

SimLMɿ Fine-tuning 17 ✓ Retriever1 BM25ʢૄϕΫτϧݕࡧͷख๏ʣͰநग़͞Εͨhard negativeΛར༻͠ɺରর ֶशΛߦ͏ ● ΫΤϦʹରͯ͠ྨࣅ౓ͷߴ͍n͔݅Βਖ਼ղҎ֎ͷ΋ͷΛhard negativeͱ͢Δ Retriever 1 Retriever 2 Retriever distill Re-ranker ࣄલֶशࡁΈΤϯίʔμͰॳظԽ #.Ͱෛྫநग़ ෛྫநग़ ෛྫநग़ ৠཹ %BUB

ରরֶश 18 positive passage query negative passage (queryͱྨࣅ౓͕ߴ͍͕) ෆਖ਼ղ ਖ਼ղ QPTJUJWF IBSEOFHBUJWF ● positiveϖΞͷྨࣅ౓͕ߴ͘ɺnegativeϖΞͷྨࣅ౓͕௿͘ͳΔΑ͏ʹֶ श

ରরֶश 19 positive passage query negative passage ਖ਼ղ QPTJUJWF (queryͱྨࣅ౓͕ߴ͍͕) ෆਖ਼ղ positive passage query negative passage ਖ਼ղ (queryͱྨࣅ౓͕ߴ͍͕) ෆਖ਼ղ positive passage query negative passage ਖ਼ղ (queryͱྨࣅ౓͕ߴ͍͕) ෆਖ਼ղ όον 
 αΠζ ɾɾɾ IBSEOFHBUJWF JOCBUDI 
 OFHBUJWFT

ରরֶश ● : ਖ਼ղͷquery, passageϖΞ ● : ྨࣅ౓ؔ਺ ίαΠϯྨࣅ౓ʹԹ౓ύϥϝʔλΛಋೖͨ͠΋ͷ ● : શෛྫͷू߹ (q+, d+) ϕ ℕ 20 Lcnt = − log ϕ(q+, d+) ϕ(q+, d+) + ∑ ni ∈ℕ (ϕ(q+, ni ) + ϕ(d+, ni ))

SimLMɿ Fine-tuning 21 ✓ Retriever1 BM25ʢૄϕΫτϧݕࡧͷख๏ʣͰநग़͞Εͨhard negativeΛར༻͠ɺରর ֶशΛߦ͏ ● ΫΤϦʹରͯ͠ྨࣅ౓ͷߴ͍n͔݅Βਖ਼ղҎ֎ͷ΋ͷΛhard negativeͱ͢Δ Retriever 1 Retriever 2 Retriever distill Re-ranker ࣄલֶशࡁΈΤϯίʔμͰॳظԽ #.Ͱෛྫநग़ ෛྫநग़ ෛྫநग़ ৠཹ %BUB

SimLMɿ Fine-tuning 22 ✓ Retriever 2 Retriever 1 Ͱநग़͞Εͨhard negativeΛར༻͠ɺରরֶशΛߦ͏ Retriever 1 Retriever 2 Retriever distill Re-ranker ࣄલֶशࡁΈΤϯίʔμͰॳظԽ #.Ͱෛྫநग़ ෛྫநग़ ෛྫநग़ ৠཹ %BUB

SimLMɿ Fine-tuning 23 ✓ Re-ranker Retriever 2 Ͱநग़͞Εͨhard negativeΛར༻͠ɺਖ਼ྫͷείΞ͕ߴ͘ɺෛྫ ͷείΞ͕௿͘ͳΔ༷ʹֶश͢Δ Retriever 1 Retriever 2 Retriever distill Re-ranker ࣄલֶशࡁΈΤϯίʔμͰॳظԽ #.Ͱෛྫநग़ ෛྫநग़ ෛྫநग़ ৠཹ %BUB

SimLMɿ Fine-tuning 24 ✓ Retriever distill Re-ranker͔Β஌ࣝৠཹ ● cross-encoderϕʔεͷϞσϧ͸ڧྗ͕ͩɺඇ࣮༻త ● ݕࡧ͢ΔͨͼʹશͯͷจষΛϕΫτϧԽ͠ͳ͍ͱ͍͚ͳ͍ Retriever 1 Retriever 2 Retriever distill Re-ranker ࣄલֶशࡁΈΤϯίʔμͰॳظԽ #.Ͱෛྫநग़ ෛྫநग़ ෛྫநग़ ৠཹ %BUB

SimLMɿ Fine-tuning 25 ✓ Retriever distill ;ͨͭͷଛࣦͷઢܗิ׬࿨Λଛࣦͱͯ͠ར༻ֶ͠श ● Retriever 2 Ͱநग़͞Εͨhard negativeΛར༻ͨ͠Contrastive Loss ● Re-rankerͷ෇༩͢ΔείΞ෼෍ͱͷKL divergence ● Re-rankerͱRetrieverͷ෇༩͢ΔείΞ෼෍͕ۙ͘ͳΔΑ͏ʹֶश Lkl = n ∑ i=1 pi ranker log pi ranker pi ret L = Lcnt + αLkl

σʔληοτ ● MS MARCO ● ͱͯ΋େن໛ͳ࣭໰Ԡ౴σʔληοτ ● 50ສ݅ͷϥϕϧ෇͖ΫΤϦͱ880ສ݅ͷจষ͔Βߏ੒ ● NQ ● 8ສͷ࣭໰ɺճ౴ϖΞͱWikipediaதͷ210ສ݅ͷจষ͔Βߏ੒͞Ε Δ࣭໰Ԡ౴σʔληοτ 26

ධՁࢦඪ (ྫ) MRR@3 27 1 3 × ( 1 3 + 1 2 ) = 0.278 ● Recall ● ༧ଌʹਖ਼ղؚ͕·ΕΔ֬཰ (ྫ) R@3 2 3 = 0.667 ༧ଌ είΞ߱ॱ ਖ਼ղ ॱҐ EPH DBU JOV JOV NJLBO NBLF NBO NBLF POF UXP UFO [FSP ● Mean reciprocal rank (MRR) ● ਖ਼ղͨ͠༧ଌʹ͍ͭͯͦͷॱҐͷٯ਺ͷฏۉ

ੑೳධՁ࣮ݧ 28 ɾɾɾ

ੑೳධՁ࣮ݧ ● multi-vector (จষͰ͸ͳ͘τʔΫϯ͝ͱʹͻͱͭͷϕΫτϧΛ֨ೲ͢Δ) ख๏Ͱ͋ΔColBERTΛ྇կ͢Δੑೳ 29 ɾɾɾ

ֶशύΠϓϥΠϯͷӨڹ෼ੳ ● ෛྫͷநग़΍Re-rankerͷৠཹͰੑೳ޲্͍ͯ͠Δ ● ෛྫΛநग़͢Δஈ֊Λ૿΍ͯ͠΋༗ҙͳੑೳͷมԽ͸ͳ͔ͬͨ 30 .4."3$0ͰධՁ

ࣄલֶश࣌ͷ໨తؔ਺ʹ͍ͭͯ ● Enc-Dec MLM: ಉ͡ΞʔΩςΫνϟͰੜ੒ثΛ࢖༻ͤͣɺ[MASK]෦෼ͷ༧ଌΛߦ͏ ● Condenser: දݱϘτϧωοΫʹจҙΛѹॖ͢ΔΑ͏ͳࣄલֶशΞʔΩςΫνϟ (ઙ ͍૚ͷจશମͷදݱͱਂ͍૚ͷ[CLS]දݱΛ࢖ͬͯMLMͰֶश) ● MLM: ΤϯίʔμͷΈ[MASK]෦෼ͷ༧ଌΛߦ͏ ● Enc-Dec RTD: ಉ͡ΞʔΩςΫνϟͰஔ׵τʔΫϯͷݕग़Λߦ͏ ● AutoEncoder: Τϯίʔμͷ[CLS]͔Βσίʔμ͕ೖྗΛ෮ݩ͢Δ 31 .4."3$0ͰධՁ

ࣄલֶश࣌ͷ໨తؔ਺ʹ͍ͭͯ ● ಉ͡ΞʔΩςΫνϟͰൺֱ͢ΔͱɺSimLM > Enc-Dec MLM >> Enc-Dec RTD ● ݕࡧλεΫʹ͓͍ͯ͸ MLM > RTD Ͱ͋Δ ● (ࢲ৴) SimLMͷ໨తؔ਺͸ஔ׵લͷ༧ଌͰ͋ΓɺMLMͱੑೳ͕ۙ ͍͜ͱ͔Βࣝผ͢ΔΑ͏ͳλεΫΑΓτʔΫϯΛ༧ଌ͢ΔΑ͏ͳ λεΫͷํ͕ྑ͍ʁ 32 .4."3$0ͰධՁ

ࣄલֶशίʔύεʹΑΔӨڹ ● MS-MARCOͰ͸BM25ʹΑΔෛྫɺNQͰ͸நग़͞ΕͨෛྫΛར ༻ͨ͠Fine-tuningΛߦ͔ͬͯΒධՁ ● ࣄલֶशͱFine-tuningɺධՁσʔλ͕ಉ࣌͡ߴ͍ੑೳ ● טΈ߹͍ͬͯͳ͍࣌Ͱ΋ੑೳͷ޲্͸େ͖͘ɺࣄલֶशʹΑͬͯ ৘ใΛ[CLS]දݱʹѹॖ͢Δೳྗ͕֫ಘ͞Ε͍ͯΔ 33

·ͱΊ ● ີϕΫτϧݕࡧͷͨΊͷࣄલֶशख๏SimLMͷఏҊ ● Τϯίʔμͷग़ྗͷ͏ͪɺදݱϘτϧωοΫ (ࠓճ͸[CLS]) Λઙ͍σίʔμ ʹ౉͢͜ͱͰɺશͯͷҙຯ৘ใΛ
 දݱϘτϧωοΫʹѹॖ͢Δ͜ͱ
 Λֶशͤ͞ɺྑ޷ͳੑೳΛࣔͨ͠ 34 You never know what you’re going to get in life. You never [MASK] what you’re going to [MASK] in life. ੜ੒Ϟσϧ ( fi x) You never expect what you’re going to be in life. [CLS] You never tell what you’re going to learn in life. Τϯίʔμ ઙ͍σίʔμ [CLS]දݱ

ײ૝ ● ͱͯ΋ίετ͕͔͔ͬͯେมͦ͏ ● ର৅σʔλ͝ͱʹࣄલֶश͕ඞཁͳͷ͸ਏ͍ ● දݱϘτϧωοΫΛྑ͘͢ΔͨΊͷࣄલֶशΞʔΩςΫνϟ͕໘ന͍ ● ݕࡧք۾ʹ͸৭ʑ͋Γͦ͏ʁ ● CondenserͳͲ… 35