Share
Embed
Transcript
54 slides
-
Text Embeddings by Weakly-Supervised Contrastive Pre-training Graduate School of Informatics,
Nagoya University, Japan. ൃදऀ: Hayato Tsukagoshi Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, Furu Wei https://arxiv.org/abs/2212.03533 -
•େنͳࣄલରরֶशʹΑΓߏங͞Εͨ ςΩετຒΊࠐΈϞσϧE5ΛఏҊ • ߏԽσʔλͱϑΟϧλϦϯάΛ ༻͍ͨऑڭࢣ͋ΓσʔληοτΛߏங • όοναΠζ32000Ͱͷpre-training • hard negativeͱCross-Encoder͔Βͷ
ࣝৠཹΛ׆༻ͨ͠ fi ne-tuning •ධՁͷ݁ՌछʑͷϕϯνϚʔΫͰ ฏۉͯ͠طଘϞσϧΛ্ճΔ ֓ཁ 2 #Layers hidden size #params E5-small 12 384 33M E5-base 12 768 110M E5-large 24 1024 330M -
•ۙͷϞσϧͷଟ͘ҙػߏ(Attention Mechanism)ʹجͮ͘ TransformerͰߏ •͍Ζ͍Ζͳछྨ͕ଘࡏ ࣗݾճؼܕݴޠϞσϧ (Causal LM) •ࠨ͔Βӈʹ୯ޠΛ༧ଌͯ͠܇࿅ •ྫ: GPT,
GPT-2, GPT-3, Llama 2, … ݴޠϞσϧ: Language Models 6 ଞʹݴޠϞσϧʹ͞·͟·ͳछྨ͕ଘࡏɻྫ: XLNet, ELECTRA, UL2, … BERTͷ֓ཁਤ -
•ۙͷϞσϧͷଟ͘ҙػߏ(Attention Mechanism)ʹجͮ͘ TransformerͰߏ •͍Ζ͍Ζͳछྨ͕ଘࡏ ࣗݾճؼܕݴޠϞσϧ (Causal LM) •ࠨ͔Βӈʹ୯ޠΛ༧ଌͯ͠܇࿅ •ྫ: GPT,
GPT-2, GPT-3, Llama 2, … ϚεΫݴޠϞσϧ (Masked LM) •จதͷҰ෦ΛӅ͢ɾ༧ଌͯ͠܇࿅ •ྫ: BERT, RoBERTa, DeBERTa, … ݴޠϞσϧ: Language Models 7 ଞʹݴޠϞσϧʹ͞·͟·ͳछྨ͕ଘࡏɻྫ: XLNet, ELECTRA, UL2, … BERTͷ֓ཁਤ -
•ϕΫτϧྻΛೖྗʹϕΫτϧྻΛग़ྗ͢Δػߏ •ೖྗΛQ (Query), K (Key), V (Value)ʹ͚ͯܭࢉ • K, V:
nݸͷd࣍ݩϕΫτϧ • Q: mݸͷd࣍ݩϕΫτϧ ҙػߏ (Attention Mechanism) 8 ਤ Jaegle et al., Perceiver IO: A General Architecture for Structured Inputs & Outputs, ICLR 2022. ΑΓҾ༻ Θ͔Γ͍͢ղઆ: ʲਂֶशʳAttention - શྖҬʹԠ༻͞Ε࠷ߴਫ਼Λୟ͖ग़͢ҙػߏͷΈʲσΟʔϓϥʔχϯάͷੈք vol. 24ʳ -
•ϕΫτϧྻΛೖྗʹϕΫτϧྻΛग़ྗ͢Δػߏ •ೖྗΛQ (Query), K (Key), V (Value)ʹ͚ͯܭࢉ • K, V:
nݸͷd࣍ݩϕΫτϧ • Q: mݸͷd࣍ݩϕΫτϧ •Qʹର͢ΔVͷॏཁΛQͱKͷੵˠSoftmaxͰܭࢉ • Attention Weights: ܭࢉͷ݁ՌಘΒΕΔ(m × n)ߦྻ ҙػߏ (Attention Mechanism) 9 ਤ Jaegle et al., Perceiver IO: A General Architecture for Structured Inputs & Outputs, ICLR 2022. ΑΓҾ༻ Θ͔Γ͍͢ղઆ: ʲਂֶशʳAttention - શྖҬʹԠ༻͞Ε࠷ߴਫ਼Λୟ͖ग़͢ҙػߏͷΈʲσΟʔϓϥʔχϯάͷੈք vol. 24ʳ -
•ϕΫτϧྻΛೖྗʹϕΫτϧྻΛग़ྗ͢Δػߏ •ೖྗΛQ (Query), K (Key), V (Value)ʹ͚ͯܭࢉ • K, V:
nݸͷd࣍ݩϕΫτϧ • Q: mݸͷd࣍ݩϕΫτϧ •Qʹର͢ΔVͷॏཁΛQͱKͷੵˠSoftmaxͰܭࢉ • Attention Weights: ܭࢉͷ݁ՌಘΒΕΔ(m × n)ߦྻ •Self-Attention (ࣗݾҙػߏ): Q, K, VΛಉ͡ϕΫτϧྻ͔Βߏ (i.e. n=m) •Cross-Attention: ʮQʯͱʮK, VʯΛҟͳΔϕΫτϧྻ͔Βߏ ҙػߏ (Attention Mechanism) 10 ਤ Jaegle et al., Perceiver IO: A General Architecture for Structured Inputs & Outputs, ICLR 2022. ΑΓҾ༻ Θ͔Γ͍͢ղઆ: ʲਂֶशʳAttention - શྖҬʹԠ༻͞Ε࠷ߴਫ਼Λୟ͖ग़͢ҙػߏͷΈʲσΟʔϓϥʔχϯάͷੈք vol. 24ʳ -
•ҙػߏͷΈͰߏ͞ΕͨϞσϧߏ • ͦΕ·ͰNLPͰΑ͘ར༻͞Ε͍ͯͨ RNN, LSTMCNNΛഉআ • Transformer 11 Vaswani etl
al., Attention Is All You Need, NeurIPS 2017. Θ͔Γ͍͢ղઆ: ʲਂֶशʳTransformer - Multi-Head AttentionΛཧղͯ͠Ζ͏͡Όͳ͍ͷʲσΟʔϓϥʔχϯάͷੈքvol.28ʳ ֓ཁਤ Encoder Decoder -
•ҙػߏͷΈͰߏ͞ΕͨϞσϧߏ • ͦΕ·ͰNLPͰΑ͘ར༻͞Ε͍ͯͨ RNN, LSTMCNNΛഉআ •ϕΫτϧྻΛೖྗʹϕΫτϧྻΛग़ྗ͢Δػߏ • ೖྗϕΫτϧಉ࢜ͷ૬ޓ࡞༻Λߟྀ Transformer 12
Vaswani etl al., Attention Is All You Need, NeurIPS 2017. Θ͔Γ͍͢ղઆ: ʲਂֶशʳTransformer - Multi-Head AttentionΛཧղͯ͠Ζ͏͡Όͳ͍ͷʲσΟʔϓϥʔχϯάͷੈքvol.28ʳ ֓ཁਤ Encoder Decoder -
•ҙػߏͷΈͰߏ͞ΕͨϞσϧߏ • ͦΕ·ͰNLPͰΑ͘ར༻͞Ε͍ͯͨ RNN, LSTMCNNΛഉআ •ϕΫτϧྻΛೖྗʹϕΫτϧྻΛग़ྗ͢Δػߏ • ೖྗϕΫτϧಉ࢜ͷ૬ޓ࡞༻Λߟྀ •EncoderͱDecoderͷೋछྨ͕ଘࡏ •
EncoderͷΈ: BERT, LUKE, … • DecoderͷΈ: GPT, GPT-2, GPT-3, … • Encoder-Decoder: BART, T5, UL2, … Transformer 13 Vaswani etl al., Attention Is All You Need, NeurIPS 2017. Θ͔Γ͍͢ղઆ: ʲਂֶशʳTransformer - Multi-Head AttentionΛཧղͯ͠Ζ͏͡Όͳ͍ͷʲσΟʔϓϥʔχϯάͷੈքvol.28ʳ ֓ཁਤ Encoder Decoder -
•Transformer EncoderΛෳॏͶͯେنʹࣄલֶशͨ͠Ϟσϧ • base12 (1ԯύϥϝʔλ)ɺlarge24 (3.3ԯύϥϝʔλ) •ࣄલֶश (pre-training) → ඍௐ
( fi ne-tuning) ͱ͍͏ύϥμΠϜ͕ීٴ BERT: Bidirectional Encoder Representations from Transformers 14 Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, NAACL 2019. -
•ࣗવݴޠจͷີϕΫτϧදݱ •ϕΫτϧͷڑ͕จͷҙຯͷۙ͞Λදݱ จຒΊࠐΈ: Sentence Embedding 15 ͜Ͳ͕Ոʹ͔͍ͬͯΔɻ ͜Ͳֶ͕ߍ͔ΒՈʹ͔͍ͬͯΔɻ ͜Ͳ͕ਤॻؗʹ͍Δɻ ͜Ͳ͕ޕޙʹา͍͍ͯΔɻ
จຒΊࠐΈۭؒ [0.1, 0.2, ...] [0.1, 0.3, ...] [0.9, 0.8, ...] [0.5, 0.7, ...] -
•ࣗવݴޠจͷີϕΫτϧදݱ •ϕΫτϧͷڑ͕จͷҙຯͷۙ͞Λදݱ จຒΊࠐΈ: Sentence Embedding 16 ͜Ͳ͕Ոʹ͔͍ͬͯΔɻ ͜Ͳֶ͕ߍ͔ΒՈʹ͔͍ͬͯΔɻ ͜Ͳ͕ਤॻؗʹ͍Δɻ ͜Ͳ͕ޕޙʹา͍͍ͯΔɻ
จຒΊࠐΈۭؒ [0.1, 0.2, ...] [0.1, 0.3, ...] [0.9, 0.8, ...] [0.5, 0.7, ...] ҙຯతʹྨࣅ ͍ۙҙຯΛ࣋ͭจ ۙ͘ʹ ϕΫτϧؒͷڑ͕ ҙຯతͳؔΛදݱ -
ॳظ (~2018) •੩త୯ޠຒΊࠐΈ(Word2Vec, GloVe)͔ΒจຒΊࠐΈΛߏ͢Δख๏͕ओྲྀ • SIF, uSIF, All-but-the-Top, … •LSTMΛར༻ͯ͠from
scratchʹֶश͢Δख๏͍͔ͭ͘ଘࡏ • SkipThought, InferSent, Universal Sentence Encoder (USE), … ࣄલֶशϞσϧඍௐख๏ͷ಄ (2019~2021) •BERTͷ fi ne-tuningʹΑΓจຒΊࠐΈϞσϧΛಘΔख๏͕૿Ճ • BERT- fl ow, Sentence-BERT (SBERT), … จຒΊࠐΈݚڀͷมભ 17 ҙ: ͬ͘͟ΓͰ͢ -
ࣗવݴޠਪ (Natural Language Inference; NLI) •จϖΞ (લఏจɾԾઆจ) ʹϥϕϧ (ؚҙɾໃ६ɾதཱ) ͕༩
•จϖΞͷҙຯؔΛ༧ଌ͢ΔλεΫ NLIσʔληοτ 18 લఏจ Ծઆจ ϥϕϧ A man playing an electric guitar on stage. A man playing guitar on stage. ؚҙ A man playing an electric guitar on stage. A man playing banjo on the fl oor. ໃ६ A man playing an electric guitar on stage. A man is performing for cash. தཱ -
ରরֶशོظ (2021~) •ը૾Ͱྲྀߦ͍ͯͨ͠ରরֶशख๏͕จຒΊࠐΈʹ •ಛʹ SimCSE ͕දతͳख๏ʹ • ڭࢣ͋Γɾڭࢣͳ͠ͷೋͭͷख๏ΛఏҊ Unsupervised SimCSE
1. ಉ͡ೖྗʹର͠ҟͳΔdropout maskͰforward 2. ಘΒΕͨʮಉ͡ೖྗʹର͢ΔҟͳΔग़ྗʯಉ࢜Λਖ਼ྫʹରরֶश Supervised SimCSE • NLIσʔληοτதͷʮؚҙʯؔʹ͋ΔจϖΞΛਖ਼ྫʹରরֶश จຒΊࠐΈݚڀͷมભ 19 ຊޠSimCSEͷςΫχΧϧϨϙʔτͪ͜Β -
•දݱֶश (representation learning) ͷख๏ͷҰͭ •ਖ਼ྫಉ͕࢜ۙͮ͘Α͏ʹɺ͔ͭɺෛྫಉ͕࢜ΕΔΑ͏ʹֶश͢Δ • ਖ਼ྫಉ࢜ͷྨࣅ࠷େԽ & ෛྫಉ࢜ͷྨࣅ࠷খԽ ଛࣦ
(InfoNCE) ͷܭࢉ •ਖ਼ྫಉ࢜ͷຒΊࠐΈදݱͷcosྨࣅΛٻΊΔ •ෛྫಉ࢜ͷຒΊࠐΈදݱͷcosྨࣅΛٻΊΔ •ྨࣅΛฒͯԹύϥϝʔλΛద༻͢Δ •SoftmaxؔΛద༻ͯ֬͠ͱΈͳ͢ •ਖ਼ྫʹ͚ͩ1ཱ͕ͭʹ͚ۙͮΔ ରরֶश 21 -
εέʔϦϯάظ (2022~) •܇࿅ͷେنԽ͕ੵۃతʹߦΘΕΔΑ͏ʹ • σʔλྔͱόοναΠζͷ૿େʹΑΔରরֶशͷεέʔϦϯά • ϞσϧύϥϝʔλͷεέʔϦϯά •multi-stage contrastive learningͷಋೖ
• ऑڭࢣσʔλΛ༻͍ͨࣄલֶशˠڭࢣ͋ΓֶशʹΑΔFine-tuning • Ϟσϧྫ: E5, GTE, BGE, … •େنݴޠϞσϧ(LLM)Λ༻͍ͨςΩετຒΊࠐΈͷݚڀൃలத • PromptEOL, E5-Mistral, LLM2Vec, … จຒΊࠐΈݚڀͷมભ 23 ຊޠSimCSEͷςΫχΧϧϨϙʔτͪ͜Β -
•େنͳࣄલରরֶशʹΑΓߏங͞Εͨ ςΩετຒΊࠐΈϞσϧE5ΛఏҊ • ߏԽσʔλͱϑΟϧλϦϯάΛ ༻͍ͨऑڭࢣ͋ΓσʔληοτΛߏங • όοναΠζ32000Ͱͷpre-training • hard negativeͱCross-Encoder͔Βͷ
ࣝৠཹΛ׆༻ͨ͠ fi ne-tuning •ධՁͷ݁ՌछʑͷϕϯνϚʔΫͰ ฏۉͯ͠طଘϞσϧΛ্ճΔ ֓ཁ: ࠶ܝ 25 #Layers hidden size #params E5-small 12 384 33M E5-base 12 768 110M E5-large 24 1024 330M -
•ਂֶशϞσϧͷ܇࿅Ͱσʔλͷ࣭ͱଟ༷ੑ͕ੑೳΛେ͖͘ࠨӈ •͔͠͠ςΩετຒΊࠐΈϞσϧֶशͷͨΊͷσʔληοτগ • طଘݚڀ Stanford NLI MS-MARCO ͳͲਓखখنσʔλΛར༻ •େنͳςΩετຒΊࠐΈϞσϧ܇࿅༻σʔληοτΛߏங͢Δ
• ߏԽ͞Εͨσʔλ͔ΒςΩετϖΞΛऩू (ϑΟϧλલ: 1.3B pairs) CCPairs: ࣄલରরֶशͷͨΊͷେنσʔληοτ 27 Source Query Passage Size Wikipedia entity + section title passage 24M Reddit post upvoted comment 60M Common Crawl title passage 69M ࠷ऴతʹऩू͞Εͨσʔλͱܗࣜͷྫ -
•σʔλͷ্࣭ɾ܇࿅ίετݮͷͨΊϑΟϧλϦϯάΛ࣮ࢪ • ࠷ऴతʹ270M·Ͱݮ •ਂֶशϞσϧͷʮnoisyͳσʔλதͷ៉ྷͳࣄྫ͔Β֮͑ΔʯڍಈΛར༻ CCPairs: Consistency fi lterʹΑΔϊΠζআڈ 28 ޙଓͷGTEͰ
fi ltering͞Ε͍ͯͳ͍ ϞσϧԚ͍σʔλதͷ៉ྷͳσʔλ͔Β͍֮͑ͯ͘ →֮͑ΒΕͳ͔ͬͨσʔλʮԚ͍ʯ -
•σʔλͷ্࣭ɾ܇࿅ίετݮͷͨΊϑΟϧλϦϯάΛ࣮ࢪ • ࠷ऴతʹ270M·Ͱݮ •ਂֶशϞσϧͷʮnoisyͳσʔλதͷ៉ྷͳࣄྫ͔Β֮͑ΔʯڍಈΛར༻ Consistency-based data fi ltering 1. 1.3BͷnoisyͳσʔληοτͰϞσϧΛ܇࿅
2. 1MͷจষΛϥϯμϜʹநग़ͯ͠༻ҙ 3. ͋ΔΫΤϦʹର͠ਖ਼ྫจষͱϥϯμϜநग़͞Ε֤ͨจষͱͷྨࣅΛ1ͷ ϞσϧΛͬͯܭࢉ 4. ਖ਼ྫจষͷྨࣅॱҐ͕2Ҏ্ͷࣄྫͷΈ͢ CCPairs: Consistency fi lterʹΑΔϊΠζআڈ 29 ޙଓͷGTEͰ fi ltering͞Ε͍ͯͳ͍ -
•2ஈ֊ͷֶशख๏Λ࠾༻ 1. Contrastive Pre-training •௨ৗͷରরֶशଛࣦͱCCPairsΛ༻͍ͯڊେόοναΠζͰ܇࿅ • σʔλ͕noisyͳ߹΄ͲόοναΠζେ͖ͨ͘͠΄͏͕ྑͦ͞͏ •ೖྗʹଐੑใΛද͢pre fi xΛՃ
2. Fine-tuning •ਓखͰ࡞͞Εͨϥϕϧ͖σʔληοτͰ fi ne-tuning • ରরֶशଛࣦͷଞʹࣝৠཹଛࣦ༻͍Δ ֶशख๏ 30 “query:” ͱ “passage:” ͷೋͭ -
•ΫΤϦɾจষͷຒΊࠐΈಉ࢜ͰྨࣅΛܭࢉ • ΫΤϦ—ෛྫͷྨࣅΑΓΫΤϦ—ਖ਼ྫͷྨࣅ͕ߴ͘ͳΔΑ͏ֶश • ҙ༁: ΫΤϦͱจষͱͷྨࣅߦྻʹ͓͚Δର֯ͷ࠷େԽ •ಉ͡όονͷଞͷࣄྫΛෛྫʹ͢Δ: in-batch negatives ֶशख๏:
ରরֶश / Contrastive Pre-training 31 -
•ΫΤϦɾจষͷຒΊࠐΈಉ࢜ͰྨࣅΛܭࢉ • ΫΤϦ—ෛྫͷྨࣅΑΓΫΤϦ—ਖ਼ྫͷྨࣅ͕ߴ͘ͳΔΑ͏ֶश • ҙ༁: ΫΤϦͱจষͱͷྨࣅߦྻʹ͓͚Δର֯ͷ࠷େԽ •ಉ͡όονͷଞͷࣄྫΛෛྫʹ͢Δ: in-batch negatives ֶशख๏:
ରরֶश / Contrastive Pre-training 32 ΫΤϦ ਖ਼ྫจষ Model Model ਖ਼ྫͷຒΊࠐΈΛ͚ۙͮΔ యܕతʹcosྨࣅ -
•ΫΤϦɾจষͷຒΊࠐΈಉ࢜ͰྨࣅΛܭࢉ • ΫΤϦ—ෛྫͷྨࣅΑΓΫΤϦ—ਖ਼ྫͷྨࣅ͕ߴ͘ͳΔΑ͏ֶश • ҙ༁: ΫΤϦͱจষͱͷྨࣅߦྻʹ͓͚Δର֯ͷ࠷େԽ •ಉ͡όονͷଞͷࣄྫΛෛྫʹ͢Δ: in-batch negatives ֶशख๏:
ରরֶश / Contrastive Pre-training 33 ΫΤϦ ਖ਼ྫจষ Model Model ਖ਼ྫͷຒΊࠐΈΛ͚ۙͮΔ batch size batch size ΫΤϦͱจষΛผʑʹ encode͢ΔͷͰDual- EncoderͱݺΕΔ ॏΈڞ༗ (ಉ͡Ϟσϧ) యܕతʹcosྨࣅ -
•ਖ਼ྫ͚ͩ֬1ͷ֬Λࢦ͢ •ଛࣦΛԼ͛ΔͨΊʹ… • ਖ਼ྫͷྨࣅΛେ͖͘͢Δ • ෛྫͷྨࣅΛখ͘͢͞Δ ଛࣦؔͷײతཧղ: ରরֶशଛࣦ 34 ੵʹsoftmaxΛ͔͚ΔͷͰ֬ͱΈͳͤΔ
Pcl 1 Pstu … in-batch negatives one-hotͳʹ͚ۙͮΔ hard negatives Query Passage -
•ϥϕϧ͖σʔλͰͷ fi ne-tuningͰσʔλͷ࣭͕ॏཁ 1. hard negativesͷར༻ •ͺͬͱݟΘ͔Βͳ͍͍͠ࣄྫ • ϞσϧͷදݱྗΛߴΊΔɺඍࡉͳใΛଊ͑ΒΕΔΑ͏ʹ͢ΔޮՌ •MS-MARCONatural
Questions (NQ)ͰෛྫΛminingͯ͠ར༻ 2. ରরֶशͱࣝৠཹͷΈ߹Θͤ •ڭࢣ৴߸ΛΑΓϦονʹ͢ΔͨΊCross-Encoderͷग़ྗΛڭࢣͱͯ͠ར༻ •ରরֶशଛࣦͱࣝৠཹଛࣦΛΈ߹ΘͤͨϚϧνλεΫֶश ֶशख๏: Fine-tuning 35 ࣝৠཹଛࣦ ରরֶशଛࣦ -
•Dual-Encoder (DE) ΫΤϦɾจষؒͷ૬ޓ࡞༻ΛߟྀͰ͖ͳ͍ •Cross-Encoder (CE) ૬ޓ࡞༻ΛݟΕΔ͕ඇޮ •CEΛ฿Ͱ͖ΔΑ͏ʹDEΛֶश͢ΕղܾͰʁ🧐 •CEͷྨࣅʹDEͷग़ྗྨࣅΛ͚ۙͮΔ ֶशख๏: Cross-Encoder͔Βͷࣝৠཹ
36 ΫΤϦ+จষ Model ྨࣅείΞ ΫΤϦ จষ Model Model ྨࣅ Dual-Encoder Cross-Encoder -
•CEͷྨࣅͷʹDEͷΛ͚ۙͮΔ •ରরֶशͱҧ͍ྨࣅͷ্͛Լ͕͛ෆఆ • negativeͰ্͛ΔΑ͏ʹֶश͞Ε͏Δ • ણࡉͳྨࣅͷؔΛDEʹڭ͑Δ ଛࣦؔͷײతཧղ: ࣝৠཹଛࣦ 37 ੵʹsoftmaxΛ͔͚ΔͷͰ֬ͱΈͳͤΔ
༧ଌ CEͷ CEͷʹ͚ۙͮΔ Query Passage -
•ରরֶशͰin-batch negativesΛར༻͢Δ͕false negative͕ൃੜ͠͏Δ • ຊਖ਼ྫ(͚͍ۙͮͨ)ͳͷʹෛྫͱͯ͠ॲཧͯ͠͠·͏ݱ ࣝৠཹͷར: false negativeͷ੍ 38 1
… ຊਖ਼ղ͕ͩ… ରরֶशଛࣦͷ߹ ྨࣅΛաʹԼ͛Δํ ֶश͞Εͯ͠·͏ -
•ରরֶशͰin-batch negativesΛར༻͢Δ͕false negative͕ൃੜ͠͏Δ • ຊਖ਼ྫ(͚͍ۙͮͨ)ͳͷʹෛྫͱͯ͠ॲཧͯ͠͠·͏ݱ •ࣝৠཹͰCross-Encoderͷग़ྗΛٖࣅతͳڭࢣϥϕϧͱͯ͠ར༻ • false negativeͷΛ؇ ࣝৠཹͷར:
false negativeͷ੍ 39 ຊਖ਼ղ͕ͩ ྨࣅ͕খ͍͞ false negativeͷ؇ ྨࣅΛ্͛ΔΑ͏ʹֶशՄೳ ࣝৠཹʹΑΔଛࣦͷ߹ -
ଛࣦؔͷײతཧղ: ͚͔ۙͮͨͷൺֱ 40 ༧ଌ ਖ਼ղ 1 … CEͷʹ͚ۙͮΔ DKL one-hotͳʹ͚ۙͮΔ
ରরֶशଛࣦ ΄΅͚ۙͮΔ͕ҟͳΔ͚ͩ -
Summary: E5ͷ࡞Γํ 41 Un fi ltered Corpus Consistency-based Filtering CCPairs
Masked LM E5-PT Contrastive Pre-training E5 Contrastive Fine-tuning Labeled Data Knowledge Distillation Reranker encoder-only -
Summary: E5ͷ࡞Γํ 42 rerankerͷ࡞खॱݪஶจʹશવॻ͍͍ͯͳ͍ Un fi ltered Corpus Consistency-based Filtering
CCPairs Masked LM E5-PT Contrastive Pre-training E5 Contrastive Fine-tuning Labeled Data Knowledge Distillation Reranker Retriever 1 Retriever 2 ৄࡉઌߦݚڀͷSimLMจΛࢀরͷ͜ͱ encoder-only ৭ʑΊͪΌͪ͘Όؤு͍ͬͯΔ -
•Poolingख๏: Average Pooling (ग़ྗຒΊࠐΈͷฏۉΛऔΔ) • Transformerͷग़ྗϕΫτϧྻɺ୯ҰϕΫτϧʹ͢ΔͨΊͷૢ࡞͕Pooling Ϟσϧઃఆɾ܇࿅ৄࡉ 43 E5-large pre-training
fi ne-tuning #GPUs (V100) 64 8 batch size 32000 256 max length 128 192 #iteration 20000 steps 3 epochs Թύϥϝʔλ (τ) 0.01 0.01 ଛࣦͷॏΈ (α) N/A 0.2 #hard negatives N/A 7 Dataset CCPairs MS-MARCO, NQ, NLI -
•Poolingख๏: Average Pooling (ग़ྗຒΊࠐΈͷฏۉΛऔΔ) • Transformerͷग़ྗϕΫτϧྻɺ୯ҰϕΫτϧʹ͢ΔͨΊͷૢ࡞͕Pooling Ϟσϧઃఆɾ܇࿅ৄࡉ 44 E5-large pre-training
fi ne-tuning #GPUs (V100) 64 8 batch size 32000 256 max length 128 192 #iteration 20000 steps 3 epochs Թύϥϝʔλ (τ) 0.01 0.01 ଛࣦͷॏΈ (α) N/A 0.2 #hard negatives N/A 7 Dataset CCPairs MS-MARCO, NQ, NLI SimCSEͷ0.05ΑΓখ͍͞ Թύϥϝʔλ͕খ͍͞ →ͦ͜·ͰྨࣅΛ ઑΒͤͳ͍͍ͯ͘ ଟ༷ͳσʔλΛֶश ͢Δނͷྀʁ (ແཧʹྨࣅΛߴΊΑ͏ ͱ͠ͳ͍Α͏ʹ) -
•ରরֶशͰ࠷ॏཁͳϋΠύϥͷҰͭ •SoftmaxલͷΛՃͯ͠Softmaxޙͷ ͷܗঢ়ΛมԽͤ͞Δ Թύϥϝʔλͷิ: Πϝʔδ 45 ༧ଌ ߴԹύϥϝʔλ ྫ: 10
Թύϥϝʔλ ྫ: 0.01 ͕ฏୱʹ ͕ٸफ़ʹ -
•ରরֶशͰ࠷ॏཁͳϋΠύϥͷҰͭ •SoftmaxલͷΛՃͯ͠Softmaxޙͷ ͷܗঢ়ΛมԽͤ͞Δ Թύϥϝʔλͷิ: Πϝʔδ 46 ༧ଌ ߴԹύϥϝʔλ ྫ: 10
Թύϥϝʔλ ྫ: 0.01 ͕ฏୱʹ ͕ٸफ़ʹ Ϟσϧ͕ؤுͬͯΛ ઑΒͤΔඞཁ͋Γ Ϟσϧ͕ؤுΒͳͯ͘ ͕ઑΔ -
•SimCSEContrieverͱ͍ͬͨطଘख๏Λ্ճΔੑೳ • E5͜ΕΒͷख๏ΑΓσʔληοτ࡞ Λؤு͍ͬͯΔ •Contrastive Pre-trainingͷΈ͔ͳΓڧ͍ • ςΩετຒΊࠐΈͷͨΊͷࣄલֶश͕ޮՌత •E5ͷ fi
ne-tuningͷσʔληοτݶఆత • طଘख๏ʹෛ͚ͯΔλεΫׂͱ͋Δ • ଟ༷ԽɾେنԽ͢Δ͜ͱͰੑೳ্Λ ࢦͤͦ͏ ࣮ݧ݁Ռ: BEIR🍺 48 දॾʑলུͨ݁͠Ռɺৄࡉͳ࣮ݧ݁ՌݪஶจΛࢀরͷ͜ͱ Avg. BM25 41.7 SimCSEbase 20.3 Contrieverunsup 36.0 E5-PTlarge 44.2 Contrieversup 46.6 ColBERT 44.4 E5large 50.0 ڭࢣͳ͠ ڭࢣ͋Γ -
• fi ne-tuningʹ͏σʔληοτΛม͑ͯMTEBͰੑೳධՁ •Contrastive Pre-training͚ͩΑΓ fi ne-tuningͨ͠ํ͕ฏۉੑೳߴ͍ • ͕ɺNLIσʔληοτ͚ͩͰ fi
ne-tuning͢ΔͱݕࡧੑೳΉ͠ΖԼ • ݕࡧ+QAͰੑೳ͕͔ͳΓ্͢Δ͕ɺSTSͷੑೳ࠷େԽʹNLI͕ඞཁ •શͯΛࠞͥͯ͏͜ͱͰฏۉͯ͠࠷ߴੑೳɺଟ༷Խ͕େࣄ ੳ: fi ne-tuningσʔληοτͷଟ༷ੑ 51 NLIͰtuning͞ΕͨจຒΊࠐΈϞσϧݕࡧ༻్ʹ͔ͳ͍Ͱ͢(ࢲݟ) -
•େنͳࣄલରরֶशʹΑΓߏங͞ΕͨςΩετຒΊࠐΈϞσϧE5ΛఏҊ •ߏԽσʔλͱϑΟϧλϦϯάΛ༻͍ͨऑڭࢣ͋ΓσʔληοτΛߏங •2ஈ֊ͷֶशख๏Λ࠾༻ 1. Contrastive Pre-training 2. ରরֶशͱࣝৠཹΛΈ߹ΘͤͨFine-tuning ؾʹͳͬͨ͜ͱ •͔ͳΓςΫχΧϧϨϙʔτ෩ͳจͰResearch
Question͕͋·Γͳ͍ • ࣝৠཹͷ༗༻ੑͷݕূͳ͠ɾhard negativeΛͲ͏࡞Δ͔ᐆດ •pre fi x༗༻ͦ͏͕ͩޮՌ͕ݕূ͞Ε͍ͯͳ͍ ·ͱΊ 53 -
•E5ઌߦݚڀʹ͋ͨΔSimLMͷࣝΛલఏͱ͍ͯ͠Δ෦͕͋Δ • ಛʹRerankerͷ࡞Γํnegative miningʹ͍ͭͯSimLMจ͕ৄࡉ •ޙଓݚڀͷGTEษڧʹͳΔͷͰ߹ΘͤͯͲ͏ͧ •pre-trainingʹrerankerΛΘͳ͍ͷίετͷͬΆ͍ • ಉ༷ʹcontrastive pre-trainingͰhard negative
mining͍ͯ͠ͳ͍ •E5ͷଟݴޠ൛Ͱ͋ΔMultilingual E5ඇৗʹڧ͘ϕʔεϥΠϯͱͯ͠ྑ • ຊޠʹ͓͚Δݕࡧੑೳɾݕࡧ֦ுੜͰͷධՁจͪ͜Β ิ 54