Share
Embed
[輪講資料] SimCSE: Simple Contrastive Learning of Sentence Embeddings
事前学習済み言語モデルと対照学習を用いて、非常にシンプルながら文埋め込み手法のState-of-the-Artを更新したSimCSEという手法について解説します。
Hayato Tsukagoshi
February 11, 2022
Transcript
55 slides
-
SimCSE: Simple Contrastive Learning of Sentence Embeddings Tianyu Gao, Xingcheng,
and Danqi Chen EMNLP 2021 URL: https://aclanthology.org/2021.emnlp-main.552.pdf ൃදऀ: Hayato Tsukagoshi Graduate school of Informatics, Nagoya University, Japan. -
จ֓ཁ •Contrastive LearningΛ༻͍ͨจຒΊࠐΈख๏ SimCSE ΛఏҊ •ਖ਼ྫͷ࡞ΓํͰ Unsupervised SimCSE / Supervised
SimCSE ͷ2छྨʹ͔ΕΔ Unsupervised SimCSE (unsup-SimCSE) •ಉ͡จʹରͯ͠ҟͳΔdropout maskΛద༻ͯ͠࡞ͨ͠ೋͭͷจຒΊࠐΈΛਖ਼ྫͱ͢Δ • ಉ͡จΛ2ճಉ͡Ϟσϧʹ௨͚ͩ͢ •STSλεΫʹ͓͍ͯڭࢣͳ͠ख๏ͰSOTA • طଘͷڭࢣ͋ΓϕʔεϥΠϯ(Sentence-BERT)ͱಉੑೳ Supervised SimCSE (sup-SimCSE) •NLIσʔληοτͷؚҙ (entailment) ϖΞΛਖ਼ྫͱ͢Δ • hard negativeͱͯ͠ໃ६ (contradiction) ϖΞΛ༻͍ͯ͞Βʹੑೳ্ •STSλεΫͰଞͷจຒΊࠐΈख๏Λେ্͖͘ճͬͯSOTA 2 -
࣍ •ಋೖ • จຒΊࠐΈ • Contrastive Learning / ରরֶश •
Semantic Textual Similarity (STS) λεΫ •จຒΊࠐΈͷؔ࿈ݚڀ • ୯ޠຒΊࠐΈ͔ΒจຒΊࠐΈΛߏ͢Δख๏ / จྨࣅܭࢉख๏ • BERTҎલͷจຒΊࠐΈϞσϧ/ख๏ • BERTҎޙͷจຒΊࠐΈϞσϧ/ख๏ •SimCSE • ࣄલௐࠪ • ख๏֓ཁ • ࣮ݧઃఆ / ࣮ݧ݁Ռ 3 -
໔ࣄ߲ / උߟ •அΓ͕ͳ͍ݶΓਤදจ͔ΒͷҾ༻Ͱ͢ •εϥΠυ༰ͷޡΓؾʹͳΔ͜ͱ͕͋Δ߹ɺ@hayato_tkgs ·Ͱ͓ئ͍͍ͨ͠·͢ • ͦͷଞͷࡶஊݚڀ/ਐ࿏૬ஊͳͲେৎͰ͢ɺͳΜͰ͝࿈བྷ͍ͩ͘͞ •ؔ࿈ݚڀΛհ͢Δઅ͕͋Γ·͕͢ɺͦͷઅจຒΊࠐΈશൠͷؔ࿈ݚڀΛհ͍ͯ͠·͢ •
SimCSEจͰݴٴ͞Ε͍ͯΔؔ࿈ݚڀͷΈΛ֬ೝ͍ͨ͠߹ݪஶจΛ͋ͨͬͯΈͯ ͍ͩ͘͞ • ޡͬͨཧղɺݴٴ͖ؔ͢࿈ݚڀͷݟಀؚ͕͠·ΕΔ͔͠Ε·ͤΜɻ܁Γฦ͠Ͱ͕͢ɺ ͦ͏ݴͬͨ߹͝ࢦఠ͍͚ͨͩΔͱخ͍͠Ͱ͢ •🧐←͜ͷϚʔΫ͕͍͍ͭͯΔهड़චऀͷײͰ͢ 4 -
ಋೖ: จຒΊࠐΈ •จຒΊࠐΈ: จΛݻఆͷϕΫτϧͱͯ͠දݱ͢Δ • ҙຯతྨࣅܭࢉɺจॻ/࣭Ԡ ݕࡧɺจ/จॻΫϥελϦϯάʹ༗༻ •ݪཧతʹԿΛຒΊࠐΜͰྑ͍(ҙຯҎ֎ͷߏจใͳͲ) • ଟ͘ͷݚڀจͷҙຯΛຒΊࠐΜͩϕΫτϧදݱʹfocus
•ྨࣅܭࢉΛखܰʹίετͰ࣮ߦͰ͖Δ • ϕΫτϧۙ୳ࡧϥΠϒϥϦ[1, 2] ͰߴʹྨࣅϕΫτϧΛܭࢉՄೳ(ۙࣅతʹ) •จຒΊࠐΈΛଞͷλεΫͷͨΊͷಛྔͱͯ͠͏͜ͱՄೳ ධՁࢦඪ •Semantic Textual Similarity (STS) λεΫ •ςΩετྨͳͲͷԼྲྀλεΫͰͷੑೳධՁ … දྫ: SentEval [3] •ΫϥελϦϯάͷੑೳ(clustering accuracy) 6 [1] https://github.com/facebookresearch/faiss [2] https://github.com/nmslib/nmslib [3] Conneau+: SentEval: An Evaluation Toolkit for Universal Sentence Representations, LREC ‘18 -
ಋೖ: Semantic Textual Similarity (STS) λεΫ •จϖΞͱͦͷҙຯతྨࣅ͕༩͞ΕͨSTSσʔληοτΛ༻͍Δ •จϖΞͷҙຯతͳྨࣅΛܭࢉɺਓखධՁʹΑΔྨࣅͱͷ ૬ؔΛଌΔ͜ͱͰϞσϧ͕Ͳͷఔ จͷҙຯΛଊ͍͑ͯΔ͔ΛධՁ
•จຒΊࠐΈͷධՁʹҰൠతʹ… • จຒΊࠐΈಉ࢜ͷίαΠϯྨࣅΛ จͷҙຯྨࣅͱͯ͠ར༻ • Unsupervisedઃఆ: STSσʔληοτͰͷ ֶश͠ͳ͍(ྨࣅͷճؼϞσϧͳͲ) • ϞσϧͷྨࣅͱਓखධՁͷྨࣅͱͷ SpearmanͷॱҐ૬ؔΛଌΔ 7 STSσʔληοτͷ ࣮ࡍͷσʔλˠ -
ಋೖ: Semantic Textual Similarity (STS) λεΫ •STSλεΫʹҰൠతʹ STS12-16 [4-8], STS
Benchmark [9], SICK-R [10] ͕༻͍ΒΕΔ • ͍ͣΕͷσʔληοτจϖΞͱ࣮ͷҙຯతྨࣅ͕ϥϕϧ͚͞Ε͍ͯΔ • ҙຯతྨࣅͷൣғ STS12-16, STS Benchmark ͕ 0-5, SICK-R ͕ 1-5 • STS12-16test setͷΈɺSTS Benchmark ʹ train / dev / test set ͕ଘࡏ •STS Benchmark dev setΛͬͯϋΠύϥௐ͢Δ͜ͱ͕͋Δ • SimCSEֶशͳͲͷνϡʔχϯάͷ΄͔ɺධՁʹ༻͢ΔcheckpointͷબͷͨΊʹ ܇࿅த250step͝ͱʹධՁͯ͠࠷ྑ͍checkpointΛར༻ •STSλεΫධՁख๏͕จ͝ͱʹҟͳΔ͕࣌͋Γɺҙ͕ඞཁ • Spearman / Pearson ͷ(ॱҐ)૬ؔͷ ୯७ / ॏΈ͖ ฏۉΛ͏…ͳͲ • SimCSEจͷAppendix.Bʹهड़͕͋ΔͷͰҰಡΛਪ 8 [4] Agirre+: SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity, *SEM ’12 [5] Agirre+: *SEM 2013 shared task: Semantic Textual Similarity, *SEM ‘13 [6] Agirre+: SemEval-2014 Task 10: Multilingual Semantic Textual Similarity, SemEval ‘14 [7] Agirre+: SemEval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability, SemEval ’15 [8] Agirre+: SemEval-2016 Task 1: Semantic Textual Similarity, Monolingual and Cross-Lingual Evaluation, SemEval ’16 [9] Cer+: SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation, SemEval ’17 [10] Marelli+: A SICK cure for the evaluation of compositional distributional semantic models, LREC ‘14 -
ಋೖ: Semantic Textual Similarity (STS) λεΫ ิ •BERT͔ΒφΠʔϒʹநग़ͨ͠จຒΊࠐΈSTSʹ͓͚Δੑೳ͕͍͜ͱ͕ΒΕ͍ͯΔ[11] • BERTͷจ຺Խ୯ޠຒΊࠐΈͷฏۉCLSͷϕΫτϧͳͲ
• GloVefastTextͳͲͷ੩తͳ୯ޠຒΊࠐΈͷฏۉΛͱͬͨํ͕BERTΑΓੑೳ͕͍͍ •ҰํͰɺԼྲྀλεΫ(sentiment classi fi cationͳͲ)ʹ͓͚ΔBERT༝དྷͷจຒΊࠐΈͷੑೳ ͋Δఔߴ͍ʹҙ •BERTͳͲࣄલֶशࡁΈݴޠϞσϧͷຒΊࠐΈۭؒҟํੑ(anisotropy)Λ࣋ͪ[12]ɺ͜Ε͕ STSλεΫͷੑೳʹѱӨڹΛ༩͍͑ͯΔՄೳੑ͕ࣔࠦ͞Ε͍ͯΔ[13] 9 [11] Reimers+: Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, EMNLP '19 [12] Ethayarajh, How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings, EMNLP ’19 [13] Li+: On the Sentence Embeddings from Pre-trained Language Models, EMNLP '20 -
ಋೖ: Contrastive Learning / ରরֶश •ਖ਼ྫͱෛྫͷಛදݱΛϞσϧ͔Βग़ྗ •ਖ਼ྫಉ࢜ͷྨࣅ͕ߴ͘ͳΔΑ͏ʹֶशΛߦ͏ •Computer VisionͰେਓؾɺNLPͰྲྀߦத SimCLR
[15] •ಉ͡ը૾ʹରͯ͠ҟͳΔ data augmentationΛͨ͠ ը૾ಉ࢜Λਖ਼ྫʹ͢Δ •ޙஈͷը૾ྨλεΫͳͲ Ͱߴ͍ੑೳ •CVʹ͓͚ΔදݱֶशͷͨΊ ͷpre-trainingͱͯ͠༗ޮ 10 ը૾ϒϩά[16]ΑΓҾ༻ [14] Oord+: Representation Learning with Contrastive Predictive Coding, arXiv ‘18 [15] Chen+: A Simple Framework for Contrastive Learning of Visual Representations, ICML ’20 [16] Advancing Self-Supervised and Semi-Supervised Learning with SimCLR, ’20 [17] Chen+: Big Self-Supervised Models are Strong Semi-Supervised Learners, NeurIPS ’20 -
ಋೖ: Contrastive Learning / ֶशखॱ •mini-batchͷ͋Δࣄྫʹ͍ͭͯɺଞͷࣄྫΛෛྫͱͯ͠ߟ͑Δ → in-batch negatives •(batch
size x batch size) ͷྨࣅߦྻΛܭࢉɺର֯(ࣗࣗ)Λਖ਼ղͱ͢Δ • ର֯ͷྨࣅ࠷େԽ == ࣗࣗͱͷྨࣅ࠷େԽ •࣮తʹCross Entropy LossΛܭࢉ ↓ଛࣦؔ •ଟ͘ͷ߹ྨࣅͱͯ͠ίαΠϯྨࣅΛ͏ •ਖ਼ྫͷ࡞Γํ͕ඇৗʹॏཁ • SimCLRͰaugmentationख๏ʹΑͬͯੑೳ͕มԽ 11 ↑ྨࣅߦྻɺ੨৭෦͕ਖ਼ղʹ͋ͨΔ ℒi = − log esim(hi ,h+ i )/τ ∑N j=1 esim(hi ,h+ j )/τ ը૾ 1 ը૾ 2 ը૾ 3 ը૾ 4 ը૾ 5 ը ૾ 1’ ը ૾ 2’ ը ૾ 3’ ը ૾ 4’ ը ૾ 5’ batch size -
[18] Wang+: Understanding Contrastive Representation Learning through Alignment and Uniformity
on the Hypersphere, ICML ’20 ಋೖ: Contrastive Learning / Alignment, Uniformity •Contrastive Learningʹ͓͚Δಛදݱ͕࣋ͭͱ·͍͠ੑ࣭͔Β ߟ͑ͯಛදݱͷྑ͞ΛଌΔ(ඍՄೳͳ)ࢦඪΛఏҊ [18] • lower is better Alignment •ࣅͨαϯϓϧ͕ಛ্ۭؒͰۙ͘ʹͯ͘͠ΕΔ͔ Uniformity •ಛදݱ͕Ͳͷఔಛ্ۭؒͷ ୯Ґٿ໘্ʹҰ༷͢Δ͔ 12 ਤจ͔ΒҾ༻ -
ؔ࿈ݚڀ: ୯ޠຒΊࠐΈˠจຒΊࠐΈ / จྨࣅܭࢉख๏ p-mean: จຒΊࠐΈΛ୯ޠຒΊࠐΈͷ Ͱܭࢉ [19] SWEM: ୯ޠຒΊࠐΈΛ
ฏۉ / max / ฏۉͱmaxͷconcat / ہॴ૭͝ͱʹฏۉ͔ͯ͠Βmax ΛͱΔ [20] GEM: จதͷ୯ޠຒΊࠐΈͷߦجఈΛͱʹnoveltyͳͲॏΈΛܭࢉͯ͠୯ޠຒΊࠐΈΛॏΈ͚ [21] DynaMax: ೋͭͷจͷ୯ޠຒΊࠐΈΛstackͨ͠ߦྻΛ࡞ΓFuzzy setͷߟ͑ΛݩʹFuzzy JaccardΛܭࢉ [22] SIF: ΛܭࢉˠຒΊࠐΈߦྻΛಛҟղˠୈҰಛҟϕΫτϧ Ͱ Λܭࢉ [23] uSIF: ෳͷಛҟϕΫτϧΛར༻ɺಛҟͷ૯Λ͏ϋΠύϥௐෆཁͳSIF [24] P-SIF: ୯ޠͷτϐοΫϕΫτϧΛͬͨSIF [25] All-but-the-Top: ୯ޠຒΊࠐΈͷू߹ΛPCA্ͯ͠ҐओΛআ͘ [26] ( xp 1 + xp 1 + . . . + xp n n ) 1 p 1 |s| ∑ w∈s a a + p(w) vw u vs − uuTvs 14 [19] Ru ̈ckle ́+: Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations, arXiv ’18 [20] Shen+: Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms, ACL ’18 [21] Yang: Parameter-free Sentence Embedding via Orthogonal Basis, EMNLP-IJCNLP '19 [22] Zhelezniak+: Don't Settle for Average, Go for the Max: Fuzzy Sets and Max-Pooled Word Vectors, ICLR '19 [23] Arora+: A Simple but Tough-to-Beat Baseline for Sentence Embeddings, ICLR '17 [24] Ethayarajh: Unsupervised Random Walk Sentence Embeddings: A Strong but Simple Baseline, Rep4NLP '18 [25] Gupta+: P-SIF: Document Embeddings Using Partition Averaging, AAAI '20 [26] Mu+: All-but-the-Top: Simple and E ff ective Postprocessing for Word Representations, ICLR '18 -
ؔ࿈ݚڀ: ୯ޠຒΊࠐΈˠจຒΊࠐΈ / จྨࣅܭࢉख๏ Word Mover's Distance: จͷٯΛҰ༷ͳ࣭֬ྔɼίετΛϢʔΫϦουڑͱͯ͠࠷ద༌ૹ [27] Word
Mover's Embedding: จ(ॻ)ͱαϯϓϦϯάͨ͠ෳͷจ(ॻ)ͱͷWMDͷྻΛจ(ॻ)ϕΫτϧͱ͢Δ [28] Word Rotator's Distance: ୯ޠຒΊࠐΈͷϊϧϜΛ࣭֬ྔɼίετΛίαΠϯྨࣅͱͯ͠࠷ద༌ૹ [29] 15 [27] Kusner+: From Word Embeddings To Document Distances, ICML '15 [28] Wu+: Word Mover's Embedding: From Word2Vec to Document Embedding, EMNLP '18 [29] Yokoi+: Word Rotator’s Distance, EMNLP '20 -
จຒΊࠐΈͷؔ࿈ݚڀ: BERTҎલ 16 SkipThought: લޙͷจΛ࠶ߏ͢ΔΑ͏ʹBooks CorpusͰLSTMͷencoder-decoderϞσϧΛֶश [30] FastSent: લޙͷจͷBag-of-WordsΛ࠶ߏ͢ΔΑ͏ʹSkip-gramతͳϞσϧΛֶश [31]
SDAE: ϊΠζͷՃΘͬͨೖྗจ͔ΒϊΠζΛআڈͨ͠จΛ࠶ߏͰ͖ΔΑ͏ʹLSTMΛֶश [31] SCDV: ୯ޠຒΊࠐΈΛΫϥελϦϯάˠτϐοΫΛߟྀͨ͠୯ޠຒΊࠐΈͱεύʔεͳจຒΊࠐΈΛ֫ಘ [32] QuickThought: ࣍ͷจΛਖ਼ྫɼͦͷଞͷจΛෛྫͱͯ͠GRUΛରরֶश [33] Sent2Vec: n-gramΛߟྀͨ͠CBOWΛֶश [34] InferSent*: NLIྨͰLSTMΛֶश [35] Universal Sentence Encoder: DAN/TransformerΛNLIσʔληοτͰSkipThoughtతʹڭࢣͳֶ͠श [36] [30] Kiros+: Skip-Thought Vectors, NIPS '15 [31] Hill+: Learning Distributed Representations of Sentences from Unlabelled Data, NAACL ’16 [32] Mekala+: SCDV : Sparse Composite Document Vectors using soft clustering over distributional representations, ACL ’17 [33] Logesmaran+: An e ff i cient framework for learning sentence representations, ICLR ’18 [34] Pagliardini+:, Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features, NAACL ’18 [35] Conneau+: Supervised Learning of Universal Sentence Representations from Natural Language Inference Data, EMNLP '17 [36] Cer+: Universal Sentence Encoder, arXiv, Mar 2018 *ڭࢣ͋Γֶश -
จຒΊࠐΈͷؔ࿈ݚڀ: BERTҎޙ / Post-processing BERT- fl ow: ҟํతͳBERTͷจຒΊࠐΈۭ͔ؒΒํతͳજࡏۭؒͷࣸ૾Λֶश [37] BERT-whitening:
จຒΊࠐΈͷฏۉ͕0ɼڞࢄߦྻ͕୯ҐߦྻʹͳΔΑ͏ʹઢܗม(+࣍ݩݮ) [38] WhiteningBERT: จຒΊࠐΈͷฏۉ͕0ɼڞࢄߦྻ͕୯ҐߦྻʹͳΔΑ͏ʹઢܗม(w/ ͍ΖΜͳϞσϧ) [39] SBERT-WK: BERT/SBERTͷผͷಛྔΛ༻͍ͯܭࢉͨ͠noveltyͳͲͷΛͱʹจຒΊࠐΈΛߏ [40] 17 [37] Li+: On the Sentence Embeddings from Pre-trained Language Models, EMNLP '20 [38] Su+: Whitening Sentence Representations for Better Semantics and Faster Retrieval, arXiv '21 [39] Huang+: WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach, arXiv ’21 [40] Wang+: SBERT-WK: A Sentence Embedding Method by Dissecting BERT-Based Word Models, IEEE/ACM Transactions on Audio, Speech, and Language Processing ’20 ͯ͢ͷख๏͕Ϟσϧͱͯ͠BERT, Sentence-BERTΛϕʔεʹ͓ͯ͠Γɺ fi ne-tuning͠ͳ͍ -
จຒΊࠐΈͷؔ࿈ݚڀ: BERTҎޙ / Unsupervised IS-BERT: จຒΊࠐΈͱจதͷn-gramͷຒΊࠐΈͷ૬ޓใྔΛ࠷େԽ͢ΔΑ͏ʹֶश [41] ˑDeCLUTR: ಉ͡จॻͷҟͳΔεύϯಉ࢜Λਖ਼ྫͱͯ͠ରরֶश [42]
ˑBERT-CT: ҟͳΔೋͭͷಉ͡Ϟσϧͷɼಉ͡จʹର͢ΔจຒΊࠐΈಉ࢜ͷੵ͕େ͖͘ͳΔΑ͏ʹֶश [43] ˑConSERT: ݩͷจΛਖ਼ྫͱͯ͠ɼఢରతઁಈ / ୯ޠɾಛྔআ / γϟοϑϧ / Dropoutͯ͠ରরֶश [44] ˑSG/SG-OPT: จຒΊࠐΈͱBERTͷதؒͷຒΊࠐΈΛ͚ۙͮΔΑ͏ʹରরֶश [45] ˑCLEAR: ݩͷจΛਖ਼ྫͱͯ͠ɼ୯ޠɾεύϯআ / ޠॱೖΕସ͑ / ಉٛޠஔͯ͠ࣄલֶश࣌ʹରরֶश [46] ˑCOCO-LM: ୯ޠஔͨ͠จͱ୯ޠআͨ͠จಉ࢜Λਖ਼ྫͱͯ͠ରরֶश [47] TSDAE**: SDAEͷTransformer൛, จΛ࠶ߏͯ͠จຒΊࠐΈϞσϧΛֶश [48] 18 [41] Zhang+: An Unsupervised Sentence Embedding Method by Mutual Information Maximization, EMNLP '20 [42] Giorgi+: DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations, ACL '21 [43] Carlsson+: Semantic Re-tuning with Contrastive Tension, ICLR '21 [44] Yan+: ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer, ACL ’21 [45] Kim+: Self-Guided Contrastive Learning for BERT Sentence Representations, ACL ’21 [46] Wu+: CLEAR: Contrastive Learning for Sentence Representation, arXiv '20 [47] Meng+: COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining, NeurIPS ‘21 [48] Wang+:, TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning, EMNLP fi ndings '21 ˑContrastive LearningΛ༻͍ͨख๏ **ࣄલֶशࡁΈݴޠϞσϧͷ fi ne-tuningͰͳ͘εΫϥονͰϞσϧΛֶश͢Δ -
[11] Reimers+: Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, EMNLP ’19
[49] Thakur+: Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks, NAACL ’21 [50] Tsukagoshi+: DefSent: Sentence Embeddings using De fi nition Sentences, ACL ’21 [51] Zhang+: Pairwise Supervised Contrastive Learning of Sentence Representations, EMNLP ’21 [58] Jiang+: PromptBERT: Improving BERT Sentence Embeddings with Prompts, arXiv ‘22 จຒΊࠐΈͷؔ࿈ݚڀ: BERTҎޙ / Supervised Sentence-BERT: จຒΊࠐΈΛϕʔεʹͨ͠NLIྨͰBERTΛ fi ne-tuning [11] Augmented SBERT: Cross-encoderͰྨࣅϥϕϧΛ༩ٖͯ͠ࣅσʔληοτͰBi-encoderΛֶश [49] DefSent: ࣄલֶशࡁΈݴޠϞσϧͷ୯ޠ༧ଌΛར༻ɺఆٛจͷຒΊࠐΈ͔ΒͦΕ͕ද͢୯ޠΛ༧ଌ [50] ˑPairSupCon※: ؚҙؔͷจϖΞͷจຒΊࠐΈ͕ۙ͘ͳΔΑ͏ͳରরֶश+NLIྨͷಉֶ࣌श [51] ˑPromptBERT***: ؚҙؔͷຒΊࠐΈ͕ۙ͘ͳΔΑ͏ͳରরֶश+ςϯϓϨʔτ༝དྷͷϊΠζআڈ [58] 19 ˑContrastive LearningΛ༻͍ͨख๏ ※ SimCSEͱඇৗʹྨࣅɺจதͰSimCSEͷ͜ͱΛconcurrent workͱͯ͠Ҿ༻ ***SimCSEΑΓޙൃͷݚڀɺSimCSEΑΓߴ͍ੑೳΛୡͨ͠ͷͰཁνΣοΫ -
SimCSE: Simple Contrastive Sentence Embedding •ࣄલֶशࡁΈݴޠϞσϧΛContrastive LearningͰ fi ne-tuning͢ΔจຒΊࠐΈख๏ •ਖ਼ྫͷ࡞ΓํͰ
Unsupervised SimCSE / Supervised SimCSE ͷ2छྨʹ͔ΕΔ Unsupervised SimCSE (unsup-SimCSE) •ಉ͡จʹରͯ͠ҟͳΔdropout maskΛద༻ͯ͠࡞ͨ͠ೋͭͷจຒΊࠐΈΛਖ਼ྫͱ͢Δ • ಉ͡จΛ2ճಉ͡Ϟσϧʹ௨͚ͩ͢ • “minimal” ͳ data augmentation ͱͯ͠ͷ dropout Supervised SimCSE (sup-SimCSE) •NLIσʔληοτͷؚҙ (entailment) ϖΞΛਖ਼ྫͱ͢Δ • ྨΛղ͘͜ͱͤͣɺਖ਼ྫΛ࡞ΔͨΊʹڭࢣϥϕϧΛؒతʹར༻ • hard negativeͱͯ͠ໃ६ (contradiction) ϖΞΛ༻͍ͯ͞Βʹੑೳ্ •ҙຯతʹؚҙؔʹ͋Δจ͕จຒΊࠐΈ্ۭؒͰۙ͘ʹ͢ΔΑ͏ʹֶश 21 -
SimCSE: طଘݚڀͱͷൺֱ Unsupervised SimCSE •ϥϕϧͳ͠ςΩετͷΈΛ༻͍ͨରরֶशͰطଘͷڭࢣ͋ΓϕʔεϥΠϯͱಉੑೳ •ෳࡶͳdata augmentationɺՃͷωοτϫʔΫ͕ෆཁ •࣮͕ඇৗʹ༰қɺ֦ுੑʹ༏ΕΔ Supervised SimCSE
•Contrastive LearningͱϥϕϧσʔληοτΛ׆༻ͯ͠จຒΊࠐΈΛֶश͢Δख๏ΛఏҊ • ҙຯΛදݱ͢ΔจຒΊࠐΈҎ֎ʹ༷ʑʹԠ༻Մೳ •طଘͷڭࢣ͋ΓϕʔεϥΠϯΛେ্͖͘ճΔੑೳ •࣮͕ඇৗʹ༰қɺ֦ுੑʹ༏ΕΔ 🧐ֶशޮ͕ඇৗʹߴ҆͘ఆ( fi ne-tuningͷૣ͍ஈ֊Ͱੑೳ͕େ্͖͘͢Δ) •Contrastive LearningʹڭࢣϥϕϧΛ׆༻͢Δํ๏طʹଘࡏ (Ҿ༻͞Ε͍ͯͳ͍͕) [52] 22 [52] Khosla+: Supervised Contrastive Learning, arXiv ‘20 -
[53] Bowman+: A large annotated corpus for learning natural language
inference, EMNLP ‘15 [54] Williams+: A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference, NAACL ‘18 SimCSE: Contradiction as hard negatives NLIσʔληοτ (SNLI [53], MNLI [54]) ࡞ͷࡍͷखॱˣ •1ͭͷpremise (લఏจ) ͕Ξϊςʔλʹఏࣔ͞ΕΔ •Ξϊςʔλ͕premiseʹରͯ͠entailment (ؚҙ), neutral (தཱ), contradiction (ໃ६) ؔʹ ͋Δจ (hypothesis; Ծઆจ) Λهड़ → 1ͭͷpremiseʹରͯ͠entailment ͱ contradictionͷจ͕ͦΕͧΕଘࡏ •contradictionΛhard negativeͱͯ͠Ճ͑Δ͜ͱͰSTSͷੑೳ্ 24 premise hypothesis label A man playing an electric guitar on stage. A man playing banjo on the fl oor. contradiction A man playing an electric guitar on stage. A man playing guitar on stage. entailment A man playing an electric guitar on stage. A man is performing for cash. neutral දͷྫSNLI[53]͔Β -
•ಉ͡Ϟσϧʹಉ͡ܥྻΛ2ճೖྗ •batch_size x batch_size ͷྨࣅߦྻΛ࡞ •ԹύϥϝʔλͰׂΔ • Թύϥϝʔλ0.05͘Β͍ •ਖ਼ղྨࣅߦྻͷର֯ •ଛࣦؔCrossEntropy
(softmaxͯ͠ਖ਼ղΛ࠷େԽ) Ͱܭࢉ(ָ) Unsupervised SimCSE: pseudo code 28 ਤhttps://carbon.now.sh/Ͱ࡞ -
Supervised SimCSE: pseudo code 30 ਤhttps://carbon.now.sh/Ͱ࡞ •લఏจͱؚҙɾໃ६ؔͷԾઆจΛϞσϧʹೖྗ •લఏจͱؚҙͷԾઆจɺલఏจͱໃ६ͷԾઆจͰͦΕͧΕྨࣅߦྻΛ࡞ •ԣʹ͚ͬͭ͘Δ •ਖ਼ղྨࣅߦྻͷର֯
•ଛࣦؔCrossEntropy (softmaxͯ͠ਖ਼ղΛ࠷େԽ) Ͱܭࢉ(ָ) -
ࣄલௐࠪ •SimCSEͷ༗ޮੑɺSimCSE͕༗ޮͳ໘ΛௐΔͨΊʹෳͷ࣮ݧΛ࣮ࢪ ௐ߲ࠪ: Unsupervised SimCSE •data augmentationख๏ͷҧ͍ʹΑΔੑೳͷӨڹ •Next Sentence PredictionͳͲଞͷڭࢣͳֶ͠श༻తؔͱͷൺֱ
•dropout rateͷ୳ࡧͱੑೳͷӨڹ •ख๏͝ͱͷalignment, uniformityͷධՁ ௐ߲ࠪ: Supervised SimCSE •NLIΛؚΉෳͷσʔληοτͰͷֶशͱੑೳͷൺֱධՁ •hard negativesΛՃ͢Δ͜ͱʹΑΔੑೳͷӨڹ 32 -
Unsupervised SimCSE: data augmentationʹΑΔҧ͍ •dropoutҎ֎ͷdata augmentationख๏ͱͷൺֱ •ֶशʹ͏จӳޠWikipedia͔Β ϥϯμϜʹநग़ͨ͠100ສจ •STS Benchmark
dev setͰධՁ࣮ݧ •Ͳͷࢄత(discrete)ͳdata augmentationख๏ ΑΓdropoutͷੑೳ͕ߴ͍ •ࢄతͳdata augmentationҙຯΛຒΊࠐΉ ϕΫτϧදݱͷֶशʹޮՌతͰͳ͍ʁ • ࢄతͳઁಈͰ༰қʹจҙ͕มԽ͠͏Δ ͱ͍͏ࣗવݴޠจͷੑ࣭͕ؔʁ 33 -
Unsupervised SimCSE: ଞͷڭࢣͳ͠తؔͱͷൺֱ •Next sentence prediction (NSP) ͳͲͱUnsupervised SimCSEͱͷൺֱ •
NSPBERTͷࣄલֶशʹར༻͞Ε͍ͯΔ •ଞͷख๏ͱൺͯUnsupervised SimCSE͕STS Benchmark dev setͰ࠷ྑ͍ੑೳ 🧐ࣄલֶशͷͨΊͷతؔͱ จຒΊࠐΈͷͨΊͷతؔ Λൺֱ͢Δҙਤ͋·Γ Θ͔Βͳ͍ 34 -
Unsupervised SimCSE: dropout rateʹΑΔੑೳมԽ •dropout rate ΛมԽͤͯ͞STS Benchmark dev setʹ͓͚Δੑೳݕূ
•ಛʹҎԼͷઃఆ͕ಛघ(extreme cases)ɺશ͘ಉ͡ຒΊࠐΈදݱ͕ܭࢉʹΘΕΔ • : dropoutΛ͠ͳ͍ • : dropoutΛ͢Δ͕2ճͷຒΊࠐΈͰಉ͡dropout maskΛ͏ •Transformer, BERTͷσϑΥϧτͰ͋Δ 0.1 ͕࠷Α͍ੑೳ • dropout rateͷɺdropoutͷํʹରͯͦ͠ΕͳΓʹහײ • , ͷ݁Ռ͔Β શ͘ಉ͡ຒΊࠐΈදݱͰͷ fi ne-tuning ͩͱੑೳ͕Լ͢Δ͜ͱ͕ࣔࠦ 🧐ٯʹݴ͑ Ͱׂͱ ֶशͰ͖ͯΔ(ੑೳ্ͯ͠Δ) p p = 0.0 p = Fixed 0.1 p = 0.0 Fixed 0.1 p = 0.0 35 -
Unsupervised SimCSE: alignment, uniformityͷධՁ •SimCSEΛ༻ֶ͍ͯशதͷalignment, uniformityͷͷมԽΛ؍ •alignment, uniformityͱʹ͍ํ͕ྑ͍ •“No dropout”͓Αͼ
“Fixed 0.1” uniformity͕ݮগ͢Δ͕alignment͕૿େ • ҙຯతʹۙͯ͘ɺͱʹ͔͘શͯͷ จຒΊࠐΈͷڑΛͯ͠͠·͍ͬͯΔ •“Unsupervised SimCSE”uniformity͕ ݮগͯ͠alignment͕૿େ(ѱԽ)͠ͳ͍ 🧐alignmentͱuniformity͚͔ͩΒஅ͢Δͱ “Delete one word”͕Ұ൪ྑͦ͞͏ʹ ݟ͑Δ͕…ʁ • STSͷੑೳͱ៉ྷʹ૬͍ؔͯ͠ΔΘ͚ Ͱͳͦ͞͏ɺ͋͘·Ͱ҆ʁ 36 -
Supervised SimCSE: σʔληοτ͝ͱͷൺֱ •Supervised SimCSENLIσʔληοτҎ֎Λ༻͢Δͷߟ͑ΒΕΔ • hard negativesΛߟ͑ͳ͚ΕɺจϖΞσʔληοτͳΒͳΜͰ͍͍ •NLIσʔληοτ͕STSͰ࠷ྑͷ݁Ռ •NLIσʔληοτਓखͰ࡞͞Εͯ
͓Γߴ࣭ • ͔ͭɺจϖΞͷޠኮͷॏෳ͕ খ͘͞ͳΔΑ͏ઃܭ͞Ε͍ͯΔ 38 Quora Question Pairs ը૾Ωϟϓγϣϯ ٯ༁ʹΑΔݴ͍͑ neutralϖΞΛਖ਼ྫʹ contradictionϖΞΛਖ਼ྫʹ ͯ͢ͷϖΞΛਖ਼ྫʹ entailmentϖΞΛਖ਼ྫʹ -
ධՁ࣮ݧ •Unsupervised STS taskͰͷධՁ • STS12-16, STS Benchmark, SICK-R Λσʔληοτͱͯ͠ར༻
(ඪ४తͳઃఆ) •SentEvalΛ༻͍ͨධՁ (Appendix.E) • sentiment classi fi cationͳͲͷςΩετྨΛɺจຒΊࠐΈΛೖྗͱ͢Δ(ϩδε ςΟοΫճؼ)ྨثΛֶशͯ͠ੑೳΛଌΔ͜ͱͰɺจຒΊࠐΈͷੑ࣭ΛݟΔ 40 -
ධՁ࣮ݧ: STS •Unsupervised models ͕ڭࢣ ϥϕϧΛར༻͠ͳ͍ख๏ • Supervised models ͕ڭࢣ
ϥϕϧΛར༻ͨ͠ख๏ • STSσʔληοτΛڭࢣ ͱ͍ͯͬͯ͠ΔΘ͚Ͱ ͳ͍ʹҙ •Unsupervised / Supervised ͱʹSimCSE͕ߴ͍ੑೳ •Unsup-SimCSE-BERT_base͕ SBERT_baseΑΓߴ͍ੑೳ • ڭࢣ͋ΓϕʔεϥΠϯΛ ڭࢣͳ͠ख๏Ͱӽ͑Δ •Sup-SimCSEͰSOTA 41 -
ධՁ࣮ݧ: STS •Unsupervised models ͕ڭࢣ ϥϕϧΛར༻͠ͳ͍ख๏ • Supervised models ͕ڭࢣ
ϥϕϧΛར༻ͨ͠ख๏ • STSσʔληοτΛڭࢣ ͱ͍ͯͬͯ͠ΔΘ͚Ͱ ͳ͍ʹҙ •Unsupervised / Supervised ͱʹSimCSE͕ߴ͍ੑೳ •Unsup-SimCSE-BERT_base͕ SBERT_baseΑΓߴ͍ੑೳ • ڭࢣ͋ΓϕʔεϥΠϯΛ ڭࢣͳ͠ख๏Ͱӽ͑Δ •Sup-SimCSEͰSOTA 42 -
ධՁ࣮ݧ: SentEval 🧐STSͱҟͳΓඇৗʹߴ͍ੑೳͱ ͍͏Θ͚Ͱͳ͍ • ಛʹUnsup-SimCSE จຒΊࠐΈۭؒͷre fi ne͕ϝΠϯ ͔ͩΒʁ
•SBERTͱSup-SimCSE-BERTͰ લऀͷํ͕ߴ͍͕ɺSRoBERTaͱ Sup-SimCSE-RoBERTaͰޙऀ ͷ΄͏͕ߴ͍ੑೳ •ԼྲྀλεΫʹ͓͚Δੑೳ Ϟσϧࣗମͷೳྗʹґଘ • fi ne-tuningதʹMLMΛ͢Δ͜ͱͰ ԼྲྀλεΫͷੑೳ্͕ • catastrophic forgetting Λ͙ • STSͰͷੑೳ্ͳ͍ 43 -
ධՁ࣮ݧ: SentEval 🧐STSͱҟͳΓඇৗʹߴ͍ੑೳͱ ͍͏Θ͚Ͱͳ͍ • ಛʹUnsup-SimCSE จຒΊࠐΈۭؒͷre fi ne͕ϝΠϯ ͔ͩΒʁ
•SBERTͱSup-SimCSE-BERTͰ લऀͷํ͕ߴ͍͕ɺSRoBERTaͱ Sup-SimCSE-RoBERTaͰޙऀ ͷ΄͏͕ߴ͍ੑೳ •ԼྲྀλεΫʹ͓͚Δੑೳ Ϟσϧࣗମͷೳྗʹґଘ • fi ne-tuningதʹMLMΛ͢Δ͜ͱͰ ԼྲྀλεΫͷੑೳ্͕ • catastrophic forgetting Λ͙ • STSͰͷੑೳ্ͳ͍ 44 -
ධՁ࣮ݧ: SentEval 🧐STSͱҟͳΓඇৗʹߴ͍ੑೳͱ ͍͏Θ͚Ͱͳ͍ • ಛʹUnsup-SimCSE จຒΊࠐΈۭؒͷre fi ne͕ϝΠϯ ͔ͩΒʁ
•SBERTͱSup-SimCSE-BERTͰ લऀͷํ͕ߴ͍͕ɺSRoBERTaͱ Sup-SimCSE-RoBERTaͰޙऀ ͷ΄͏͕ߴ͍ੑೳ •ԼྲྀλεΫʹ͓͚Δੑೳ Ϟσϧࣗମͷೳྗʹґଘ • fi ne-tuningதʹMLMΛ͢Δ͜ͱͰ ԼྲྀλεΫͷੑೳ্͕ • catastrophic forgetting Λ͙ • STSͰͷੑೳ্ͳ͍ 45 -
ධՁ࣮ݧ: ख๏͝ͱͷalignment, uniformityͷධՁ •BERT- fl owBERT-whiteningͳͲ ͷpost-processingతͳख๏ alignment͕͔ͳΓྑ͘ͳ͍ • ҰํͰSBERT-
fl owͳͲͷSTSੑೳ ͔ͳΓߴ͍ •BERTͷຒΊࠐΈͷฏۉ uniformity͕ྑ͘ͳ͍ • ҟํੑʹ͍ͭͯͷطଘݚڀͱҰக •Contrastive LearningΛ͍ͯ͠ͳ͍ SBERTuniformity͕BERTΑΓվળ •Supervised / Unsupervised SimCSE uniformity͕վળ 46 -
ධՁ࣮ݧ: ख๏͝ͱͷalignment, uniformityͷධՁ •BERT- fl owBERT-whiteningͳͲ ͷpost-processingతͳख๏ alignment͕͔ͳΓѱ͍(ߴ͍) • ʹ͔͔ΘΒͣɺSBERT-
fl owͳͲ ͷSTSੑೳ͔ͳΓߴ͍ •BERTͷຒΊࠐΈͷฏۉ uniformity͕ѱ͍ • ҟํੑʹ͍ͭͯͷطଘݚڀͱҰக •Contrastive LearningΛ͍ͯ͠ͳ͍ SBERTuniformity͕BERTΑΓվળ •Supervised / Unsupervised SimCSE uniformity͕վળ 47 -
ධՁ࣮ݧ: ख๏͝ͱͷalignment, uniformityͷධՁ •BERT- fl owBERT-whiteningͳͲ ͷpost-processingతͳख๏ alignment͕͔ͳΓѱ͍(ߴ͍) • ʹ͔͔ΘΒͣɺSBERT-
fl owͳͲ ͷSTSੑೳ͔ͳΓߴ͍ •BERTͷຒΊࠐΈͷฏۉ uniformity͕ѱ͍ • ҟํੑʹ͍ͭͯͷطଘݚڀͱҰக •Contrastive LearningΛ͍ͯ͠ͳ͍ SBERTuniformity͕BERTΑΓվળ •Supervised / Unsupervised SimCSE uniformity͕վળ 48 -
ධՁ࣮ݧ: ಛҟͷ •จຒΊࠐΈͷू߹Λಛҟղͯ͠ಛҟͷΛ؍ •ಛҟ͕େ͖͍࣠େ͖ͳใྔΛ࣋ͭͱ(ϥϑʹ)ղऍͰ͖Δ • ภ͍ͬͯΔͱಛ্ۭؒͷ෦ۭؒʹจຒΊࠐΈ͕͕ͪ͠ɺͱ͍͏͜ͱ͕Θ͔Δ •SimCSEBERTΑΓগ͠ ͕ͳͩΒ͔ʹ •BERT- fl
owBERT-whitening ΑΓతʹຒΊࠐΈදݱ ͕ํత(isotropic)ʹ͢Δ Α͏ʹิਖ਼͢Δख๏ͳͷͰɺ ಛҟͷͳͩΒ͔ 49 -
·ͱΊ •Contrastive LearningΛ༻͍ͨඇৗʹγϯϓϧͳจຒΊࠐΈֶशख๏ΛఏҊ Unsupervised SimCSE •ೖྗจʹҟͳΔdropout maskΛֻ͚ͨจຒΊࠐΈಉ࢜Λਖ਼ྫʹ •STSλεΫʹ͓͍ͯڭࢣ͋ΓϕʔεϥΠϯΛ্ճΔ Supervised SimCSE
•NLIσʔληοτΛར༻ؚͯ͠ҙؔͷจϖΞΛਖ਼ྫʹ •طଘख๏Λେ্͖͘ճͬͯSOTA •alignmentͱuniformtyΛվળͨ͜͠ͱͰࣄલֶशࡁΈݴޠϞσϧͷҟํੑΛղফͯ͠ߴ࣭ ͳจຒΊࠐΈΛಘΒΕͨͱɺจຒΊࠐΈͷʹ͍ͭͯͷ࣮ݧ͔Βߟ •γϯϓϧͳͷͰద༻ൣғ͕͍ɺԠ༻͕ޮ͘ 52 -
ײ🧐 •Unsupervised SimCSEಉ͡Ϟσϧʹ2ճ௨͚ͩ͢ͱ͍͏γϯϓϧ͞ͰૉΒ͍͠ • Ͱ͖Δ͕͍Ζ͍Ζ͋ΔͷͰੜݚڀ͕ͨ͘͞Μੜ·Εͦ͏ • ྫ: ڭࢣͳ͠ͰͲ͏ͬͯྑ͍hard negativesΛ࡞Δ͔ʁͳͲ •Unsupervised
SimCSEͱSupervised SimCSE๊͕͖߹Θͤͳͷ͕ؾʹͳΔ • Supervised SimCSESupConSimCLRΛจຒΊࠐΈʹద༻͢ΔͳͷͰɺͦΕ୯ମ ͩͱcontribution͕খ͍͞ͱࢥΘΕΔͷΛආ͚ͨͷ͔͠Εͳ͍ʁ •alignmentͱuniformityΛվળ͢Δ͜ͱ͕ຊʹSTSλεΫʹͱͬͯ༗༻͔Ͳ͏͔ͷݕূ͕ ͍ؾ͕͢Δ • STSͰ͋Μ·Γ૬͍ؔͯ͠ͳ͍ • SentEvalͰͦΜͳʹ૬͍ؔͯ͠ͳͦ͞͏…? CVͱݴޠͰҟͳΔʁ • ޙൃͷPromptBERT[58] Ͱ୯ޠͷසใ͕࠷ӨڹΛ༩͍͑ͯΔͱͷੳΞϦ 53 -
[55] Wu+: Smoothed Contrastive Learning for Unsupervised Sentence Embedding, arXiv
Sep. ’21 [56] Wu+: ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding, arXiv Sep. ’21 [57] Zhang+: S-SimCSE: Sampled Sub-networks for Contrastive Learning of Sentence Embedding, arXiv Nov. ’21 SimCSEͷੜݚڀ GS-InfoNCE: ِͷෛྫ(ҙຯతʹಉ͡ͳͷʹෛྫ)ͷӨڹΛݮ͢ΔͨΊʹεϜʔδϯά [55] ESimCSE: ୯ޠෳͰจຒΊࠐΈͷ͞ใͷόΠΞεݮ && Momentum EncoderͰෛྫ૿Ճ [56] S-SimCSE: dropout rateΛจͷຒΊࠐΈ͝ͱʹมԽͤ͞Δ [57] 55 චऀײ: Ӎޙͷ