All-but-the-Top: ୯ޠຒΊࠐΈͷू߹ΛPCA্ͯ͠ҐओΛআ͘ [26] ( xp 1 + xp 1 + . . . + xp n n ) 1 p 1 |s| ∑ w∈s a a + p(w) vw u vs − uuTvs 14 [19] Ru ̈ckle ́+: Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations, arXiv ’18
[20] Shen+: Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms, ACL ’18
[21] Yang: Parameter-free Sentence Embedding via Orthogonal Basis, EMNLP-IJCNLP '19
[22] Zhelezniak+: Don't Settle for Average, Go for the Max: Fuzzy Sets and Max-Pooled Word Vectors, ICLR '19
[23] Arora+: A Simple but Tough-to-Beat Baseline for Sentence Embeddings, ICLR '17
[24] Ethayarajh: Unsupervised Random Walk Sentence Embeddings: A Strong but Simple Baseline, Rep4NLP '18
[25] Gupta+: P-SIF: Document Embeddings Using Partition Averaging, AAAI '20
[26] Mu+: All-but-the-Top: Simple and E ff ective Postprocessing for Word Representations, ICLR '18
ؔ࿈ݚڀ: ୯ޠຒΊࠐΈˠจຒΊࠐΈ / จྨࣅܭࢉख๏ Word Mover's Distance: จͷٯΛҰ༷ͳ࣭֬ྔɼίετΛϢʔΫϦουڑͱͯ͠࠷ద༌ૹ [27]
Word Mover's Embedding: จ(ॻ)ͱαϯϓϦϯάͨ͠ෳͷจ(ॻ)ͱͷWMDͷྻΛจ(ॻ)ϕΫτϧͱ͢Δ [28] Word Rotator's Distance: ୯ޠຒΊࠐΈͷϊϧϜΛ࣭֬ྔɼίετΛίαΠϯྨࣅͱͯ͠࠷ద༌ૹ [29] 15 [27] Kusner+: From Word Embeddings To Document Distances, ICML '15
[28] Wu+: Word Mover's Embedding: From Word2Vec to Document Embedding, EMNLP '18
[40] Wang+: SBERT-WK: A Sentence Embedding Method by Dissecting BERT-Based Word Models, IEEE/ACM Transactions on Audio, Speech, and Language Processing ’20 ͯ͢ͷख๏͕Ϟσϧͱͯ͠BERT, Sentence-BERTΛϕʔεʹ͓ͯ͠Γɺ fi ne-tuning͠ͳ͍
[53] Bowman+: A large annotated corpus for learning natural language inference, EMNLP ‘15
[54] Williams+: A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference, NAACL ‘18 SimCSE: Contradiction as hard negatives NLIσʔληοτ (SNLI [53], MNLI [54]) ࡞ͷࡍͷखॱˣ •1ͭͷpremise (લఏจ) ͕Ξϊςʔλʹఏࣔ͞ΕΔ
→ 1ͭͷpremiseʹରͯ͠entailment ͱ contradictionͷจ͕ͦΕͧΕଘࡏ •contradictionΛhard negativeͱͯ͠Ճ͑Δ͜ͱͰSTSͷੑೳ্ 24 premise hypothesis label A man playing an electric guitar on stage. A man playing banjo on the fl oor. contradiction A man playing an electric guitar on stage. A man playing guitar on stage. entailment A man playing an electric guitar on stage. A man is performing for cash. neutral දͷྫSNLI[53]͔Β