Slide 1

Slide 1 text

ఆٛจΛ༻͍ͨจຒΊࠐΈߏ੒๏ ௩ӽॣɹ࡫໺ྒྷฏɹ෢ాߒҰ ݴޠॲཧֶձୈ30ճ೥࣍େձ (NLP 2024) ট଴࿦จ 2024/03/11 ໊ݹ԰େֶେֶӃ৘ใֶݚڀՊ

Slide 2

Slide 2 text

• ࣗવݴޠจͷີϕΫτϧදݱ • ϕΫτϧͷڑ཭͕จͷҙຯͷۙ͞Λදݱ จຒΊࠐΈ / Sentence Embedding 2 ͜Ͳ΋͕Ոʹ޲͔͍ͬͯΔɻ ͜Ͳ΋ֶ͕ߍ͔ΒՈʹ޲͔͍ͬͯΔɻ ͜Ͳ΋͕ਤॻؗʹ͍Δɻ ͜Ͳ΋͕ޕޙʹา͍͍ͯΔɻ จຒΊࠐΈۭؒ

Slide 3

Slide 3 text

ࣗવݴޠਪ࿦ (NLI) λεΫ • จϖΞͷҙຯؔ܎Λ༧ଌ • ؚҙɾໃ६ɾͦͷଞͷࡾ஋෼ྨ SBERT • NLI෼ྨ༻ͷ૚ΛBERTʹ௥Ճ • BERTΛ100ສจϖΞͰ fi ne-tuning ୅දతͳطଘख๏: Sentence-BERT (SBERT) 3 จB จA BERT BERT ໃ६ ؚҙ ͦͷଞ ϥϕϧ༧ଌ૚ Pooling Pooling

Slide 4

Slide 4 text

• NLIσʔληοτ͸ଟ͕͘ӳޠ • NLIϕʔεख๏͸ӳޠҎ֎ͷݴޠ΁ͷద༻͕೉͍͠ • طଘͷݴޠࢿݯͰจຒΊࠐΈϞσϧΛ֫ಘͰ͖ͳ͍͔ʁ → ଟ͘ͷݴޠͰ੔උ͞Ε͍ͯΔ୯ޠࣙॻʹண໨ • ࣙॻͰ͸୯ޠͱͦͷఆٛจ͕ಉ͡ҙຯΛද͢ → ୯ޠͱఆٛจͷҙຯ(ϕΫτϧ)͕ରԠ͢ΔΑ͏ʹ܇࿅ طଘख๏ͷ໰୊఺ɾղܾࡦ 4

Slide 5

Slide 5 text

• จຒΊࠐΈ • DefSent: ఆٛจΛ༻͍ͨจຒΊࠐΈߏ੒๏ • ڭࢣ৴߸ͷҧ͍ʹண໨ͨ͠ੑ࣭෼ੳ ໨࣍ 5

Slide 6

Slide 6 text

ఆٛจ w|V| w1 w2 w3 ... BERT ୯ޠ༧ଌ૚ Pooling • ఆٛจˠ୯ޠͷ༧ଌλεΫͰ܇࿅ • จͷߏ੒తͳҙຯΛཧղ • ఆٛจˠ୯ޠ༧ଌ૚ʹ͸
 ࣄલֶश(MLM)࣌ͷ૚Λར༻ • ࣄલֶशͰ֫ಘ͞ΕͨҙຯۭؒΛ׆༻ DefSent: ఆٛจΛ༻͍ͨจຒΊࠐΈߏ੒๏ 6 ৽نύϥϝʔλ͕ͳ͍ →ֶश΋ޮ཰త

Slide 7

Slide 7 text

• ఆٛจˠ୯ޠ༧ଌλεΫ • STS (Semantic Textual Similarity) λεΫ • SentEval ධՁ࣮ݧ 7

Slide 8

Slide 8 text

σʔληοτ • Oxford Dictionaryͷ୯ޠ—ఆٛจͷϖΞ5.4ສ݅ • Ϟσϧͷޠኮʹؚ·ΕΔ୯ޠ(ͱఆٛจ)ͷΈར༻ ධՁࢦඪ • ฏۉٯॱҐ (Mean Reciprocal Rank: MRR) • Top1, Top3, Top10 ਖ਼ղ཰ ఆٛจˠ୯ޠ༧ଌλεΫ: ධՁ࣮ݧ 8

Slide 9

Slide 9 text

• Top10ਖ਼ղ཰50%Ҏ্Ͱ༧ଌՄೳʹ ఆٛจˠ୯ޠ༧ଌλεΫ: ධՁ࣮ݧ 9 Ϟσϧ Pooling MRR Top1 Top3 Top10 BERT-
 base CLS 30.1 20.4 35.7 53.2 Mean 29.3 19.5 35.0 52.6 Max 27.4 17.6 32.5 50.4 RoBERTa-
 base CLS 32.3 21.8 38.4 56.8 Mean 31.8 21.4 37.8 56.4 Max 29.5 19.8 34.9 53.0

Slide 10

Slide 10 text

• ॳݟͷ(ఆٛ)จʹ͍ͭͯ΋ଥ౰ͳ୯ޠΛ༧ଌ ఆٛจˠ୯ޠ༧ଌλεΫ: ఆੑධՁ 10 ਖ਼ղ୯ޠ ఆٛจ ༧ଌ୯ޠ cost be expensive for (someone) cost charge pay preserve prevent (food) from rotting preserve keep spoil chief a person who is in charge leader boss master - not good bad poor wrong

Slide 11

Slide 11 text

• ʮϞσϧ͕ܭࢉͨ͠ྨࣅ౓ʯͱ
 ʮਓؒධՁʯͱͷ૬ؔ܎਺ΛධՁ • ҙຯΛଊ͑ΔೳྗΛଌΔ • จຒΊࠐΈධՁͰ͸ڭࢣͳ͠ઃఆ • STSσʔληοτͰͷ܇࿅ͳ͠ • ૬ؔ܎਺͕ߴ͍ˠྑ͍จຒΊࠐΈ STS (Semantic Textual Similarity) λεΫ 11 จA จB จຒΊࠐΈϞσϧ ਓखධՁͱͷ
 ૬ؔ܎਺ͰධՁ จྨࣅ౓

Slide 12

Slide 12 text

• SBERTͱಉ౳ఔ౓ͷੑೳ • ܇࿅σʔλྔ͸SBERTͷ1/20ఔ౓ (܇࿅࣌ؒ5෼΄Ͳ) STS: ࣮ݧ݁Ռ / BERT-base 12 Ϟσϧ STS12 STS13 STS14 STS15 STS16 STS-B SICK Avg. GloVe 55.1 70.7 59.7 68.3 63.7 58.0 53.8 61.3 FTͳ͠ 21.5 32.1 21.3 37.9 44.3 20.3 42.4 31.4 SBERT 69.8 72.5 70.4 78.0 73.5 76.0 72.3 73.2 DefSent 67.3 81.8 71.8 78.2 76.9 77.0 73.5 75.2

Slide 13

Slide 13 text

• ςΩετ෼ྨͳͲͷλεΫ͕
 ू·ͬͨϕϯνϚʔΫ • จຒΊࠐΈʹجͮ͘෼ྨثΛ܇࿅
 →จຒΊࠐΈͷ࣭Λ෼ྨੑೳͰධՁ • จຒΊࠐΈϞσϧͷύϥϝʔλ͸ݻఆ • ෼ྨੑೳ͕ߴ͍ˠྑ͍จຒΊࠐΈ SentEval 13 จ ෼ྨੑೳ͔Β
 จຒΊࠐΈͷ඼࣭ΛධՁ จຒΊࠐΈϞσϧ ෼ྨث

Slide 14

Slide 14 text

• ͪ͜Β΋SBERTͱಉ౳ఔ౓ͷੑೳ SentEval: ࣮ݧ݁Ռ / BERT-base 14 Ϟσϧ MR CR SUBJ MPQA SST-2 TREC MRPC Avg. GloVe 77.3 78.3 91.2 87.9 80.2 83.0 72.9 81.5 FTͳ͠ 81.8 87.9 95.5 88.2 86.5 91.0 72.3 86.2 SBERT 82.7 89.4 93.4 89.7 88.2 85.9 76.2 86.5 DefSent 81.8 88.0 94.9 89.9 86.3 90.1 75.4 86.6

Slide 15

Slide 15 text

• จຒΊࠐΈ • DefSent: ఆٛจΛ༻͍ͨจຒΊࠐΈߏ੒๏ • ڭࢣ৴߸ͷҧ͍ʹண໨ͨ͠ੑ࣭෼ੳɾ౷߹ ໨࣍ 15

Slide 16

Slide 16 text

• ػցֶशϞσϧ͸ֶशσʔλʹΑΓৼΔ෣͍Λม͑Δ • ಉ౳ੑೳͷϞσϧͰ΋ಘҙɾෆಘҙ͕͋Δ͸ͣ • ҟͳΔϞσϧͷ౷߹͸ੑೳ޲্΋ૂ͑Δ (Ξϯαϯϒϧ) • DefSent͕SBERT͸Ϟσϧ͕ಉ͡ɾڭࢣ৴߸͕ҟͳΔ • จຒΊࠐΈͷੑ࣭ʹ͸ҧ͍͕͋Δͷ͔ʁ • ૊Έ߹ΘͤΔ͜ͱͰ͞ΒʹੑೳվળͰ͖Δ͔ʁ ڭࢣ৴߸ͷҧ͍ʹண໨ͨ͠ੑ࣭෼ੳɾ౷߹ 16

Slide 17

Slide 17 text

ڭࢣ৴߸ͷҧ͍ʹண໨ͨ͠ੑ࣭෼ੳ 17 Semantic Textual Similarity (STS) ᶃ จͷιʔε ᶄ จϖΞͷද૚తྨࣅ౓ ൺֱ؍఺ SentEval ᶅ ԼྲྀλεΫ͝ͱͷੑೳ ᶆ ݴޠֶత৘ใͷ෼ྨੑೳ ൺֱ؍఺ ද૚తྨࣅ౓ SBERT DefSent •܇࿅σʔλʹ͍ۙจͷํ͕
 ͏·͘ྨࣅ౓ΛଌΕΔ •SBERT͸ද૚త৘ใ͕গͳ͍ •DefSent͸੍࣌ͳͲද૚త৘ใ͕ ଟ͘ɺϑϨʔζͷߏ੒͕ಘҙ ߴ ௿

Slide 18

Slide 18 text

• DefSentͱSBERTΛ
 ૊Έ߹ΘͤͯධՁ • ϚϧνλεΫֶश΍
 ຒΊࠐΈͷฏۉɾ݁߹ • SBERT→DefSent͸ڧ͘
 DefSent→SBERT͸ऑ͍ • ഁ໓త๨٫ͷӨڹΛࣔࠦ • Average͕୯७&ߴੑೳ DefSentͱSBERTͷ౷߹ 18 BERT-base STS SentEval SBERT 73.2 86.5 DefSent 75.2 86.6 SBERT→DefSent 78.5 86.8 DefSent→SBERT 72.9 86.1 ϚϧνλεΫ 72.9 86.2 Average 77.8 87.5 Concat 76.0 87.9

Slide 19

Slide 19 text

• ఆٛจˠ୯ޠ༧ଌʹΑΔ
 จຒΊࠐΈख๏ DefSent ΛఏҊ • SBERTͷ1/20ఔ౓ͷσʔλͰಉ౳ੑೳ • ෼ੳͷ݁Ռੑ࣭ͷҧ͍Λ໌Β͔ʹ • e.g. ද૚తྨࣅ౓ʹΑΔӨڹ • ౷߹: ୯७ͳฏۉ͕ߴੑೳ ͓ΘΓʹ 19 ఆٛจ w|V| w1 w2 w3 ... BERT ୯ޠ༧ଌ૚ Pooling