Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
SNLP2019
Search
Ayana Niwa
September 25, 2019
Research
1
520
SNLP2019
第11回最先端NLP勉強会 発表資料
Ayana Niwa
September 25, 2019
Tweet
Share
More Decks by Ayana Niwa
See All by Ayana Niwa
A Quick Overview to Unlock the Potential of LLMs through Prompt Engineering
ayaniwa
0
140
Learning To Retrieve Prompts for In-Context Learning
ayaniwa
0
1.1k
UnNatural Language Inference
ayaniwa
0
410
Trends in Natural Language Processing at NeurIPS 2019.
ayaniwa
8
4.4k
Other Decks in Research
See All in Research
Self-supervised audiovisual representation learning for remote sensing data
satai
3
220
[輪講] SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
nk35jk
2
480
RapidPen: AIエージェントによるペネトレーションテスト 初期侵入全自動化の研究
laysakura
0
1.5k
クラウドのテレメトリーシステム研究動向2025年
yuukit
3
950
EarthMarker: A Visual Prompting Multimodal Large Language Model for Remote Sensing
satai
3
340
近似動的計画入門
mickey_kubo
4
970
When Submarine Cables Go Dark: Examining the Web Services Resilience Amid Global Internet Disruptions
irvin
0
210
Mechanistic Interpretability:解釈可能性研究の新たな潮流
koshiro_aoki
1
300
Looking for Escorts in Sydney?
lunsophia
1
120
NLP Colloquium
junokim
1
150
NLP2025SharedTask翻訳部門
moriokataku
0
300
学生向けアンケート<データサイエンティストについて>
datascientistsociety
PRO
0
3.1k
Featured
See All Featured
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
KATA
mclloyd
30
14k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
45
7.5k
Adopting Sorbet at Scale
ufuk
77
9.4k
Gamification - CAS2011
davidbonilla
81
5.3k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
29
9.5k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
50k
Art, The Web, and Tiny UX
lynnandtonic
299
21k
How to train your dragon (web standard)
notwaldorf
94
6.1k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
34
3k
Transcript
Probing for Semantic Classes: Diagnosing the Meaning Content of Word
Embeddings Yadollah Yaghoobzadeh, Katharina Kann, Timothy J. Hazen, Eneko Agirre, Hinrich Schutze ACL2019 11 NLP 2019/09/28 <
[email protected]
>
Outline 2 H9P=O8 O8 "!PC
$@K !.S%,'3 F5T&*'$:I P=O8$.S%,'PV ! R<72$PC H/+%)- >#"!.S O8! H9E69NR<72;BJ?Q1L >#".S(rare senses)P=O8APM0 " 4D('% UG
Background 3 # " !Word2VecGrove #"NLPIR* - ELMoBERT
$'+ ) %" full-space , % & L "$( " # &!
Background 4 # "@!Semantic class>*( S-class , "@=6:-&C?4=6
B $8 SEMCAT, HyperLex… =1;+=62%A → 0# .9/ - WIKI-PSE(WIKIpedia-based resource for Probing Semantics in word Embeddings) - "@=1;+sense embedding'7) “Lamb” Food3< Living-thing5< "@ !
Background 5 # Apple Apple Apple
+ Word embeddingsense embedding Arora et al. (2018) Arora et al. (2018) Word embedding sparse coding WIKI-PSEsense embedding Word embedding Sense embedding Sense embedding
WIKI-PSE Resource 6 Wikipedia .!2,$# .! ()
S-class 113 FIGER types%+134"10 person/authorperson/politician… à person -S-class *'.!/.! & - S-class0
WIKI-PSE Resource 7 "$% %2& +:! 343,000-6.-S-class#5'5000;-!% 75,.6,.83
+*1! /4 44,250- -S-class - S-class )0organization (9food @apple@ – food @apple@ – organization
8 Word embedding (word) @apple@ WIKI-PSE
word embedding Sense embedding @apple@ @apple@ - food @apple@ - organization @ word/S-class Uniform sum (unifΣ) !" Weighted sum (wghtΣ) !" # = % " !" #&" # Aggregated word embedding #&" Sense (word/S-class) embeddings word, unifΣ, wghtΣ
Experiments 9 Problem settings SkipGram Structured SkipGram
WIKI-PSE (LR) # (MLP k*)KNN) 1. word % word embedding 2. unifΣ sense embedding " 3. wghtΣ sense embedding ! $'& (" Word embedding
Experiments 10 Probing Task 1S-class Prediction S-class
@apple@ +food ∩ +organization ∩ -event S-class
Experiments 11 MLP > LRKNN 8&!S-class ."7;5
KNN )#, *:3, <(3-/ unifΣ > word > wghtΣ 'rare sense 0$ 97< à Rare sense42%+6 unifΣ Probing Task 1S-class Prediction F11 4
Experiments 12 Probing Task 1S-class Prediction
unifΣ $ #rare sense (13,000) F1! "
Experiments 13 Probing Task 1S-class Prediction * #4(! Recall
) S-class& sense embedding+%3 Rare S-class/"12 %"0, .$- Recall Dominance S-class' Recall
Experiments 14 Probing Task 1S-class Prediction /&)5. !"$' ,(20-
3+1(#41( Personlocation,(20* S-class !"6 20% Recall Recall #4 ,(20 Typicality Recall
Experiments 15 " Probing Task 2Ambiguity
Prediction ! L2 SSKIP$" LR / KNN / MLP ! ! unifΣ > word > wghtΣ KNN à % FREQUENCY(Baseline) # &LR
NLP Application Experiments 16 wghtΣ > word >(=) unifΣ <--
Probing task #!! -the U.S. Attorney’s Office announced Friday → location Common S-classtime Rare S-class location Friday mountain unifΣ ( !+ entity mention MC ), CRMR $)& SUBJ %*' MRPC " S-class
Summary & Conclusion 17 * $2"(3 WIKI-PSE'. 1)/& 0+$2/&
1, 1)/&% # 0+$2- a a e Rare sense e harder NLP Rare sense !# e