Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
SNLP2019
Search
Ayana Niwa
September 25, 2019
Research
1
500
SNLP2019
第11回最先端NLP勉強会 発表資料
Ayana Niwa
September 25, 2019
Tweet
Share
More Decks by Ayana Niwa
See All by Ayana Niwa
A Quick Overview to Unlock the Potential of LLMs through Prompt Engineering
ayaniwa
0
130
Learning To Retrieve Prompts for In-Context Learning
ayaniwa
0
1k
UnNatural Language Inference
ayaniwa
0
390
Trends in Natural Language Processing at NeurIPS 2019.
ayaniwa
8
4.3k
Other Decks in Research
See All in Research
[輪講] Transformer Layers as Painters
nk35jk
4
690
NeurIPS 2024 参加報告 & 論文紹介 (SACPO, Ctrl-G)
reisato12345
0
350
移動ビッグデータに基づく地理情報の埋め込みベクトル化
tam1110
0
240
ECCV2024読み会: Minimalist Vision with Freeform Pixels
hsmtta
1
430
打率7割を実現する、プロダクトディスカバリーの7つの極意(pmconf2024)
geshi0820
0
350
Weekly AI Agents News! 12月号 プロダクト/ニュースのアーカイブ
masatoto
0
330
Leveraging LLMs for Unsupervised Dense Retriever Ranking (SIGIR 2024)
kampersanda
2
310
大規模言語モデルを用いたニュースデータのセンチメント判定モデルの開発および実体経済センチメントインデックスの構成
nomamist
1
130
Tiaccoon: コンテナネットワークにおいて複数トランスポート方式で統一的なアクセス制御
hiroyaonoe
0
430
Neural Fieldの紹介
nnchiba
2
720
Vision Language Modelと完全自動運転AIの最新動向
tsubasashi
0
250
SpectralMamba: Efficient Mamba for Hyperspectral Image Classification
satai
3
150
Featured
See All Featured
The World Runs on Bad Software
bkeepers
PRO
67
11k
The Straight Up "How To Draw Better" Workshop
denniskardys
232
140k
The Cult of Friendly URLs
andyhume
78
6.2k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
6
580
We Have a Design System, Now What?
morganepeng
51
7.4k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
100
18k
How to train your dragon (web standard)
notwaldorf
91
5.9k
Java REST API Framework Comparison - PWX 2021
mraible
29
8.4k
Gamification - CAS2011
davidbonilla
80
5.2k
Music & Morning Musume
bryan
46
6.4k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
30
2.2k
Transcript
Probing for Semantic Classes: Diagnosing the Meaning Content of Word
Embeddings Yadollah Yaghoobzadeh, Katharina Kann, Timothy J. Hazen, Eneko Agirre, Hinrich Schutze ACL2019 11 NLP 2019/09/28 <
[email protected]
>
Outline 2 H9P=O8 O8 "!PC
$@K !.S%,'3 F5T&*'$:I P=O8$.S%,'PV ! R<72$PC H/+%)- >#"!.S O8! H9E69NR<72;BJ?Q1L >#".S(rare senses)P=O8APM0 " 4D('% UG
Background 3 # " !Word2VecGrove #"NLPIR* - ELMoBERT
$'+ ) %" full-space , % & L "$( " # &!
Background 4 # "@!Semantic class>*( S-class , "@=6:-&C?4=6
B $8 SEMCAT, HyperLex… =1;+=62%A → 0# .9/ - WIKI-PSE(WIKIpedia-based resource for Probing Semantics in word Embeddings) - "@=1;+sense embedding'7) “Lamb” Food3< Living-thing5< "@ !
Background 5 # Apple Apple Apple
+ Word embeddingsense embedding Arora et al. (2018) Arora et al. (2018) Word embedding sparse coding WIKI-PSEsense embedding Word embedding Sense embedding Sense embedding
WIKI-PSE Resource 6 Wikipedia .!2,$# .! ()
S-class 113 FIGER types%+134"10 person/authorperson/politician… à person -S-class *'.!/.! & - S-class0
WIKI-PSE Resource 7 "$% %2& +:! 343,000-6.-S-class#5'5000;-!% 75,.6,.83
+*1! /4 44,250- -S-class - S-class )0organization (9food @apple@ – food @apple@ – organization
8 Word embedding (word) @apple@ WIKI-PSE
word embedding Sense embedding @apple@ @apple@ - food @apple@ - organization @ word/S-class Uniform sum (unifΣ) !" Weighted sum (wghtΣ) !" # = % " !" #&" # Aggregated word embedding #&" Sense (word/S-class) embeddings word, unifΣ, wghtΣ
Experiments 9 Problem settings SkipGram Structured SkipGram
WIKI-PSE (LR) # (MLP k*)KNN) 1. word % word embedding 2. unifΣ sense embedding " 3. wghtΣ sense embedding ! $'& (" Word embedding
Experiments 10 Probing Task 1S-class Prediction S-class
@apple@ +food ∩ +organization ∩ -event S-class
Experiments 11 MLP > LRKNN 8&!S-class ."7;5
KNN )#, *:3, <(3-/ unifΣ > word > wghtΣ 'rare sense 0$ 97< à Rare sense42%+6 unifΣ Probing Task 1S-class Prediction F11 4
Experiments 12 Probing Task 1S-class Prediction
unifΣ $ #rare sense (13,000) F1! "
Experiments 13 Probing Task 1S-class Prediction * #4(! Recall
) S-class& sense embedding+%3 Rare S-class/"12 %"0, .$- Recall Dominance S-class' Recall
Experiments 14 Probing Task 1S-class Prediction /&)5. !"$' ,(20-
3+1(#41( Personlocation,(20* S-class !"6 20% Recall Recall #4 ,(20 Typicality Recall
Experiments 15 " Probing Task 2Ambiguity
Prediction ! L2 SSKIP$" LR / KNN / MLP ! ! unifΣ > word > wghtΣ KNN à % FREQUENCY(Baseline) # &LR
NLP Application Experiments 16 wghtΣ > word >(=) unifΣ <--
Probing task #!! -the U.S. Attorney’s Office announced Friday → location Common S-classtime Rare S-class location Friday mountain unifΣ ( !+ entity mention MC ), CRMR $)& SUBJ %*' MRPC " S-class
Summary & Conclusion 17 * $2"(3 WIKI-PSE'. 1)/& 0+$2/&
1, 1)/&% # 0+$2- a a e Rare sense e harder NLP Rare sense !# e