Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Zhou et al. 2019. Density Matching for Bilingual Word Embedding. NAACL
tosho
July 04, 2019
Science
3
100
Zhou et al. 2019. Density Matching for Bilingual Word Embedding. NAACL
tosho
July 04, 2019
Tweet
Share
More Decks by tosho
See All by tosho
Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation
tosho
0
200
Shaham and Levy, 2021. Neural Machine Translation without Embeddings. NAACL2021
tosho
0
34
Liu et al., 2021. Pay Attention to MLPs. arXiv
tosho
0
52
Huang et al. 2020 Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting
tosho
0
190
Ive, Madhyastha, Specia_2019_EMNLP_Deep Copycat Networks for Text-to-Text Generation
tosho
0
39
Tan, Bansal_2019_EMNLP_LXMERT Learning Cross-Modality Encoder Representations from Transformers
tosho
0
91
Tsai et al._2019_ACL_Multimodal Transformer for Unaligned Multimodal Language Sequences
tosho
0
99
Oral: Multimodal Machine Translation with Embedding Prediction
tosho
0
28
Caglayan et al. - NAACL 2019 - Probing the Need for Visual Context in Multimodal Machine Translation
tosho
0
150
Other Decks in Science
See All in Science
FreeCADで簡易版バスケットボールのモデル
kamakiri1225
0
320
20220216_球体周りの流れ抗力係数2_blockMeshでベースメッシュ作成
kamakiri1225
0
240
Quaternion Rotation
usamik26
0
380
機械学習における評価指標~AUC&C-index~
taka88
0
330
Accumulated Local Effects(ALE)で機械学習モデルを解釈する / TokyoR95
dropout009
2
3.2k
抵抗にあったっていいじゃないか、にんげんだもの / Principles of Human Resistance
aki_moon
0
570
Unlocking the Potential of Cloud Native Science with Pangeo
rabernat
0
110
アダプティブなカード
fukuyori
2
170
深層学習による自然言語処理 輪読会#5 資料
tok41
0
140
[勉強会資料メモ] Double/Debiased ML
masa_asa
0
280
20220216_球体周りの流れ抗力係数1_FreeCADで球体モデル作成
kamakiri1225
0
310
統計的因果探索: 領域知識とデータによる因果構造グラフの推測
sshimizu2006
4
1.5k
Featured
See All Featured
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
15
980
Principles of Awesome APIs and How to Build Them.
keavy
113
15k
How to train your dragon (web standard)
notwaldorf
60
3.9k
Streamline your AJAX requests with AmplifyJS and jQuery
dougneiner
127
8.5k
Building Adaptive Systems
keathley
25
1.2k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
655
120k
Fashionably flexible responsive web design (full day workshop)
malarkey
396
62k
Designing for Performance
lara
597
63k
A better future with KSS
kneath
226
16k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
6
570
What's in a price? How to price your products and services
michaelherold
229
9.4k
Build your cross-platform service in a week with App Engine
jlugia
219
17k
Transcript
Density Matching for Bilingual Word Embedding Chunting Zhou, Xuezhe Ma,
Di Wang, Graham Neubig Language Technologies Institute Carnegie Mellon University
@: • *)5:5> ' -9 ; •
02?=/:5> ' 34% • B<CAIdentical Words (72 ; ,+ • ; • 8&1# Refinement !# • Bilingual Lexicon Induction (BLI) "$.6%
Cross-lingual Word Embedding • B,-7?7A)+5")+ • /*>?7 -7/ 69 •
48(@1; • high-resource -7(@ .= low-resource -748 • ' • Online: !D 0<#?7A)+(@ • Offline: ?-7 (@ 3-7?7A)+215" :C%,(@&$
Offline Cross-lingual Word Embedding • +TO@>:O@KFIS.7&1 • KF)N5;D,3/RG721?J4M • Wasserstein
RG JS #$! %" 721 • CKF)N5;GN-Q • 8=< • KF)N5;<5; D,(/6A09 * 4MPE: D, 3 L • KF)N5; <HB'
DeMa-BWE • Density Matching for Bilingual Word Embedding • OH*V69ADW1/:*U69
• ;Q(,5POH*V69 • <HAD*U\L73J[5P • ADE- "%!% # • GI2KFRCM' B@E-J[ • 5P8. +0 • R`]back-translation$ • Y >N 5PIdentical words ?4 • _1&S=XT)*U^Z1
Contribution • MUSE $&! • )#% morphologically rich
#% * • • "' * +(
Normalizing flows • "- • $!< &1 • 7; !<
='+( )9 • 02# ,/ >35% *. >35% $!< 6:48 !< >35% $!<
Density Estimation in Monolingual Space • %$ • $
! % " • % # %$ x_i " $ & ! %
Density Matching • "$+< ;7#?,.2 ! •
684:(&5% #>03 • KL -*) +< • (&'/ Normalizing flows x #> y #> 1= #> 9= #> 03
Density Matching • %'.A @;&D/15#!#$ • #":<8? +)9(&C37 •
KL #0-, .A • +)*2 Normalizing flows x &C y &C # 4B &C >B &C y &C x >B= 6E #!# 37
Conditional Density Matching • Conditional Density Matching • •
• • •
Weak Orthogonality Constraint • /4.# + Orthogonality ) • *15"*1/4
!8$'7"(/4 ,613, %2 • 7:9 ,-0&
Weak Supervision with Identical Words •
Objectives for DeMa-BWE • Conditional Density Matching Weak Orthogonality
Constraint Weak Supervision with Identical Words
Cross-Domain Similarity Local Scaling • CSLS • % •
CSLS-D • '* #") ! $& '* k-NN ( '*
Iterative Procrustes Refinement • X Y
•
Experiment • MUSE • English ó Spanish; Japanese; Finnish;
... • Pretrained Word Embedding: FastText w/ Wikipedia • Normalizing, Centering • : 0.01 (en), 0.015 (morph-rich), 0.02 (others) • Vocabulary: 10,000 (en-ja), 20,000 (other pairs) • Loss: • back-translation loss: λ = 0.5 • supervised loss: α = 5 (en-zh), 10 (other pairs)
Precision@1 for MUSE BLI task
SL-unsup-ID
Morphologically complex languages
Pearson rank correlation •
Ablation study • Identical Words • en-ja identical
words • Density matching loss • Back-translation loss •
Conclusion • -,0!$// (1)' • &,(1# #*% • .
• Identical Words ++"