Zhou et al. 2019. Density Matching for Bilingual Word Embedding. NAACL

Density Matching for Bilingual Word Embedding Chunting Zhou, Xuezhe Ma,
Di Wang, Graham Neubig Language Technologies Institute Carnegie Mellon University

@: • *)5:5> ' -9 ; •
02?=/:5> ' 34% • B<CAIdentical Words (72 ; ,+ • ; • 8&1# Refinement !# • Bilingual Lexicon Induction (BLI) "$.6%

Cross-lingual Word Embedding • B,-7?7A)+5")+ • /*>?7 -7/ 69 •
48(@1; • high-resource -7(@ .= low-resource -748 • ' • Online: !D 0<#?7A)+(@ • Offline: ?-7 (@ 3-7?7A)+215" :C%,(@&$

Offline Cross-lingual Word Embedding • +TO@>:O@KFIS.7&1 • KF)N5;D,3/RG721?J4M • Wasserstein
RG JS #$! %" 721 • CKF)N5;GN-Q • 8=< • KF)N5;<5; D,(/6A09 * 4MPE: D, 3 L • KF)N5; <HB'

DeMa-BWE • Density Matching for Bilingual Word Embedding • OH*V69ADW1/:*U69
• ;Q(,5POH*V69 • <HAD*U\L73J[5P • ADE- "%!% # • GI2KFRCM' B@E-J[ • 5P8. +0 • R`]back-translation$ • Y >N 5PIdentical words ?4 • _1&S=XT)*U^Z1

Contribution • MUSE $&! • )#% morphologically rich
#% * • • "' * +(

Normalizing ﬂows • "- • $!< &1 • 7; !<
='+( )9 • 02# ,/ >35% *. >35% $!< 6:48 !< >35% $!<

Density Estimation in Monolingual Space • %$ • $
! % " • % # %$ x_i " $ & ! %

Density Matching • "$+< ;7#?,.2 ! •
684:(&5% #>03 • KL -*) +< • (&'/ Normalizing flows x #> y #> 1= #> 9= #> 03

Density Matching • %'.A @;&D/15#!#$ • #":<8? +)9(&C37 •
KL #0-, .A • +)*2 Normalizing flows x &C y &C # 4B &C >B &C y &C x >B= 6E #!# 37

Conditional Density Matching • Conditional Density Matching • •
• • •

Weak Orthogonality Constraint • /4.# + Orthogonality ) • *15"*1/4
!8$'7"(/4 ,613, %2 • 7:9 ,-0&

Weak Supervision with Identical Words •

Objectives for DeMa-BWE • Conditional Density Matching Weak Orthogonality
Constraint Weak Supervision with Identical Words

Cross-Domain Similarity Local Scaling • CSLS • % •
CSLS-D • '* #") ! $& '* k-NN ( '*

Iterative Procrustes Refinement • X Y
•

Experiment • MUSE • English ó Spanish; Japanese; Finnish;
... • Pretrained Word Embedding: FastText w/ Wikipedia • Normalizing, Centering • : 0.01 (en), 0.015 (morph-rich), 0.02 (others) • Vocabulary: 10,000 (en-ja), 20,000 (other pairs) • Loss: • back-translation loss: λ = 0.5 • supervised loss: α = 5 (en-zh), 10 (other pairs)

Precision@1 for MUSE BLI task
SL-unsup-ID

Morphologically complex languages

Pearson rank correlation •

Ablation study • Identical Words • en-ja identical
words • Density matching loss • Back-translation loss •

Conclusion • -,0!$// (1)' • &,(1# #*% • .
• Identical Words ++"

Zhou et al. 2019. Density Matching for Bilingua...

Zhou et al. 2019. Density Matching for Bilingual Word Embedding. NAACL

tosho

More Decks by tosho

Other Decks in Science

Featured

Transcript

Density Matching for Bilingual Word Embedding Chunting Zhou, Xuezhe Ma,

@: • *)5:5> ' -9 ; •

Cross-lingual Word Embedding • B,-7?7A)+5")+ • /*>?7 -7/ 69 •

Offline Cross-lingual Word Embedding • +TO@>:O@KFIS.7&1 • KF)N5;D,3/RG721?J4M • Wasserstein

DeMa-BWE • Density Matching for Bilingual Word Embedding • OHV69ADW1/:U69

Contribution • MUSE $&! • )#% morphologically rich

Normalizing ﬂows • "- • $!< &1 • 7; !<

Density Estimation in Monolingual Space • %$ • $

Density Matching • "$+< ;7#?,.2 ! •

Density Matching • %'.A @;&D/15#!#$ • #":<8? +)9(&C37 •

Conditional Density Matching • Conditional Density Matching • •

Weak Orthogonality Constraint • /4.# + Orthogonality ) • 15"1/4

Weak Supervision with Identical Words •

Objectives for DeMa-BWE • Conditional Density Matching Weak Orthogonality

Cross-Domain Similarity Local Scaling • CSLS • % •

Iterative Procrustes Refinement • X Y

Experiment • MUSE • English ó Spanish; Japanese; Finnish;

Precision@1 for MUSE BLI task

Morphologically complex languages

Pearson rank correlation •

Ablation study • Identical Words • en-ja identical

Conclusion • -,0!$// (1)' • &,(1# #*% • .