Zhou et al. 2019. Density Matching for Bilingual Word Embedding. NAACL

Transcript

1. Density Matching for Bilingual Word Embedding Chunting Zhou, Xuezhe Ma, Di Wang, Graham Neubig Language Technologies Institute Carnegie Mellon University      
2. @:   • *)5:5> ' -9 ;  • 02?=/:5> ' 34% • B
   <CA
   Identical Words (72 ; ,+ • ; • 8&1#  Refinement !# • Bilingual Lexicon Induction (BLI) "$.6%
3. Cross-lingual Word Embedding • B,-7?7A)+5")+ • /*>?7 -7/ 69 • 48(@1; • high-resource -7(@ .= low-resource -748 •  ' • Online: !D 0<#?7A)+(@ • Offline: ?-7 (@ 3-7?7A)+215" :C%,(@&$ 
4. Offline Cross-lingual Word Embedding • +TO@>:O@KFIS.7&1 • KF)N5;D,3/RG721?J4M • Wasserstein RG  JS #$! %" 721 • CKF)N5;GN-Q • 8=< • KF)N5;<5; D,(/6A09 * 4MPE: D, 3 L   • KF)N5; <HB'  
5. DeMa-BWE • Density Matching for Bilingual Word Embedding • OH*V69ADW1/:*U69 • ;Q(,5POH*V69 • <HAD*U\L73J[5P • ADE- "%!% # • GI2KFRCM' B@E-J[ • 5P8.  +0 • R`]back-translation$ • Y >N 5PIdentical words ?4 • _1&S=XT)*U^Z1
6. Contribution • MUSE 
     $&! • )#% morphologically rich #%  *  •   • "' * +( 
7. Normalizing flows • "- • $!< &1 • 7; !< ='+( )9 •  02#
   ,/ >35%  *. >35% $!<   6:48    !< >35% $!<
8. Density Estimation in Monolingual Space • %$ •  $ ! % " • % #  
   
    %$ x_i "  $ & ! %
9. Density Matching • "$+< ;7#?,.2   ! •  684:
   (&5% #>03 • KL  -*)  +< • (&'/ Normalizing flows x #> y #>  1= #> 9= #>    03
10. Density Matching • %'.A @;&D/15#!#$ • #":<8?
    +)9(&C37  • KL #0-, .A  • +)*2 Normalizing flows x &C y &C # 4B &C >B &C y &C x >B= 6E   #!# 37
11. Conditional Density Matching • Conditional Density Matching •  •   •   •  • 
    
12. Weak Orthogonality Constraint • /4.# + Orthogonality ) • *15"*1/4 !8$'7"(/4  ,613,  %2  • 7:9 ,-0& 
13. Weak Supervision with Identical Words • 
       
14. Objectives for DeMa-BWE • 
     Conditional Density Matching Weak Orthogonality Constraint Weak Supervision with Identical Words
15. Cross-Domain Similarity Local Scaling • CSLS •  % • CSLS-D •  '* 
    #") ! $& '* k-NN ( '*
16. Iterative Procrustes Refinement •   
    X  Y  •  
17. Experiment • MUSE
     • English ó Spanish; Japanese; Finnish; ... • Pretrained Word Embedding: FastText w/ Wikipedia • Normalizing, Centering •   : 0.01 (en), 0.015 (morph-rich), 0.02 (others) • Vocabulary: 10,000 (en-ja), 20,000 (other pairs) • Loss: • back-translation loss: λ = 0.5 • supervised loss: α = 5 (en-zh), 10 (other pairs)
18. Precision@1 for MUSE BLI task          SL-unsup-ID 
19. Morphologically complex languages
20. Pearson rank correlation • 
      
21. Ablation study • Identical Words  • en-ja  identical words   • Density matching loss  • Back-translation loss    •  
22. Conclusion • -,0!$// (1)' • &,(1# 
    #*%  • .  • Identical Words ++"