Zhou et al. 2019. Density Matching for Bilingual Word Embedding. NAACL

Transcript

1. Density Matching for Bilingual Word Embedding Chunting Zhou, Xuezhe Ma,

   Di Wang, Graham Neubig Language Technologies Institute Carnegie Mellon University      

2. @:   • *)5:5> ' -9 ;  •

   02?=/:5> ' 34% • B
   <CA
   Identical Words (72 ; ,+ • ; • 8&1#  Refinement !# • Bilingual Lexicon Induction (BLI) "$.6%

3. Cross-lingual Word Embedding • B,-7?7A)+5")+ • /*>?7 -7/ 69 •

   48(@1; • high-resource -7(@ .= low-resource -748 •  ' • Online: !D 0<#?7A)+(@ • Offline: ?-7 (@ 3-7?7A)+215" :C%,(@&$ 

4. Offline Cross-lingual Word Embedding • +TO@>:O@KFIS.7&1 • KF)N5;D,3/RG721?J4M • Wasserstein

   RG  JS #$! %" 721 • CKF)N5;GN-Q • 8=< • KF)N5;<5; D,(/6A09 * 4MPE: D, 3 L   • KF)N5; <HB'  

5. DeMa-BWE • Density Matching for Bilingual Word Embedding • OH*V69ADW1/:*U69

   • ;Q(,5POH*V69 • <HAD*U\L73J[5P • ADE- "%!% # • GI2KFRCM' B@E-J[ • 5P8.  +0 • R`]back-translation$ • Y >N 5PIdentical words ?4 • _1&S=XT)*U^Z1

6. Contribution • MUSE 
     $&! • )#% morphologically rich

   #%  *  •   • "' * +( 

7. Normalizing flows • "- • $!< &1 • 7; !<

   ='+( )9 •  02#
   ,/ >35%  *. >35% $!<   6:48    !< >35% $!<

8. Density Estimation in Monolingual Space • %$ •  $

   ! % " • % #  
   
    %$ x_i "  $ & ! %

9. Density Matching • "$+< ;7#?,.2   ! • 

   684:
   (&5% #>03 • KL  -*)  +< • (&'/ Normalizing flows x #> y #>  1= #> 9= #>    03

10. Density Matching • %'.A @;&D/15#!#$ • #":<8?
    +)9(&C37  •

    KL #0-, .A  • +)*2 Normalizing flows x &C y &C # 4B &C >B &C y &C x >B= 6E   #!# 37

11. Conditional Density Matching • Conditional Density Matching •  •

      •   •  • 
    

12. Weak Orthogonality Constraint • /4.# + Orthogonality ) • *15"*1/4

    !8$'7"(/4  ,613,  %2  • 7:9 ,-0& 

13. Weak Supervision with Identical Words • 
     

     

14. Objectives for DeMa-BWE • 
     Conditional Density Matching Weak Orthogonality

    Constraint Weak Supervision with Identical Words

15. Cross-Domain Similarity Local Scaling • CSLS •  % •

    CSLS-D •  '* 
    #") ! $& '* k-NN ( '*

16. Iterative Procrustes Refinement •   
    X  Y 

    •  

17. Experiment • MUSE
     • English ó Spanish; Japanese; Finnish;

    ... • Pretrained Word Embedding: FastText w/ Wikipedia • Normalizing, Centering •   : 0.01 (en), 0.015 (morph-rich), 0.02 (others) • Vocabulary: 10,000 (en-ja), 20,000 (other pairs) • Loss: • back-translation loss: λ = 0.5 • supervised loss: α = 5 (en-zh), 10 (other pairs)

18. Precision@1 for MUSE BLI task     

        SL-unsup-ID 

19. Morphologically complex languages

20. Pearson rank correlation • 
      

21. Ablation study • Identical Words  • en-ja  identical

    words   • Density matching loss  • Back-translation loss    •  

22. Conclusion • -,0!$// (1)' • &,(1# 
    #*%  • .

     • Identical Words ++"