Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Zhou et al. 2019. Density Matching for Bilingual Word Embedding. NAACL

F16d24f8c3767910d0ef9dd3093ae016?s=47 tosho
July 04, 2019

Zhou et al. 2019. Density Matching for Bilingual Word Embedding. NAACL

F16d24f8c3767910d0ef9dd3093ae016?s=128

tosho

July 04, 2019
Tweet

More Decks by tosho

Other Decks in Science

Transcript

  1. Density Matching for Bilingual Word Embedding Chunting Zhou, Xuezhe Ma,

    Di Wang, Graham Neubig Language Technologies Institute Carnegie Mellon University      
  2. @:   • *)5:5> ' -9 ;  •

    02?=/:5> ' 34% • B<CAIdentical Words (72 ; ,+ • ; • 8&1#  Refinement !# • Bilingual Lexicon Induction (BLI) "$.6%
  3. Cross-lingual Word Embedding • B,-7?7A)+5")+ • /*>?7 -7/ 69 •

    48(@1; • high-resource -7(@ .= low-resource -748 •  ' • Online: !D 0<#?7A)+(@ • Offline: ?-7 (@ 3-7?7A)+215" :C%,(@&$ 
  4. Offline Cross-lingual Word Embedding • +TO@>:O@KFIS.7&1 • KF)N5;D,3/RG721?J4M • Wasserstein

    RG  JS #$! %" 721 • CKF)N5;GN-Q • 8=< • KF)N5;<5; D,(/6A09 * 4MPE: D, 3 L   • KF)N5; <HB'  
  5. DeMa-BWE • Density Matching for Bilingual Word Embedding • OH*V69ADW1/:*U69

    • ;Q(,5POH*V69 • <HAD*U\L73J[5P • ADE- "%!% # • GI2KFRCM' B@E-J[ • 5P8.  +0 • R`]back-translation$ • Y >N 5PIdentical words ?4 • _1&S=XT)*U^Z1
  6. Contribution • MUSE   $&! • )#% morphologically rich

    #%  *  •   • "' * +( 
  7. Normalizing flows • "- • $!< &1 • 7; !<

    ='+( )9 •  02# ,/ >35%  *. >35% $!<   6:48    !< >35% $!<
  8. Density Estimation in Monolingual Space • %$ •  $

    ! % " • % #   %$ x_i "  $ & ! %
  9. Density Matching • "$+< ;7#?,.2   ! • 

    684:(&5% #>03 • KL  -*)  +< • (&'/ Normalizing flows x #> y #>  1= #> 9= #>    03
  10. Density Matching • %'.A @;&D/15#!#$ • #":<8? +)9(&C37  •

    KL #0-, .A  • +)*2 Normalizing flows x &C y &C # 4B &C >B &C y &C x >B= 6E   #!# 37
  11. Conditional Density Matching • Conditional Density Matching •  •

      •   •  •  
  12. Weak Orthogonality Constraint • /4.# + Orthogonality ) • *15"*1/4

    !8$'7"(/4  ,613,  %2  • 7:9 ,-0& 
  13. Weak Supervision with Identical Words •    

     
  14. Objectives for DeMa-BWE •  Conditional Density Matching Weak Orthogonality

    Constraint Weak Supervision with Identical Words
  15. Cross-Domain Similarity Local Scaling • CSLS •  % •

    CSLS-D •  '* #") ! $& '* k-NN ( '*
  16. Iterative Procrustes Refinement •   X  Y 

    •  
  17. Experiment • MUSE  • English ó Spanish; Japanese; Finnish;

    ... • Pretrained Word Embedding: FastText w/ Wikipedia • Normalizing, Centering •   : 0.01 (en), 0.015 (morph-rich), 0.02 (others) • Vocabulary: 10,000 (en-ja), 20,000 (other pairs) • Loss: • back-translation loss: λ = 0.5 • supervised loss: α = 5 (en-zh), 10 (other pairs)
  18. Precision@1 for MUSE BLI task     

        SL-unsup-ID 
  19. Morphologically complex languages

  20. Pearson rank correlation •    

  21. Ablation study • Identical Words  • en-ja  identical

    words   • Density matching loss  • Back-translation loss    •  
  22. Conclusion • -,0!$// (1)' • &,(1# #*%  • .

     • Identical Words ++"