Zhou et al. 2019. Density Matching for Bilingual Word Embedding. NAACL

F16d24f8c3767910d0ef9dd3093ae016?s=47 tosho
July 04, 2019

Zhou et al. 2019. Density Matching for Bilingual Word Embedding. NAACL

F16d24f8c3767910d0ef9dd3093ae016?s=128

tosho

July 04, 2019
Tweet

Transcript

  1. Density Matching for Bilingual Word Embedding Chunting Zhou, Xuezhe Ma,

    Di Wang, Graham Neubig Language Technologies Institute Carnegie Mellon University      
  2. @:   • *)5:5> ' -9 ;  •

    02?=/:5> ' 34% • B<CAIdentical Words (72 ; ,+ • ; • 8&1#  Refinement !# • Bilingual Lexicon Induction (BLI) "$.6%
  3. Cross-lingual Word Embedding • B,-7?7A)+5")+ • /*>?7 -7/ 69 •

    48(@1; • high-resource -7(@ .= low-resource -748 •  ' • Online: !D 0<#?7A)+(@ • Offline: ?-7 (@ 3-7?7A)+215" :C%,(@&$ 
  4. Offline Cross-lingual Word Embedding • +TO@>:O@KFIS.7&1 • KF)N5;D,3/RG721?J4M • Wasserstein

    RG  JS #$! %" 721 • CKF)N5;GN-Q • 8=< • KF)N5;<5; D,(/6A09 * 4MPE: D, 3 L   • KF)N5; <HB'  
  5. DeMa-BWE • Density Matching for Bilingual Word Embedding • OH*V69ADW1/:*U69

    • ;Q(,5POH*V69 • <HAD*U\L73J[5P • ADE- "%!% # • GI2KFRCM' B@E-J[ • 5P8.  +0 • R`]back-translation$ • Y >N 5PIdentical words ?4 • _1&S=XT)*U^Z1
  6. Contribution • MUSE   $&! • )#% morphologically rich

    #%  *  •   • "' * +( 
  7. Normalizing flows • "- • $!< &1 • 7; !<

    ='+( )9 •  02# ,/ >35%  *. >35% $!<   6:48    !< >35% $!<
  8. Density Estimation in Monolingual Space • %$ •  $

    ! % " • % #   %$ x_i "  $ & ! %
  9. Density Matching • "$+< ;7#?,.2   ! • 

    684:(&5% #>03 • KL  -*)  +< • (&'/ Normalizing flows x #> y #>  1= #> 9= #>    03
  10. Density Matching • %'.A @;&D/15#!#$ • #":<8? +)9(&C37  •

    KL #0-, .A  • +)*2 Normalizing flows x &C y &C # 4B &C >B &C y &C x >B= 6E   #!# 37
  11. Conditional Density Matching • Conditional Density Matching •  •

      •   •  •  
  12. Weak Orthogonality Constraint • /4.# + Orthogonality ) • *15"*1/4

    !8$'7"(/4  ,613,  %2  • 7:9 ,-0& 
  13. Weak Supervision with Identical Words •    

     
  14. Objectives for DeMa-BWE •  Conditional Density Matching Weak Orthogonality

    Constraint Weak Supervision with Identical Words
  15. Cross-Domain Similarity Local Scaling • CSLS •  % •

    CSLS-D •  '* #") ! $& '* k-NN ( '*
  16. Iterative Procrustes Refinement •   X  Y 

    •  
  17. Experiment • MUSE  • English ó Spanish; Japanese; Finnish;

    ... • Pretrained Word Embedding: FastText w/ Wikipedia • Normalizing, Centering •   : 0.01 (en), 0.015 (morph-rich), 0.02 (others) • Vocabulary: 10,000 (en-ja), 20,000 (other pairs) • Loss: • back-translation loss: λ = 0.5 • supervised loss: α = 5 (en-zh), 10 (other pairs)
  18. Precision@1 for MUSE BLI task     

        SL-unsup-ID 
  19. Morphologically complex languages

  20. Pearson rank correlation •    

  21. Ablation study • Identical Words  • en-ja  identical

    words   • Density matching loss  • Back-translation loss    •  
  22. Conclusion • -,0!$// (1)' • &,(1# #*%  • .

     • Identical Words ++"