Upgrade to Pro — share decks privately, control downloads, hide ads and more …

論文紹介:Unsupervised Learning of Distributional Re...

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for Koki Washio Koki Washio
August 04, 2018
160

論文紹介:Unsupervised Learning of Distributional Relation Vectors

Avatar for Koki Washio

Koki Washio

August 04, 2018

Transcript

  1.  • A6) /5>I7 • E621FJ  • GloVeG-D:HB+ •

    "($ 39 .; • C0 • QA =?#$&*8  • Relation Extraction4K'% <!$ @, 
  2.     • & • Paris is a

    capital city of France ($*) • Macron succeeded Hollande as president ((+0) • " • , -RNNCNN • "#"Relation Extraction • "  • )/ ![Mikolov+ 2013] • , ' %)/ .[Fan+ 2015]
  3.  • (!, #* $%,& '#! • & )+* PMI,

    $ ! • & % "- )+. 
  4.  • ``To the best of our knowledge, existing methods

    for learning relation vectors are all based on manipulating pre-trained word vectors.’’ • )+ &  '$,"(  • &  ,"'$# )*  %!… • Latent Relational Analysis [Turney, 2006] • Universal Schemas [Riedel+ 2013] • Primal-Dual Relation Semantic Spaces [Oiwa and Tsujii, 2014]
  5. GloVe quick brown fox jump over ... ... 1. .0'

      )-.0 1$ -,+2 !& 2! ∈ ℝ|%|×|'| 2.  ("% *)/ # X i X j:Xij 6=0 f(Xij)(wi · ˜ wj + bi + ˜ bj log Xij)2 <latexit sha1_base64="sPTlIOGfi3h1788XQs98+HJUNGE=">AAAC6HichVHLahRBFD1pX3F8ZNSN4KZwiIwEh9tBULIKunGZh5MMpGPT3VMz1qT6YXfNhNjMD7hyF8SVARfiB/gBbvQDXOQTxIWLCCq48HZPi2gw3qb63jp1zq06XD/RKjNE+1PWseMnTp6aPl07c/bc+Zn6hYtrWTxMA9kOYh2nHd/LpFaRbBtltOwkqfRCX8t1f+tucb4+kmmm4ui+2UnkZuj1I9VTgWcYcuuBkw1DV4ky5YOFjpurwVg4kXwkaCx6zQlwXTS3C1bQjY1wjNJdmW+7TJwTPuNzvzC/wG4IR8d9USkfzLv1BrWoDHG4sKuigSqW4vobOOgiRoAhQkhEMFxreMj424ANQsLYJnLGUq5UeS4xRo21Q2ZJZniMbvG/z7uNCo14X/TMSnXAt2heKSsFZukDvaIDekev6SP9+GevvOxRvGWHsz/RysSdeXJ59et/VSFng4e/VUcofGYf7cmgh9ulF8XekhIpXAaT/qPHuwerCyuz+TXao0/s7wXt01t2GI2+BC+X5cpz1HhA9t/jOFyszbdsatnLNxuLd6pRTeMKrqLJ87iFRdzDEtp873t8xjd8twbWU2vXejahWlOV5hL+CGvvJylEthY=</latexit> <latexit sha1_base64="sPTlIOGfi3h1788XQs98+HJUNGE=">AAAC6HichVHLahRBFD1pX3F8ZNSN4KZwiIwEh9tBULIKunGZh5MMpGPT3VMz1qT6YXfNhNjMD7hyF8SVARfiB/gBbvQDXOQTxIWLCCq48HZPi2gw3qb63jp1zq06XD/RKjNE+1PWseMnTp6aPl07c/bc+Zn6hYtrWTxMA9kOYh2nHd/LpFaRbBtltOwkqfRCX8t1f+tucb4+kmmm4ui+2UnkZuj1I9VTgWcYcuuBkw1DV4ky5YOFjpurwVg4kXwkaCx6zQlwXTS3C1bQjY1wjNJdmW+7TJwTPuNzvzC/wG4IR8d9USkfzLv1BrWoDHG4sKuigSqW4vobOOgiRoAhQkhEMFxreMj424ANQsLYJnLGUq5UeS4xRo21Q2ZJZniMbvG/z7uNCo14X/TMSnXAt2heKSsFZukDvaIDekev6SP9+GevvOxRvGWHsz/RysSdeXJ59et/VSFng4e/VUcofGYf7cmgh9ulF8XekhIpXAaT/qPHuwerCyuz+TXao0/s7wXt01t2GI2+BC+X5cpz1HhA9t/jOFyszbdsatnLNxuLd6pRTeMKrqLJ87iFRdzDEtp873t8xjd8twbWU2vXejahWlOV5hL+CGvvJylEthY=</latexit> <latexit sha1_base64="sPTlIOGfi3h1788XQs98+HJUNGE=">AAAC6HichVHLahRBFD1pX3F8ZNSN4KZwiIwEh9tBULIKunGZh5MMpGPT3VMz1qT6YXfNhNjMD7hyF8SVARfiB/gBbvQDXOQTxIWLCCq48HZPi2gw3qb63jp1zq06XD/RKjNE+1PWseMnTp6aPl07c/bc+Zn6hYtrWTxMA9kOYh2nHd/LpFaRbBtltOwkqfRCX8t1f+tucb4+kmmm4ui+2UnkZuj1I9VTgWcYcuuBkw1DV4ky5YOFjpurwVg4kXwkaCx6zQlwXTS3C1bQjY1wjNJdmW+7TJwTPuNzvzC/wG4IR8d9USkfzLv1BrWoDHG4sKuigSqW4vobOOgiRoAhQkhEMFxreMj424ANQsLYJnLGUq5UeS4xRo21Q2ZJZniMbvG/z7uNCo14X/TMSnXAt2heKSsFZukDvaIDekev6SP9+GevvOxRvGWHsz/RysSdeXJ59et/VSFng4e/VUcofGYf7cmgh9ulF8XekhIpXAaT/qPHuwerCyuz+TXao0/s7wXt01t2GI2+BC+X5cpz1HhA9t/jOFyszbdsatnLNxuLd6pRTeMKrqLJ87iFRdzDEtp873t8xjd8twbWU2vXejahWlOV5hL+CGvvJylEthY=</latexit> <latexit sha1_base64="sPTlIOGfi3h1788XQs98+HJUNGE=">AAAC6HichVHLahRBFD1pX3F8ZNSN4KZwiIwEh9tBULIKunGZh5MMpGPT3VMz1qT6YXfNhNjMD7hyF8SVARfiB/gBbvQDXOQTxIWLCCq48HZPi2gw3qb63jp1zq06XD/RKjNE+1PWseMnTp6aPl07c/bc+Zn6hYtrWTxMA9kOYh2nHd/LpFaRbBtltOwkqfRCX8t1f+tucb4+kmmm4ui+2UnkZuj1I9VTgWcYcuuBkw1DV4ky5YOFjpurwVg4kXwkaCx6zQlwXTS3C1bQjY1wjNJdmW+7TJwTPuNzvzC/wG4IR8d9USkfzLv1BrWoDHG4sKuigSqW4vobOOgiRoAhQkhEMFxreMj424ANQsLYJnLGUq5UeS4xRo21Q2ZJZniMbvG/z7uNCo14X/TMSnXAt2heKSsFZukDvaIDekev6SP9+GevvOxRvGWHsz/RysSdeXJ59et/VSFng4e/VUcofGYf7cmgh9ulF8XekhIpXAaT/qPHuwerCyuz+TXao0/s7wXt01t2GI2+BC+X5cpz1HhA9t/jOFyszbdsatnLNxuLd6pRTeMKrqLJ87iFRdzDEtp873t8xjd8twbWU2vXejahWlOV5hL+CGvvJylEthY=</latexit>
  6. A Variant of GloVe 1.  X i X j:Xij

    6=0 f(Xij)(wi · ˜ wj + bi + ˜ bj log Xij)2 <latexit sha1_base64="sPTlIOGfi3h1788XQs98+HJUNGE=">AAAC6HichVHLahRBFD1pX3F8ZNSN4KZwiIwEh9tBULIKunGZh5MMpGPT3VMz1qT6YXfNhNjMD7hyF8SVARfiB/gBbvQDXOQTxIWLCCq48HZPi2gw3qb63jp1zq06XD/RKjNE+1PWseMnTp6aPl07c/bc+Zn6hYtrWTxMA9kOYh2nHd/LpFaRbBtltOwkqfRCX8t1f+tucb4+kmmm4ui+2UnkZuj1I9VTgWcYcuuBkw1DV4ky5YOFjpurwVg4kXwkaCx6zQlwXTS3C1bQjY1wjNJdmW+7TJwTPuNzvzC/wG4IR8d9USkfzLv1BrWoDHG4sKuigSqW4vobOOgiRoAhQkhEMFxreMj424ANQsLYJnLGUq5UeS4xRo21Q2ZJZniMbvG/z7uNCo14X/TMSnXAt2heKSsFZukDvaIDekev6SP9+GevvOxRvGWHsz/RysSdeXJ59et/VSFng4e/VUcofGYf7cmgh9ulF8XekhIpXAaT/qPHuwerCyuz+TXao0/s7wXt01t2GI2+BC+X5cpz1HhA9t/jOFyszbdsatnLNxuLd6pRTeMKrqLJ87iFRdzDEtp873t8xjd8twbWU2vXejahWlOV5hL+CGvvJylEthY=</latexit> <latexit sha1_base64="sPTlIOGfi3h1788XQs98+HJUNGE=">AAAC6HichVHLahRBFD1pX3F8ZNSN4KZwiIwEh9tBULIKunGZh5MMpGPT3VMz1qT6YXfNhNjMD7hyF8SVARfiB/gBbvQDXOQTxIWLCCq48HZPi2gw3qb63jp1zq06XD/RKjNE+1PWseMnTp6aPl07c/bc+Zn6hYtrWTxMA9kOYh2nHd/LpFaRbBtltOwkqfRCX8t1f+tucb4+kmmm4ui+2UnkZuj1I9VTgWcYcuuBkw1DV4ky5YOFjpurwVg4kXwkaCx6zQlwXTS3C1bQjY1wjNJdmW+7TJwTPuNzvzC/wG4IR8d9USkfzLv1BrWoDHG4sKuigSqW4vobOOgiRoAhQkhEMFxreMj424ANQsLYJnLGUq5UeS4xRo21Q2ZJZniMbvG/z7uNCo14X/TMSnXAt2heKSsFZukDvaIDekev6SP9+GevvOxRvGWHsz/RysSdeXJ59et/VSFng4e/VUcofGYf7cmgh9ulF8XekhIpXAaT/qPHuwerCyuz+TXao0/s7wXt01t2GI2+BC+X5cpz1HhA9t/jOFyszbdsatnLNxuLd6pRTeMKrqLJ87iFRdzDEtp873t8xjd8twbWU2vXejahWlOV5hL+CGvvJylEthY=</latexit> <latexit sha1_base64="sPTlIOGfi3h1788XQs98+HJUNGE=">AAAC6HichVHLahRBFD1pX3F8ZNSN4KZwiIwEh9tBULIKunGZh5MMpGPT3VMz1qT6YXfNhNjMD7hyF8SVARfiB/gBbvQDXOQTxIWLCCq48HZPi2gw3qb63jp1zq06XD/RKjNE+1PWseMnTp6aPl07c/bc+Zn6hYtrWTxMA9kOYh2nHd/LpFaRbBtltOwkqfRCX8t1f+tucb4+kmmm4ui+2UnkZuj1I9VTgWcYcuuBkw1DV4ky5YOFjpurwVg4kXwkaCx6zQlwXTS3C1bQjY1wjNJdmW+7TJwTPuNzvzC/wG4IR8d9USkfzLv1BrWoDHG4sKuigSqW4vobOOgiRoAhQkhEMFxreMj424ANQsLYJnLGUq5UeS4xRo21Q2ZJZniMbvG/z7uNCo14X/TMSnXAt2heKSsFZukDvaIDekev6SP9+GevvOxRvGWHsz/RysSdeXJ59et/VSFng4e/VUcofGYf7cmgh9ulF8XekhIpXAaT/qPHuwerCyuz+TXao0/s7wXt01t2GI2+BC+X5cpz1HhA9t/jOFyszbdsatnLNxuLd6pRTeMKrqLJ87iFRdzDEtp873t8xjd8twbWU2vXejahWlOV5hL+CGvvJylEthY=</latexit> <latexit sha1_base64="sPTlIOGfi3h1788XQs98+HJUNGE=">AAAC6HichVHLahRBFD1pX3F8ZNSN4KZwiIwEh9tBULIKunGZh5MMpGPT3VMz1qT6YXfNhNjMD7hyF8SVARfiB/gBbvQDXOQTxIWLCCq48HZPi2gw3qb63jp1zq06XD/RKjNE+1PWseMnTp6aPl07c/bc+Zn6hYtrWTxMA9kOYh2nHd/LpFaRbBtltOwkqfRCX8t1f+tucb4+kmmm4ui+2UnkZuj1I9VTgWcYcuuBkw1DV4ky5YOFjpurwVg4kXwkaCx6zQlwXTS3C1bQjY1wjNJdmW+7TJwTPuNzvzC/wG4IR8d9USkfzLv1BrWoDHG4sKuigSqW4vobOOgiRoAhQkhEMFxreMj424ANQsLYJnLGUq5UeS4xRo21Q2ZJZniMbvG/z7uNCo14X/TMSnXAt2heKSsFZukDvaIDekev6SP9+GevvOxRvGWHsz/RysSdeXJ59et/VSFng4e/VUcofGYf7cmgh9ulF8XekhIpXAaT/qPHuwerCyuz+TXao0/s7wXt01t2GI2+BC+X5cpz1HhA9t/jOFyszbdsatnLNxuLd6pRTeMKrqLJ87iFRdzDEtp873t8xjd8twbWU2vXejahWlOV5hL+CGvvJylEthY=</latexit> bi 7! bi + log Xi⇤ <latexit sha1_base64="prfydr+MgRPTVXMpdw7ca2fB27s=">AAAConichVHLShxBFD22xsck0VE3ggiNEyUYGG6LoLgS3Ygrx3F0wJGmuy0nhf2iu2ZAB1fu/AEXrhRciDtd+AFu/AEXfkLIUsFNFrnd0xASid6iqk6duudWHa4dujJWRI8dWmfXh+6e3r7cx0+f+wfyg0MbcdCIHFFxAjeIqrYVC1f6oqKkckU1jITl2a7YtPeWkvvNpohiGfjraj8U255V9+WudCzFlJkfs02p1zwrjFWgJ/ibXnODul41W3Lq0MwXqEhp6K+BkYECslgN8reoYQcBHDTgQcCHYuzCQsxjCwYIIXPbaDEXMZLpvcAhcqxtcJbgDIvZPV7rfNrKWJ/PSc04VTv8isszYqWOCXqgS3qie7qiH/Trv7VaaY3kL/u8222tCM2B45Hyy7sqj3eF739Ubyhszn7bk8Iu5lIvkr2FKZO4dNr1mwcnT+X5tYnWJJ3TT/Z3Ro90xw795rNzURJrp8hxg4x/2/EabEwXDSoapZnCwmLWql6MYhxfuR+zWMAyVlHhd49wiWvcaF+0Fa2kldupWkemGcZfodV+Axfemq8=</latexit> <latexit sha1_base64="prfydr+MgRPTVXMpdw7ca2fB27s=">AAAConichVHLShxBFD22xsck0VE3ggiNEyUYGG6LoLgS3Ygrx3F0wJGmuy0nhf2iu2ZAB1fu/AEXrhRciDtd+AFu/AEXfkLIUsFNFrnd0xASid6iqk6duudWHa4dujJWRI8dWmfXh+6e3r7cx0+f+wfyg0MbcdCIHFFxAjeIqrYVC1f6oqKkckU1jITl2a7YtPeWkvvNpohiGfjraj8U255V9+WudCzFlJkfs02p1zwrjFWgJ/ibXnODul41W3Lq0MwXqEhp6K+BkYECslgN8reoYQcBHDTgQcCHYuzCQsxjCwYIIXPbaDEXMZLpvcAhcqxtcJbgDIvZPV7rfNrKWJ/PSc04VTv8isszYqWOCXqgS3qie7qiH/Trv7VaaY3kL/u8222tCM2B45Hyy7sqj3eF739Ubyhszn7bk8Iu5lIvkr2FKZO4dNr1mwcnT+X5tYnWJJ3TT/Z3Ro90xw795rNzURJrp8hxg4x/2/EabEwXDSoapZnCwmLWql6MYhxfuR+zWMAyVlHhd49wiWvcaF+0Fa2kldupWkemGcZfodV+Axfemq8=</latexit> <latexit sha1_base64="prfydr+MgRPTVXMpdw7ca2fB27s=">AAAConichVHLShxBFD22xsck0VE3ggiNEyUYGG6LoLgS3Ygrx3F0wJGmuy0nhf2iu2ZAB1fu/AEXrhRciDtd+AFu/AEXfkLIUsFNFrnd0xASid6iqk6duudWHa4dujJWRI8dWmfXh+6e3r7cx0+f+wfyg0MbcdCIHFFxAjeIqrYVC1f6oqKkckU1jITl2a7YtPeWkvvNpohiGfjraj8U255V9+WudCzFlJkfs02p1zwrjFWgJ/ibXnODul41W3Lq0MwXqEhp6K+BkYECslgN8reoYQcBHDTgQcCHYuzCQsxjCwYIIXPbaDEXMZLpvcAhcqxtcJbgDIvZPV7rfNrKWJ/PSc04VTv8isszYqWOCXqgS3qie7qiH/Trv7VaaY3kL/u8222tCM2B45Hyy7sqj3eF739Ubyhszn7bk8Iu5lIvkr2FKZO4dNr1mwcnT+X5tYnWJJ3TT/Z3Ro90xw795rNzURJrp8hxg4x/2/EabEwXDSoapZnCwmLWql6MYhxfuR+zWMAyVlHhd49wiWvcaF+0Fa2kldupWkemGcZfodV+Axfemq8=</latexit> <latexit sha1_base64="prfydr+MgRPTVXMpdw7ca2fB27s=">AAAConichVHLShxBFD22xsck0VE3ggiNEyUYGG6LoLgS3Ygrx3F0wJGmuy0nhf2iu2ZAB1fu/AEXrhRciDtd+AFu/AEXfkLIUsFNFrnd0xASid6iqk6duudWHa4dujJWRI8dWmfXh+6e3r7cx0+f+wfyg0MbcdCIHFFxAjeIqrYVC1f6oqKkckU1jITl2a7YtPeWkvvNpohiGfjraj8U255V9+WudCzFlJkfs02p1zwrjFWgJ/ibXnODul41W3Lq0MwXqEhp6K+BkYECslgN8reoYQcBHDTgQcCHYuzCQsxjCwYIIXPbaDEXMZLpvcAhcqxtcJbgDIvZPV7rfNrKWJ/PSc04VTv8isszYqWOCXqgS3qie7qiH/Trv7VaaY3kL/u8222tCM2B45Hyy7sqj3eF739Ubyhszn7bk8Iu5lIvkr2FKZO4dNr1mwcnT+X5tYnWJJ3TT/Z3Ro90xw795rNzURJrp8hxg4x/2/EabEwXDSoapZnCwmLWql6MYhxfuR+zWMAyVlHhd49wiWvcaF+0Fa2kldupWkemGcZfodV+Axfemq8=</latexit> ˜ bj 7! ˜ bj + log X⇤j <latexit sha1_base64="EOh6uC5wKzwWJsKWohFxqm17niI=">AAACsnichVHLShxBFD121OhodIybQDZNBkUiDrdFiGQlySZLX6NDHGm623IsrX7QXTNgmvmBbF1k4UrBheQDxK1u8gNZ+AmSpYFsXOROT4Mv1FtU1alT99yqw3UjJRNNdNFlvOju6X3Z118YGHw1NFwceb2ShI3YExUvVGFcdZ1EKBmIipZaiWoUC8d3lVh1dz6371ebIk5kGCzr3Uis+049kJvSczRTdnGqpqXaEKlrb7fMmu9EiQ7N29ykWVNh3aza6fvtll0sUZmyMB8CKwcl5DEfFk9QwwZCeGjAh0AAzVjBQcJjDRYIEXPrSJmLGcnsXqCFAmsbnCU4w2F2h9c6n9ZyNuBzu2aSqT1+RfGMWWlijH7TMV3RL/pJl3T9aK00q9H+yy7vbkcrInv4+5ulf8+qfN41tm5UTyhczn7ak8YmZjMvkr1FGdN26XXqN7/9uFr6uDiWjtMh/WF/B3RB5+wwaP71jhbE4j4K3CDrfjsegpXpskVla2GmNPcpb1Uf3uIdJrgfHzCHL5hHhd/dwynOcG7MGF8Nx/A6qUZXrhnFnTDUf9+foh4=</latexit> <latexit sha1_base64="EOh6uC5wKzwWJsKWohFxqm17niI=">AAACsnichVHLShxBFD121OhodIybQDZNBkUiDrdFiGQlySZLX6NDHGm623IsrX7QXTNgmvmBbF1k4UrBheQDxK1u8gNZ+AmSpYFsXOROT4Mv1FtU1alT99yqw3UjJRNNdNFlvOju6X3Z118YGHw1NFwceb2ShI3YExUvVGFcdZ1EKBmIipZaiWoUC8d3lVh1dz6371ebIk5kGCzr3Uis+049kJvSczRTdnGqpqXaEKlrb7fMmu9EiQ7N29ykWVNh3aza6fvtll0sUZmyMB8CKwcl5DEfFk9QwwZCeGjAh0AAzVjBQcJjDRYIEXPrSJmLGcnsXqCFAmsbnCU4w2F2h9c6n9ZyNuBzu2aSqT1+RfGMWWlijH7TMV3RL/pJl3T9aK00q9H+yy7vbkcrInv4+5ulf8+qfN41tm5UTyhczn7ak8YmZjMvkr1FGdN26XXqN7/9uFr6uDiWjtMh/WF/B3RB5+wwaP71jhbE4j4K3CDrfjsegpXpskVla2GmNPcpb1Uf3uIdJrgfHzCHL5hHhd/dwynOcG7MGF8Nx/A6qUZXrhnFnTDUf9+foh4=</latexit> <latexit sha1_base64="EOh6uC5wKzwWJsKWohFxqm17niI=">AAACsnichVHLShxBFD121OhodIybQDZNBkUiDrdFiGQlySZLX6NDHGm623IsrX7QXTNgmvmBbF1k4UrBheQDxK1u8gNZ+AmSpYFsXOROT4Mv1FtU1alT99yqw3UjJRNNdNFlvOju6X3Z118YGHw1NFwceb2ShI3YExUvVGFcdZ1EKBmIipZaiWoUC8d3lVh1dz6371ebIk5kGCzr3Uis+049kJvSczRTdnGqpqXaEKlrb7fMmu9EiQ7N29ykWVNh3aza6fvtll0sUZmyMB8CKwcl5DEfFk9QwwZCeGjAh0AAzVjBQcJjDRYIEXPrSJmLGcnsXqCFAmsbnCU4w2F2h9c6n9ZyNuBzu2aSqT1+RfGMWWlijH7TMV3RL/pJl3T9aK00q9H+yy7vbkcrInv4+5ulf8+qfN41tm5UTyhczn7ak8YmZjMvkr1FGdN26XXqN7/9uFr6uDiWjtMh/WF/B3RB5+wwaP71jhbE4j4K3CDrfjsegpXpskVla2GmNPcpb1Uf3uIdJrgfHzCHL5hHhd/dwynOcG7MGF8Nx/A6qUZXrhnFnTDUf9+foh4=</latexit> <latexit sha1_base64="EOh6uC5wKzwWJsKWohFxqm17niI=">AAACsnichVHLShxBFD121OhodIybQDZNBkUiDrdFiGQlySZLX6NDHGm623IsrX7QXTNgmvmBbF1k4UrBheQDxK1u8gNZ+AmSpYFsXOROT4Mv1FtU1alT99yqw3UjJRNNdNFlvOju6X3Z118YGHw1NFwceb2ShI3YExUvVGFcdZ1EKBmIipZaiWoUC8d3lVh1dz6371ebIk5kGCzr3Uis+049kJvSczRTdnGqpqXaEKlrb7fMmu9EiQ7N29ykWVNh3aza6fvtll0sUZmyMB8CKwcl5DEfFk9QwwZCeGjAh0AAzVjBQcJjDRYIEXPrSJmLGcnsXqCFAmsbnCU4w2F2h9c6n9ZyNuBzu2aSqT1+RfGMWWlijH7TMV3RL/pJl3T9aK00q9H+yy7vbkcrInv4+5ulf8+qfN41tm5UTyhczn7ak8YmZjMvkr1FGdN26XXqN7/9uFr6uDiWjtMh/WF/B3RB5+wwaP71jhbE4j4K3CDrfjsegpXpskVla2GmNPcpb1Uf3uIdJrgfHzCHL5hHhd/dwynOcG7MGF8Nx/A6qUZXrhnFnTDUf9+foh4=</latexit> X i X j:Xij 6=0 f(Xij)(wi · ˜ wj + bi + ˜ bj PMI(i, j))2 <latexit sha1_base64="PL2v/enoRsRqrJWz51RJRyNv890=">AAAC5nichVHLbtNAFD01rxIKDbBBYjMiapWoNLqukEBdVbCBBVLakjZSUyzbmbST+oU9SVWs/EAX3aEu2AASC8QH8AEs4AdY9BMQYlVENyy4dowQVJRrje+dM+fcmaPrRJ5KNNHBmHHq9Jmz58bPly5MXLw0Wb58ZSUJ+7Erm27ohXHLsRPpqUA2tdKebEWxtH3Hk6vO1r3sfHUg40SFwSO9E8l1394IVFe5tmbIKtvtpO9bSuQp7c23rFT1hqIdyCeChqJbHQE1Ud3OWG4n1KKtldeR6bbFxBnhMD7zC3MybFY0Hj6oqpuiV6s9nrPKFapTHuJ4YRZFBUU0wvI7tNFBCBd9+JAIoLn2YCPhbw0mCBFj60gZi7lS+bnEECXW9pklmWEzusX/Dd6tFWjA+6xnkqtdvsXjFbNSYIo+0Rs6pI/0lj7Tj3/2SvMe2Vt2ODsjrYysyd1ry0f/VfmcNTZ/q05QOMw+2ZNGF3dyL4q9RTmSuXRH/QdP9w+X55em0ml6RV/Y30s6oPfsMBh8c18vyqXnKPGAzL/HcbxYmaubVDcXb1UW7hajGsd13ECV53EbC7iPBpp87wd8xXccGZvGnvHM2B9RjbFCcxV/hPHiJ3+WtBg=</latexit> <latexit sha1_base64="PL2v/enoRsRqrJWz51RJRyNv890=">AAAC5nichVHLbtNAFD01rxIKDbBBYjMiapWoNLqukEBdVbCBBVLakjZSUyzbmbST+oU9SVWs/EAX3aEu2AASC8QH8AEs4AdY9BMQYlVENyy4dowQVJRrje+dM+fcmaPrRJ5KNNHBmHHq9Jmz58bPly5MXLw0Wb58ZSUJ+7Erm27ohXHLsRPpqUA2tdKebEWxtH3Hk6vO1r3sfHUg40SFwSO9E8l1394IVFe5tmbIKtvtpO9bSuQp7c23rFT1hqIdyCeChqJbHQE1Ud3OWG4n1KKtldeR6bbFxBnhMD7zC3MybFY0Hj6oqpuiV6s9nrPKFapTHuJ4YRZFBUU0wvI7tNFBCBd9+JAIoLn2YCPhbw0mCBFj60gZi7lS+bnEECXW9pklmWEzusX/Dd6tFWjA+6xnkqtdvsXjFbNSYIo+0Rs6pI/0lj7Tj3/2SvMe2Vt2ODsjrYysyd1ry0f/VfmcNTZ/q05QOMw+2ZNGF3dyL4q9RTmSuXRH/QdP9w+X55em0ml6RV/Y30s6oPfsMBh8c18vyqXnKPGAzL/HcbxYmaubVDcXb1UW7hajGsd13ECV53EbC7iPBpp87wd8xXccGZvGnvHM2B9RjbFCcxV/hPHiJ3+WtBg=</latexit> <latexit sha1_base64="PL2v/enoRsRqrJWz51RJRyNv890=">AAAC5nichVHLbtNAFD01rxIKDbBBYjMiapWoNLqukEBdVbCBBVLakjZSUyzbmbST+oU9SVWs/EAX3aEu2AASC8QH8AEs4AdY9BMQYlVENyy4dowQVJRrje+dM+fcmaPrRJ5KNNHBmHHq9Jmz58bPly5MXLw0Wb58ZSUJ+7Erm27ohXHLsRPpqUA2tdKebEWxtH3Hk6vO1r3sfHUg40SFwSO9E8l1394IVFe5tmbIKtvtpO9bSuQp7c23rFT1hqIdyCeChqJbHQE1Ud3OWG4n1KKtldeR6bbFxBnhMD7zC3MybFY0Hj6oqpuiV6s9nrPKFapTHuJ4YRZFBUU0wvI7tNFBCBd9+JAIoLn2YCPhbw0mCBFj60gZi7lS+bnEECXW9pklmWEzusX/Dd6tFWjA+6xnkqtdvsXjFbNSYIo+0Rs6pI/0lj7Tj3/2SvMe2Vt2ODsjrYysyd1ry0f/VfmcNTZ/q05QOMw+2ZNGF3dyL4q9RTmSuXRH/QdP9w+X55em0ml6RV/Y30s6oPfsMBh8c18vyqXnKPGAzL/HcbxYmaubVDcXb1UW7hajGsd13ECV53EbC7iPBpp87wd8xXccGZvGnvHM2B9RjbFCcxV/hPHiJ3+WtBg=</latexit> <latexit sha1_base64="PL2v/enoRsRqrJWz51RJRyNv890=">AAAC5nichVHLbtNAFD01rxIKDbBBYjMiapWoNLqukEBdVbCBBVLakjZSUyzbmbST+oU9SVWs/EAX3aEu2AASC8QH8AEs4AdY9BMQYlVENyy4dowQVJRrje+dM+fcmaPrRJ5KNNHBmHHq9Jmz58bPly5MXLw0Wb58ZSUJ+7Erm27ohXHLsRPpqUA2tdKebEWxtH3Hk6vO1r3sfHUg40SFwSO9E8l1394IVFe5tmbIKtvtpO9bSuQp7c23rFT1hqIdyCeChqJbHQE1Ud3OWG4n1KKtldeR6bbFxBnhMD7zC3MybFY0Hj6oqpuiV6s9nrPKFapTHuJ4YRZFBUU0wvI7tNFBCBd9+JAIoLn2YCPhbw0mCBFj60gZi7lS+bnEECXW9pklmWEzusX/Dd6tFWjA+6xnkqtdvsXjFbNSYIo+0Rs6pI/0lj7Tj3/2SvMe2Vt2ODsjrYysyd1ry0f/VfmcNTZ/q05QOMw+2ZNGF3dyL4q9RTmSuXRH/QdP9w+X55em0ml6RV/Y30s6oPfsMBh8c18vyqXnKPGAzL/HcbxYmaubVDcXb1UW7hajGsd13ECV53EbC7iPBpp87wd8xXccGZvGnvHM2B9RjbFCcxV/hPHiJ3+WtBg=</latexit>  Reparametarize: PMI(i, j) = log( P(i, j) P(i)P(j) ) = log( XijX⇤⇤ Xi⇤X⇤j ) <latexit sha1_base64="E26r+TwlyVQta5k2bODgnW+ttVI=">AAAC2nicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuEBvh6amTqKGRpKtgqxOTkpytoxKQVJSZXBwBFszRrQbRmgAaQha4gIr46M6sWSGpp1daCeVpgXlZtrWa8gLKBngEYKGAyDKEMZQYoCMgX2M4Qw5DCkM+QzFDKkMuQypDHUAJk5zAkMhQDYTSDIYMBQwFQLJahGihWBGRlguVTGWoZuIB6S4GqUoEqEoGi2UAyHciLhormAfkgM4vBupOBtuQAcRFQpwKDqsFVg5UGnw1OGKw2eGnwB6dZ1WAzQG6pBNJJEL2pBfH8XRLB3wnqygXSJQwZCF14dCQBVeP3UwlDGoMF2C+ZQL8VgEVAvkyGmF9WNf1zsFWQarWawSKD10D/LTS4aXAY6MO8si/JSwNTg2YzcAEjyBA9OjAZYUZ6hgZ6hoEmyg5O0KjiYJBmUGLQAMaHOYMDgwdDAEMo0N7dDA8ZXjG8ZophamLqZOqGKGVihOoRZkABTFMAiQ2u7Q==</latexit> <latexit sha1_base64="E26r+TwlyVQta5k2bODgnW+ttVI=">AAAC2nicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuEBvh6amTqKGRpKtgqxOTkpytoxKQVJSZXBwBFszRrQbRmgAaQha4gIr46M6sWSGpp1daCeVpgXlZtrWa8gLKBngEYKGAyDKEMZQYoCMgX2M4Qw5DCkM+QzFDKkMuQypDHUAJk5zAkMhQDYTSDIYMBQwFQLJahGihWBGRlguVTGWoZuIB6S4GqUoEqEoGi2UAyHciLhormAfkgM4vBupOBtuQAcRFQpwKDqsFVg5UGnw1OGKw2eGnwB6dZ1WAzQG6pBNJJEL2pBfH8XRLB3wnqygXSJQwZCF14dCQBVeP3UwlDGoMF2C+ZQL8VgEVAvkyGmF9WNf1zsFWQarWawSKD10D/LTS4aXAY6MO8si/JSwNTg2YzcAEjyBA9OjAZYUZ6hgZ6hoEmyg5O0KjiYJBmUGLQAMaHOYMDgwdDAEMo0N7dDA8ZXjG8ZophamLqZOqGKGVihOoRZkABTFMAiQ2u7Q==</latexit> <latexit sha1_base64="E26r+TwlyVQta5k2bODgnW+ttVI=">AAAC2nicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuEBvh6amTqKGRpKtgqxOTkpytoxKQVJSZXBwBFszRrQbRmgAaQha4gIr46M6sWSGpp1daCeVpgXlZtrWa8gLKBngEYKGAyDKEMZQYoCMgX2M4Qw5DCkM+QzFDKkMuQypDHUAJk5zAkMhQDYTSDIYMBQwFQLJahGihWBGRlguVTGWoZuIB6S4GqUoEqEoGi2UAyHciLhormAfkgM4vBupOBtuQAcRFQpwKDqsFVg5UGnw1OGKw2eGnwB6dZ1WAzQG6pBNJJEL2pBfH8XRLB3wnqygXSJQwZCF14dCQBVeP3UwlDGoMF2C+ZQL8VgEVAvkyGmF9WNf1zsFWQarWawSKD10D/LTS4aXAY6MO8si/JSwNTg2YzcAEjyBA9OjAZYUZ6hgZ6hoEmyg5O0KjiYJBmUGLQAMaHOYMDgwdDAEMo0N7dDA8ZXjG8ZophamLqZOqGKGVihOoRZkABTFMAiQ2u7Q==</latexit> <latexit sha1_base64="E26r+TwlyVQta5k2bODgnW+ttVI=">AAAC2nicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQuEBvh6amTqKGRpKtgqxOTkpytoxKQVJSZXBwBFszRrQbRmgAaQha4gIr46M6sWSGpp1daCeVpgXlZtrWa8gLKBngEYKGAyDKEMZQYoCMgX2M4Qw5DCkM+QzFDKkMuQypDHUAJk5zAkMhQDYTSDIYMBQwFQLJahGihWBGRlguVTGWoZuIB6S4GqUoEqEoGi2UAyHciLhormAfkgM4vBupOBtuQAcRFQpwKDqsFVg5UGnw1OGKw2eGnwB6dZ1WAzQG6pBNJJEL2pBfH8XRLB3wnqygXSJQwZCF14dCQBVeP3UwlDGoMF2C+ZQL8VgEVAvkyGmF9WNf1zsFWQarWawSKD10D/LTS4aXAY6MO8si/JSwNTg2YzcAEjyBA9OjAZYUZ6hgZ6hoEmyg5O0KjiYJBmUGLQAMaHOYMDgwdDAEMo0N7dDA8ZXjG8ZophamLqZOqGKGVihOoRZkABTFMAiQ2u7Q==</latexit>
  7. A Variant of GloVe 2. PMI Smoothed PMI  

    P(i) = Xi⇤ + ↵ X⇤⇤ + n↵ , P(j) = X⇤j + ↵ X⇤⇤ + n↵ , P(i, j) = Xij + ↵ X⇤⇤ + n↵ <latexit sha1_base64="2kqQK9uAUiMPHlmOvZ0r8yB6kPM=">AAADEnichVFNS+NQFL2JztipOu3oZgY3waLUVsrNICiCIDObWdaPasFKeYmv9bVpEpK0oKF/QHAnuHA1A7MY/AGCO3Hjwq0Lf4K4rODGhTdpYKhiveHl3XveOfe9w9VsQ7ge4q0kDwx++DgU+xQfHhn9nEh+Gdtwraaj84JuGZZT1JjLDWHygic8gxdth7OGZvBNrf4zON9scccVlrnu7dl8u8GqpqgInXkElZOH+bSYUZaUUsVhul8s+yLTVrJKiRn2LmsHQCbTzppRPavk07Ueeqb2Dl3M9gpEP0E5mcIchqG8TtQoSUEUeSt5BiXYAQt0aEIDOJjgUW4AA5e+LVABwSZsG3zCHMpEeM6hDXHSNonFicEIrdO/StVWhJpUBz3dUK3TLQYth5QKTOEN/sMOXuEp3uHTm738sEfwlj3ata6W2+XEwde1x3dVDdo92P2v6qPQiN3fkwcVWAi9CPJmh0jgUu/2b+0fd9YWV6f8afyD9+TvN97iJTk0Ww/63xW+egJxGpD6chyvk43vORVz6spcavlHNKoYTMAkpGke87AMvyAPBbq3I32TJqWUfCSfyxfyZZcqS5FmHHpCvn4GRxHCBw==</latexit> <latexit sha1_base64="2kqQK9uAUiMPHlmOvZ0r8yB6kPM=">AAADEnichVFNS+NQFL2JztipOu3oZgY3waLUVsrNICiCIDObWdaPasFKeYmv9bVpEpK0oKF/QHAnuHA1A7MY/AGCO3Hjwq0Lf4K4rODGhTdpYKhiveHl3XveOfe9w9VsQ7ge4q0kDwx++DgU+xQfHhn9nEh+Gdtwraaj84JuGZZT1JjLDWHygic8gxdth7OGZvBNrf4zON9scccVlrnu7dl8u8GqpqgInXkElZOH+bSYUZaUUsVhul8s+yLTVrJKiRn2LmsHQCbTzppRPavk07Ueeqb2Dl3M9gpEP0E5mcIchqG8TtQoSUEUeSt5BiXYAQt0aEIDOJjgUW4AA5e+LVABwSZsG3zCHMpEeM6hDXHSNonFicEIrdO/StVWhJpUBz3dUK3TLQYth5QKTOEN/sMOXuEp3uHTm738sEfwlj3ata6W2+XEwde1x3dVDdo92P2v6qPQiN3fkwcVWAi9CPJmh0jgUu/2b+0fd9YWV6f8afyD9+TvN97iJTk0Ww/63xW+egJxGpD6chyvk43vORVz6spcavlHNKoYTMAkpGke87AMvyAPBbq3I32TJqWUfCSfyxfyZZcqS5FmHHpCvn4GRxHCBw==</latexit> <latexit sha1_base64="2kqQK9uAUiMPHlmOvZ0r8yB6kPM=">AAADEnichVFNS+NQFL2JztipOu3oZgY3waLUVsrNICiCIDObWdaPasFKeYmv9bVpEpK0oKF/QHAnuHA1A7MY/AGCO3Hjwq0Lf4K4rODGhTdpYKhiveHl3XveOfe9w9VsQ7ge4q0kDwx++DgU+xQfHhn9nEh+Gdtwraaj84JuGZZT1JjLDWHygic8gxdth7OGZvBNrf4zON9scccVlrnu7dl8u8GqpqgInXkElZOH+bSYUZaUUsVhul8s+yLTVrJKiRn2LmsHQCbTzppRPavk07Ueeqb2Dl3M9gpEP0E5mcIchqG8TtQoSUEUeSt5BiXYAQt0aEIDOJjgUW4AA5e+LVABwSZsG3zCHMpEeM6hDXHSNonFicEIrdO/StVWhJpUBz3dUK3TLQYth5QKTOEN/sMOXuEp3uHTm738sEfwlj3ata6W2+XEwde1x3dVDdo92P2v6qPQiN3fkwcVWAi9CPJmh0jgUu/2b+0fd9YWV6f8afyD9+TvN97iJTk0Ww/63xW+egJxGpD6chyvk43vORVz6spcavlHNKoYTMAkpGke87AMvyAPBbq3I32TJqWUfCSfyxfyZZcqS5FmHHpCvn4GRxHCBw==</latexit> <latexit sha1_base64="2kqQK9uAUiMPHlmOvZ0r8yB6kPM=">AAADEnichVFNS+NQFL2JztipOu3oZgY3waLUVsrNICiCIDObWdaPasFKeYmv9bVpEpK0oKF/QHAnuHA1A7MY/AGCO3Hjwq0Lf4K4rODGhTdpYKhiveHl3XveOfe9w9VsQ7ge4q0kDwx++DgU+xQfHhn9nEh+Gdtwraaj84JuGZZT1JjLDWHygic8gxdth7OGZvBNrf4zON9scccVlrnu7dl8u8GqpqgInXkElZOH+bSYUZaUUsVhul8s+yLTVrJKiRn2LmsHQCbTzppRPavk07Ueeqb2Dl3M9gpEP0E5mcIchqG8TtQoSUEUeSt5BiXYAQt0aEIDOJjgUW4AA5e+LVABwSZsG3zCHMpEeM6hDXHSNonFicEIrdO/StVWhJpUBz3dUK3TLQYth5QKTOEN/sMOXuEp3uHTm738sEfwlj3ata6W2+XEwde1x3dVDdo92P2v6qPQiN3fkwcVWAi9CPJmh0jgUu/2b+0fd9YWV6f8afyD9+TvN97iJTk0Ww/63xW+egJxGpD6chyvk43vORVz6spcavlHNKoYTMAkpGke87AMvyAPBbq3I32TJqWUfCSfyxfyZZcqS5FmHHpCvn4GRxHCBw==</latexit>   J 1 j = {i : Xi,j > 0} <latexit sha1_base64="ZiqHMtcWd6TAgRc4T2bKQyDU768=">AAACoXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvIeMVnxVXrGtYq2CrEVGdaRcRXZ+pk1SrYKRjE1MYLKBvoGYCBAibDEMpQZoCCgHyB7QwxDCkM+QzJDKUMuQypDHkMJUB2DkMiQzEQRjMYMhgwFADFYhmqgWJFQFYmWD6VoZaBC6i3FKgqFagiESiaDSTTgbxoqGgekA8ysxisOxloSw4QFwF1KjCoGlw1WGnw2eCEwWqDlwZ/cJpVDTYD5JZKIJ0E0ZtaEM/fJRH8naCuXCBdwpCB0IVHRxJQNX4/lTCkMViA/ZIJ9FsBWATky2SI+WVV0z8HWwWpVqsZLDJ4DfTfQoObBoeBPswr+5K8NDA1aDYDFzCCDNGjA5MRZqRnaKBnGGii7OAEjSoOBmkGJQYNYHyYMzgweDAEMIQC7W1iWMGwkWETkzKTJ1MAUxBEKRMjVI8wAwpgigYAyDqZqQ==</latexit> <latexit sha1_base64="ZiqHMtcWd6TAgRc4T2bKQyDU768=">AAACoXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvIeMVnxVXrGtYq2CrEVGdaRcRXZ+pk1SrYKRjE1MYLKBvoGYCBAibDEMpQZoCCgHyB7QwxDCkM+QzJDKUMuQypDHkMJUB2DkMiQzEQRjMYMhgwFADFYhmqgWJFQFYmWD6VoZaBC6i3FKgqFagiESiaDSTTgbxoqGgekA8ysxisOxloSw4QFwF1KjCoGlw1WGnw2eCEwWqDlwZ/cJpVDTYD5JZKIJ0E0ZtaEM/fJRH8naCuXCBdwpCB0IVHRxJQNX4/lTCkMViA/ZIJ9FsBWATky2SI+WVV0z8HWwWpVqsZLDJ4DfTfQoObBoeBPswr+5K8NDA1aDYDFzCCDNGjA5MRZqRnaKBnGGii7OAEjSoOBmkGJQYNYHyYMzgweDAEMIQC7W1iWMGwkWETkzKTJ1MAUxBEKRMjVI8wAwpgigYAyDqZqQ==</latexit> <latexit sha1_base64="ZiqHMtcWd6TAgRc4T2bKQyDU768=">AAACoXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvIeMVnxVXrGtYq2CrEVGdaRcRXZ+pk1SrYKRjE1MYLKBvoGYCBAibDEMpQZoCCgHyB7QwxDCkM+QzJDKUMuQypDHkMJUB2DkMiQzEQRjMYMhgwFADFYhmqgWJFQFYmWD6VoZaBC6i3FKgqFagiESiaDSTTgbxoqGgekA8ysxisOxloSw4QFwF1KjCoGlw1WGnw2eCEwWqDlwZ/cJpVDTYD5JZKIJ0E0ZtaEM/fJRH8naCuXCBdwpCB0IVHRxJQNX4/lTCkMViA/ZIJ9FsBWATky2SI+WVV0z8HWwWpVqsZLDJ4DfTfQoObBoeBPswr+5K8NDA1aDYDFzCCDNGjA5MRZqRnaKBnGGii7OAEjSoOBmkGJQYNYHyYMzgweDAEMIQC7W1iWMGwkWETkzKTJ1MAUxBEKRMjVI8wAwpgigYAyDqZqQ==</latexit> <latexit sha1_base64="dO1RVWgbv0QBXFnWGIM/DJuxBX8=">AAACdXichVG9TsJQGD1UVEQU3ExciARjHMhXHTROJi6O/MhPgoS05YINpW3aQoLEF2B1cHDSxMH4AD6Aiy/gwCMYR0xcHPwoJEaJ+jXtPffc73y3J0e1Dd31iAYBaSY4OzcfWggvRsJLy9FYpOBabUcTec0yLKekKq4wdFPkPd0zRMl2hNJSDVFUm4ej82JHOK5umcde1xaVltIw9bquKR5T6WosQSnyKz4N5AlIYFJW7AEnqMGChjZaEDDhMTagwOWnDBkEm7kKesw5jHT/XOAcYda2uUtwh8Jsk78N3pUnrMn70UzXV2t8i8Gvw8o4kvRMdzSkJ7qnF/r4dVbPnzH6ly6v6lgr7Gq0v5p7/1fV4tXD6ZfqD4XK3X978lDHnu9FZ2+2z4xcauP5nbPLYW4/m+xt0A29sr9rGtAjOzQ7b9ptRmSvEOZ85J9pTIPCdkqmlJwhhLCGdWxyDLs4wBHSyPN1NfRxIQWlLUke5ygFJoGu4FtJO59jaI2O</latexit> <latexit sha1_base64="xJWArSWqJFV2t3YUPYdLgkPsJ7g=">AAAClnichVE9T9tQFD1x+WoIEDohwWARUXWg0TULKBJVpS6FKQkEIiWpZbsPeMSxLduJRK0sjP0DDExFQgIxwcIPYOkf6MBPQIwgsTD0xomE2gi4lv3OO++e+3x0TM+WQUh0nVDeDAwODY+8TY6mxsYn0pOpjcBt+pYoWa7t+mXTCIQtHVEKZWiLsucLo2HaYtOsf+mcb7aEH0jXWQ/3PFFrGNuO3JKWETKlp2dW9d1v0UetrS6r1Ujmynok53fb6ieVqm09naEsxaX2A60HMuhV3k1foorvcGGhiQYEHISMbRgI+KlAA8FjroaIOZ+RjM8F2kiytsldgjsMZuv83eZdpcc6vO/MDGK1xbfY/PqsVDFHf+iU7ug3ndENPT47K4pndP5lj1ezqxWePvFzau3hVVWD1xA7T6oXFCZ3v+wpxBaWYi+SvXkx03Fpdee3fhzcreWKc9F7OqJb9veLrumKHTqte+u4IIqHSHJA2v9x9IONhaxGWa1AGME0ZvGBY1jEZ3xFHiW+bh8nOMeFklFWlHw3SiXRy/Qd/iml+Bcjgpic</latexit> <latexit sha1_base64="xJWArSWqJFV2t3YUPYdLgkPsJ7g=">AAAClnichVE9T9tQFD1x+WoIEDohwWARUXWg0TULKBJVpS6FKQkEIiWpZbsPeMSxLduJRK0sjP0DDExFQgIxwcIPYOkf6MBPQIwgsTD0xomE2gi4lv3OO++e+3x0TM+WQUh0nVDeDAwODY+8TY6mxsYn0pOpjcBt+pYoWa7t+mXTCIQtHVEKZWiLsucLo2HaYtOsf+mcb7aEH0jXWQ/3PFFrGNuO3JKWETKlp2dW9d1v0UetrS6r1Ujmynok53fb6ieVqm09naEsxaX2A60HMuhV3k1foorvcGGhiQYEHISMbRgI+KlAA8FjroaIOZ+RjM8F2kiytsldgjsMZuv83eZdpcc6vO/MDGK1xbfY/PqsVDFHf+iU7ug3ndENPT47K4pndP5lj1ezqxWePvFzau3hVVWD1xA7T6oXFCZ3v+wpxBaWYi+SvXkx03Fpdee3fhzcreWKc9F7OqJb9veLrumKHTqte+u4IIqHSHJA2v9x9IONhaxGWa1AGME0ZvGBY1jEZ3xFHiW+bh8nOMeFklFWlHw3SiXRy/Qd/iml+Bcjgpic</latexit> <latexit sha1_base64="NCbnV0U+GGAAQkdo4bxQFTiNNMc=">AAACoXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvIeMVnxVXrGtYq2CrEVGdaRcRXZ+pk1SrYKRjE1MYLKBvoGYCBAibDEMpQZoCCgHyB7QwxDCkM+QzJDKUMuQypDHkMJUB2DkMiQzEQRjMYMhgwFADFYhmqgWJFQFYmWD6VoZaBC6i3FKgqFagiESiaDSTTgbxoqGgekA8ysxisOxloSw4QFwF1KjCoGlw1WGnw2eCEwWqDlwZ/cJpVDTYD5JZKIJ0E0ZtaEM/fJRH8naCuXCBdwpCB0IVHRxJQNX4/lTCkMViA/ZIJ9FsBWATky2SI+WVV0z8HWwWpVqsZLDJ4DfTfQoObBoeBPswr+5K8NDA1aDYDFzCCDNGjA5MRZqRnaKBnGGig7OAEjSoOBmkGJQYNYHyYMzgweDAEMIQC7W1iWMGwkWETkzKTJ1MAUxBEKRMjVI8wAwpgigYAxvqZpQ==</latexit> <latexit sha1_base64="ZiqHMtcWd6TAgRc4T2bKQyDU768=">AAACoXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvIeMVnxVXrGtYq2CrEVGdaRcRXZ+pk1SrYKRjE1MYLKBvoGYCBAibDEMpQZoCCgHyB7QwxDCkM+QzJDKUMuQypDHkMJUB2DkMiQzEQRjMYMhgwFADFYhmqgWJFQFYmWD6VoZaBC6i3FKgqFagiESiaDSTTgbxoqGgekA8ysxisOxloSw4QFwF1KjCoGlw1WGnw2eCEwWqDlwZ/cJpVDTYD5JZKIJ0E0ZtaEM/fJRH8naCuXCBdwpCB0IVHRxJQNX4/lTCkMViA/ZIJ9FsBWATky2SI+WVV0z8HWwWpVqsZLDJ4DfTfQoObBoeBPswr+5K8NDA1aDYDFzCCDNGjA5MRZqRnaKBnGGii7OAEjSoOBmkGJQYNYHyYMzgweDAEMIQC7W1iWMGwkWETkzKTJ1MAUxBEKRMjVI8wAwpgigYAyDqZqQ==</latexit> <latexit sha1_base64="ZiqHMtcWd6TAgRc4T2bKQyDU768=">AAACoXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvIeMVnxVXrGtYq2CrEVGdaRcRXZ+pk1SrYKRjE1MYLKBvoGYCBAibDEMpQZoCCgHyB7QwxDCkM+QzJDKUMuQypDHkMJUB2DkMiQzEQRjMYMhgwFADFYhmqgWJFQFYmWD6VoZaBC6i3FKgqFagiESiaDSTTgbxoqGgekA8ysxisOxloSw4QFwF1KjCoGlw1WGnw2eCEwWqDlwZ/cJpVDTYD5JZKIJ0E0ZtaEM/fJRH8naCuXCBdwpCB0IVHRxJQNX4/lTCkMViA/ZIJ9FsBWATky2SI+WVV0z8HWwWpVqsZLDJ4DfTfQoObBoeBPswr+5K8NDA1aDYDFzCCDNGjA5MRZqRnaKBnGGii7OAEjSoOBmkGJQYNYHyYMzgweDAEMIQC7W1iWMGwkWETkzKTJ1MAUxBEKRMjVI8wAwpgigYAyDqZqQ==</latexit> <latexit sha1_base64="ZiqHMtcWd6TAgRc4T2bKQyDU768=">AAACoXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvIeMVnxVXrGtYq2CrEVGdaRcRXZ+pk1SrYKRjE1MYLKBvoGYCBAibDEMpQZoCCgHyB7QwxDCkM+QzJDKUMuQypDHkMJUB2DkMiQzEQRjMYMhgwFADFYhmqgWJFQFYmWD6VoZaBC6i3FKgqFagiESiaDSTTgbxoqGgekA8ysxisOxloSw4QFwF1KjCoGlw1WGnw2eCEwWqDlwZ/cJpVDTYD5JZKIJ0E0ZtaEM/fJRH8naCuXCBdwpCB0IVHRxJQNX4/lTCkMViA/ZIJ9FsBWATky2SI+WVV0z8HWwWpVqsZLDJ4DfTfQoObBoeBPswr+5K8NDA1aDYDFzCCDNGjA5MRZqRnaKBnGGii7OAEjSoOBmkGJQYNYHyYMzgweDAEMIQC7W1iWMGwkWETkzKTJ1MAUxBEKRMjVI8wAwpgigYAyDqZqQ==</latexit> <latexit sha1_base64="ZiqHMtcWd6TAgRc4T2bKQyDU768=">AAACoXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvIeMVnxVXrGtYq2CrEVGdaRcRXZ+pk1SrYKRjE1MYLKBvoGYCBAibDEMpQZoCCgHyB7QwxDCkM+QzJDKUMuQypDHkMJUB2DkMiQzEQRjMYMhgwFADFYhmqgWJFQFYmWD6VoZaBC6i3FKgqFagiESiaDSTTgbxoqGgekA8ysxisOxloSw4QFwF1KjCoGlw1WGnw2eCEwWqDlwZ/cJpVDTYD5JZKIJ0E0ZtaEM/fJRH8naCuXCBdwpCB0IVHRxJQNX4/lTCkMViA/ZIJ9FsBWATky2SI+WVV0z8HWwWpVqsZLDJ4DfTfQoObBoeBPswr+5K8NDA1aDYDFzCCDNGjA5MRZqRnaKBnGGii7OAEjSoOBmkGJQYNYHyYMzgweDAEMIQC7W1iWMGwkWETkzKTJ1MAUxBEKRMjVI8wAwpgigYAyDqZqQ==</latexit> <latexit sha1_base64="ZiqHMtcWd6TAgRc4T2bKQyDU768=">AAACoXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvIeMVnxVXrGtYq2CrEVGdaRcRXZ+pk1SrYKRjE1MYLKBvoGYCBAibDEMpQZoCCgHyB7QwxDCkM+QzJDKUMuQypDHkMJUB2DkMiQzEQRjMYMhgwFADFYhmqgWJFQFYmWD6VoZaBC6i3FKgqFagiESiaDSTTgbxoqGgekA8ysxisOxloSw4QFwF1KjCoGlw1WGnw2eCEwWqDlwZ/cJpVDTYD5JZKIJ0E0ZtaEM/fJRH8naCuXCBdwpCB0IVHRxJQNX4/lTCkMViA/ZIJ9FsBWATky2SI+WVV0z8HWwWpVqsZLDJ4DfTfQoObBoeBPswr+5K8NDA1aDYDFzCCDNGjA5MRZqRnaKBnGGii7OAEjSoOBmkGJQYNYHyYMzgweDAEMIQC7W1iWMGwkWETkzKTJ1MAUxBEKRMjVI8wAwpgigYAyDqZqQ==</latexit> <latexit sha1_base64="ZiqHMtcWd6TAgRc4T2bKQyDU768=">AAACoXicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQvIeMVnxVXrGtYq2CrEVGdaRcRXZ+pk1SrYKRjE1MYLKBvoGYCBAibDEMpQZoCCgHyB7QwxDCkM+QzJDKUMuQypDHkMJUB2DkMiQzEQRjMYMhgwFADFYhmqgWJFQFYmWD6VoZaBC6i3FKgqFagiESiaDSTTgbxoqGgekA8ysxisOxloSw4QFwF1KjCoGlw1WGnw2eCEwWqDlwZ/cJpVDTYD5JZKIJ0E0ZtaEM/fJRH8naCuXCBdwpCB0IVHRxJQNX4/lTCkMViA/ZIJ9FsBWATky2SI+WVV0z8HWwWpVqsZLDJ4DfTfQoObBoeBPswr+5K8NDA1aDYDFzCCDNGjA5MRZqRnaKBnGGii7OAEjSoOBmkGJQYNYHyYMzgweDAEMIQC7W1iWMGwkWETkzKTJ1MAUxBEKRMjVI8wAwpgigYAyDqZqQ==</latexit> 3. !(#$% ) residual variance ' () * +$  X i X j:Xij >0 1 2 j (wi · ˜ wj + ˜ bj PMIS(i, j))2 <latexit sha1_base64="sSmQCzcG9qsFIfcD04TNKxMncXo=">AAAC6nichVFNa9RQFL2NX3X86FQ3gpuHQ2WKOtwMQqULKXZjF8K047QDTRuSN2/Gl758kLyZUkP+gMtuRLqq6EL8Af4AN4JrF/0J4koqCOrCm0xEtFhvSO69551zXw7XjZRMNOLBhHHi5KnTZybPVs6dv3Bxqjp9aTUJhzEXHR6qMO66TiKUDERHS61EN4qF47tKrLlbi/n52kjEiQyDh3onEhu+MwhkX3JHE2RXhZUMfVuyIqUem2ddO5VedhczZvVjh6dmllqJHPiO7W02M1bfztm8F2pmaal6It3ObI/d+NW5eXeLtR4s2e26vMm82dnNpl2tYQOLYEcLsyxqUEYrrL4BC3oQAoch+CAgAE21AgcSetbBBISIsA1ICYupksW5gAwqpB0SSxDDIXSLvgPq1ks0oD6fmRRqTrcoemNSMpjBD/gKD/EdvsaP+OOfs9JiRv4vO5TdsVZE9tSTK+2v/1X5lDU8+q06RuES+3hPGvpwp/AiyVtUILlLPp4/evz0sD2/MpNex+f4ifzt4wG+JYfB6At/uSxW9qBCCzL/XsfRYrXZMLFhLt+uLdwrVzUJV+Ea1Gkfc7AA96EFHbr3PXyGb/DdUMau8czYG1ONiVJzGf4I48VP12q2oQ==</latexit> <latexit sha1_base64="sSmQCzcG9qsFIfcD04TNKxMncXo=">AAAC6nichVFNa9RQFL2NX3X86FQ3gpuHQ2WKOtwMQqULKXZjF8K047QDTRuSN2/Gl758kLyZUkP+gMtuRLqq6EL8Af4AN4JrF/0J4koqCOrCm0xEtFhvSO69551zXw7XjZRMNOLBhHHi5KnTZybPVs6dv3Bxqjp9aTUJhzEXHR6qMO66TiKUDERHS61EN4qF47tKrLlbi/n52kjEiQyDh3onEhu+MwhkX3JHE2RXhZUMfVuyIqUem2ddO5VedhczZvVjh6dmllqJHPiO7W02M1bfztm8F2pmaal6It3ObI/d+NW5eXeLtR4s2e26vMm82dnNpl2tYQOLYEcLsyxqUEYrrL4BC3oQAoch+CAgAE21AgcSetbBBISIsA1ICYupksW5gAwqpB0SSxDDIXSLvgPq1ks0oD6fmRRqTrcoemNSMpjBD/gKD/EdvsaP+OOfs9JiRv4vO5TdsVZE9tSTK+2v/1X5lDU8+q06RuES+3hPGvpwp/AiyVtUILlLPp4/evz0sD2/MpNex+f4ifzt4wG+JYfB6At/uSxW9qBCCzL/XsfRYrXZMLFhLt+uLdwrVzUJV+Ea1Gkfc7AA96EFHbr3PXyGb/DdUMau8czYG1ONiVJzGf4I48VP12q2oQ==</latexit> <latexit sha1_base64="sSmQCzcG9qsFIfcD04TNKxMncXo=">AAAC6nichVFNa9RQFL2NX3X86FQ3gpuHQ2WKOtwMQqULKXZjF8K047QDTRuSN2/Gl758kLyZUkP+gMtuRLqq6EL8Af4AN4JrF/0J4koqCOrCm0xEtFhvSO69551zXw7XjZRMNOLBhHHi5KnTZybPVs6dv3Bxqjp9aTUJhzEXHR6qMO66TiKUDERHS61EN4qF47tKrLlbi/n52kjEiQyDh3onEhu+MwhkX3JHE2RXhZUMfVuyIqUem2ddO5VedhczZvVjh6dmllqJHPiO7W02M1bfztm8F2pmaal6It3ObI/d+NW5eXeLtR4s2e26vMm82dnNpl2tYQOLYEcLsyxqUEYrrL4BC3oQAoch+CAgAE21AgcSetbBBISIsA1ICYupksW5gAwqpB0SSxDDIXSLvgPq1ks0oD6fmRRqTrcoemNSMpjBD/gKD/EdvsaP+OOfs9JiRv4vO5TdsVZE9tSTK+2v/1X5lDU8+q06RuES+3hPGvpwp/AiyVtUILlLPp4/evz0sD2/MpNex+f4ifzt4wG+JYfB6At/uSxW9qBCCzL/XsfRYrXZMLFhLt+uLdwrVzUJV+Ea1Gkfc7AA96EFHbr3PXyGb/DdUMau8czYG1ONiVJzGf4I48VP12q2oQ==</latexit> <latexit sha1_base64="sSmQCzcG9qsFIfcD04TNKxMncXo=">AAAC6nichVFNa9RQFL2NX3X86FQ3gpuHQ2WKOtwMQqULKXZjF8K047QDTRuSN2/Gl758kLyZUkP+gMtuRLqq6EL8Af4AN4JrF/0J4koqCOrCm0xEtFhvSO69551zXw7XjZRMNOLBhHHi5KnTZybPVs6dv3Bxqjp9aTUJhzEXHR6qMO66TiKUDERHS61EN4qF47tKrLlbi/n52kjEiQyDh3onEhu+MwhkX3JHE2RXhZUMfVuyIqUem2ddO5VedhczZvVjh6dmllqJHPiO7W02M1bfztm8F2pmaal6It3ObI/d+NW5eXeLtR4s2e26vMm82dnNpl2tYQOLYEcLsyxqUEYrrL4BC3oQAoch+CAgAE21AgcSetbBBISIsA1ICYupksW5gAwqpB0SSxDDIXSLvgPq1ks0oD6fmRRqTrcoemNSMpjBD/gKD/EdvsaP+OOfs9JiRv4vO5TdsVZE9tSTK+2v/1X5lDU8+q06RuES+3hPGvpwp/AiyVtUILlLPp4/evz0sD2/MpNex+f4ifzt4wG+JYfB6At/uSxW9qBCCzL/XsfRYrXZMLFhLt+uLdwrVzUJV+Ea1Gkfc7AA96EFHbr3PXyGb/DdUMau8czYG1ONiVJzGf4I48VP12q2oQ==</latexit>
  8. A Variant of GloVe • Residual variance 465+  '):.&1

      • 8/3 9;j! #$" ,<  • [Jameel and Schockaert, 2016] • Smoothed PMI!"# = 02-well-defined • !"# ,< (0=*  • !"# = 0%7,< 
  9. Global Relation Vectors • !, # Relation Vector $%& 

    • - !# - !-    • (i, j, k) " PMI" fox jump over the lazy dog SI1(i, j, k) = log( P(i, j)P(i, k)P(j, k) P(i)P(j)P(k)P(i, j, k) ) <latexit sha1_base64="7Al4njU+SPABk7vxC9kk7AP5JTg=">AAACy3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQu4BXvGGWpk6ihk6ShkayrYKsTk5KcraMSkFSUmVwcAJbI0QWQ2kMwCkrUgMRAbiLPBMmBRzXgBZQM9AzBQwGQYQhnKDFAQkC+wnSGGIYUhnyGZoZQhlyGVIY+hBMjOYUhkKAbCaAZDBgOGAqBYLEM1UKwIyMoEy6cy1DJwAfWWAlWlAlUkAkWzgWQ6kBcNFc0D8kFmFoN1JwNtyQHiIqBOBQZVg6sGKw0+G5wwWG3w0uAPTrOqwWaA3FIJpJMgelML4vm7JIK/E9SVC6RLGDIQuvDoSAKqxu+nEoY0BguwXzKBfisAi4B8mQwxv6xq+udgqyDVajWDRQavgf5baHDT4DDQh3llX5KXBqYGzWbgAkaQIXp0YDLCjPQMDfQMA02UHZygUcXBIM2gxKABjA9zBgcGD4YAhlCgvWsYLjHcZrjD5MtUzFTFVANRysQI1SPMgAKYWgG2WKZb</latexit> <latexit sha1_base64="7Al4njU+SPABk7vxC9kk7AP5JTg=">AAACy3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQu4BXvGGWpk6ihk6ShkayrYKsTk5KcraMSkFSUmVwcAJbI0QWQ2kMwCkrUgMRAbiLPBMmBRzXgBZQM9AzBQwGQYQhnKDFAQkC+wnSGGIYUhnyGZoZQhlyGVIY+hBMjOYUhkKAbCaAZDBgOGAqBYLEM1UKwIyMoEy6cy1DJwAfWWAlWlAlUkAkWzgWQ6kBcNFc0D8kFmFoN1JwNtyQHiIqBOBQZVg6sGKw0+G5wwWG3w0uAPTrOqwWaA3FIJpJMgelML4vm7JIK/E9SVC6RLGDIQuvDoSAKqxu+nEoY0BguwXzKBfisAi4B8mQwxv6xq+udgqyDVajWDRQavgf5baHDT4DDQh3llX5KXBqYGzWbgAkaQIXp0YDLCjPQMDfQMA02UHZygUcXBIM2gxKABjA9zBgcGD4YAhlCgvWsYLjHcZrjD5MtUzFTFVANRysQI1SPMgAKYWgG2WKZb</latexit> <latexit sha1_base64="7Al4njU+SPABk7vxC9kk7AP5JTg=">AAACy3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQu4BXvGGWpk6ihk6ShkayrYKsTk5KcraMSkFSUmVwcAJbI0QWQ2kMwCkrUgMRAbiLPBMmBRzXgBZQM9AzBQwGQYQhnKDFAQkC+wnSGGIYUhnyGZoZQhlyGVIY+hBMjOYUhkKAbCaAZDBgOGAqBYLEM1UKwIyMoEy6cy1DJwAfWWAlWlAlUkAkWzgWQ6kBcNFc0D8kFmFoN1JwNtyQHiIqBOBQZVg6sGKw0+G5wwWG3w0uAPTrOqwWaA3FIJpJMgelML4vm7JIK/E9SVC6RLGDIQuvDoSAKqxu+nEoY0BguwXzKBfisAi4B8mQwxv6xq+udgqyDVajWDRQavgf5baHDT4DDQh3llX5KXBqYGzWbgAkaQIXp0YDLCjPQMDfQMA02UHZygUcXBIM2gxKABjA9zBgcGD4YAhlCgvWsYLjHcZrjD5MtUzFTFVANRysQI1SPMgAKYWgG2WKZb</latexit> <latexit sha1_base64="dO1RVWgbv0QBXFnWGIM/DJuxBX8=">AAACdXichVG9TsJQGD1UVEQU3ExciARjHMhXHTROJi6O/MhPgoS05YINpW3aQoLEF2B1cHDSxMH4AD6Aiy/gwCMYR0xcHPwoJEaJ+jXtPffc73y3J0e1Dd31iAYBaSY4OzcfWggvRsJLy9FYpOBabUcTec0yLKekKq4wdFPkPd0zRMl2hNJSDVFUm4ej82JHOK5umcde1xaVltIw9bquKR5T6WosQSnyKz4N5AlIYFJW7AEnqMGChjZaEDDhMTagwOWnDBkEm7kKesw5jHT/XOAcYda2uUtwh8Jsk78N3pUnrMn70UzXV2t8i8Gvw8o4kvRMdzSkJ7qnF/r4dVbPnzH6ly6v6lgr7Gq0v5p7/1fV4tXD6ZfqD4XK3X978lDHnu9FZ2+2z4xcauP5nbPLYW4/m+xt0A29sr9rGtAjOzQ7b9ptRmSvEOZ85J9pTIPCdkqmlJwhhLCGdWxyDLs4wBHSyPN1NfRxIQWlLUke5ygFJoGu4FtJO59jaI2O</latexit> <latexit sha1_base64="E4InN5LL3f7ScZLfQ8Hrb0Y10wI=">AAACwHichVFNSxtRFD0ZbRtTW6Mrwc2gpCQg4U43SkEQBNGFkI9GA8aGmelL+jKTmWFmEojRP+AfcOFKwUWpe3+AG9eFLkL9A9KlhW5ceGcSKK3UvuHdd96599w3h2t4tgxCokFCGRt/9vxFciL1cvLV66n09OR24HZ8U1RM13b9qqEHwpaOqIQytEXV84XeNmyxY1hrUX6nK/xAus77sOeJvbbedGRDmnrIVD29Xt78oGXlotpaVK2cuqLWbLepZmsNXzf7BU60clG0OLY4HkZchHlbcSZmc/X0AuUpXupjoI3AAkar4KYvUcNHuDDRQRsCDkLGNnQE/O1CA8Fjbg995nxGMs4LHCLF2g5XCa7QmbU4Nvm2O2Idvkc9g1ht8is2b5+VKjL0jT7THV3TF7ql+3/26sc9on/p8WkMtcKrTx3Nln/9V9XmM8Sn36onFAZXP+0pRAPLsRfJ3ryYiVyaw/7d/eO78rtSpv+GzugH+zulAV2xQ6f70zwvitIJUjwg7e9xPAbbb/Ma5bUiIYk5zCPLY1jCKjZQQIWfu8BXfMeNsqUEyv5wlEpiNNMZ/LGUgweNd6Uk</latexit> <latexit sha1_base64="E4InN5LL3f7ScZLfQ8Hrb0Y10wI=">AAACwHichVFNSxtRFD0ZbRtTW6Mrwc2gpCQg4U43SkEQBNGFkI9GA8aGmelL+jKTmWFmEojRP+AfcOFKwUWpe3+AG9eFLkL9A9KlhW5ceGcSKK3UvuHdd96599w3h2t4tgxCokFCGRt/9vxFciL1cvLV66n09OR24HZ8U1RM13b9qqEHwpaOqIQytEXV84XeNmyxY1hrUX6nK/xAus77sOeJvbbedGRDmnrIVD29Xt78oGXlotpaVK2cuqLWbLepZmsNXzf7BU60clG0OLY4HkZchHlbcSZmc/X0AuUpXupjoI3AAkar4KYvUcNHuDDRQRsCDkLGNnQE/O1CA8Fjbg995nxGMs4LHCLF2g5XCa7QmbU4Nvm2O2Idvkc9g1ht8is2b5+VKjL0jT7THV3TF7ql+3/26sc9on/p8WkMtcKrTx3Nln/9V9XmM8Sn36onFAZXP+0pRAPLsRfJ3ryYiVyaw/7d/eO78rtSpv+GzugH+zulAV2xQ6f70zwvitIJUjwg7e9xPAbbb/Ma5bUiIYk5zCPLY1jCKjZQQIWfu8BXfMeNsqUEyv5wlEpiNNMZ/LGUgweNd6Uk</latexit> <latexit sha1_base64="+6ipzflaF06Un90/cKhew0DnaCU=">AAACy3ichVHBStxQFD1GrTq1ddSN4CZ0sMyADDduFEEQBdGFMDqOCo4dkvTN+CaZJCSZAR1duvEHXLhS6KLo3g/opuuCC2l/oLhUcOOiN5lAaaX2hXffeefec18O1/BsGYREt11Kd0/vq77+gdTrwTdvh9LDI5uB2/RNUTJd2/W3DT0QtnREKZShLbY9X+gNwxZbhrUY5bdawg+k62yE+57Ybeg1R1alqYdMVdJLxZUPWlZOqvVJ1cqpc2rZdmtqtlz1dbNd4EQ9F0WLY53jUcRFmLcVZ2I2V0lnKE/xUp8DLQEZJKvgpq9Rxke4MNFEAwIOQsY2dAT87UADwWNuF23mfEYyzgscIcXaJlcJrtCZtTjW+LaTsA7fo55BrDb5FZu3z0oVE3RDn+mevtIl/aSnf/Zqxz2if9nn0+hohVcZOhkrPv5X1eAzxN5v1QsKg6tf9hSiipnYi2RvXsxELs1O/9bB6X1xdn2i/Z4u6I79ndMtfWGHTuvB/LQm1s+Q4gFpf4/jOdicymuU19YoM7+QjKof43iHLM9jGvNYRgElfvcK3/AdP5RVJVAOlMNOqdKVaEbxx1KOfwG1GKZX</latexit> <latexit sha1_base64="7Al4njU+SPABk7vxC9kk7AP5JTg=">AAACy3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQu4BXvGGWpk6ihk6ShkayrYKsTk5KcraMSkFSUmVwcAJbI0QWQ2kMwCkrUgMRAbiLPBMmBRzXgBZQM9AzBQwGQYQhnKDFAQkC+wnSGGIYUhnyGZoZQhlyGVIY+hBMjOYUhkKAbCaAZDBgOGAqBYLEM1UKwIyMoEy6cy1DJwAfWWAlWlAlUkAkWzgWQ6kBcNFc0D8kFmFoN1JwNtyQHiIqBOBQZVg6sGKw0+G5wwWG3w0uAPTrOqwWaA3FIJpJMgelML4vm7JIK/E9SVC6RLGDIQuvDoSAKqxu+nEoY0BguwXzKBfisAi4B8mQwxv6xq+udgqyDVajWDRQavgf5baHDT4DDQh3llX5KXBqYGzWbgAkaQIXp0YDLCjPQMDfQMA02UHZygUcXBIM2gxKABjA9zBgcGD4YAhlCgvWsYLjHcZrjD5MtUzFTFVANRysQI1SPMgAKYWgG2WKZb</latexit> <latexit sha1_base64="7Al4njU+SPABk7vxC9kk7AP5JTg=">AAACy3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQu4BXvGGWpk6ihk6ShkayrYKsTk5KcraMSkFSUmVwcAJbI0QWQ2kMwCkrUgMRAbiLPBMmBRzXgBZQM9AzBQwGQYQhnKDFAQkC+wnSGGIYUhnyGZoZQhlyGVIY+hBMjOYUhkKAbCaAZDBgOGAqBYLEM1UKwIyMoEy6cy1DJwAfWWAlWlAlUkAkWzgWQ6kBcNFc0D8kFmFoN1JwNtyQHiIqBOBQZVg6sGKw0+G5wwWG3w0uAPTrOqwWaA3FIJpJMgelML4vm7JIK/E9SVC6RLGDIQuvDoSAKqxu+nEoY0BguwXzKBfisAi4B8mQwxv6xq+udgqyDVajWDRQavgf5baHDT4DDQh3llX5KXBqYGzWbgAkaQIXp0YDLCjPQMDfQMA02UHZygUcXBIM2gxKABjA9zBgcGD4YAhlCgvWsYLjHcZrjD5MtUzFTFVANRysQI1SPMgAKYWgG2WKZb</latexit> <latexit sha1_base64="7Al4njU+SPABk7vxC9kk7AP5JTg=">AAACy3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQu4BXvGGWpk6ihk6ShkayrYKsTk5KcraMSkFSUmVwcAJbI0QWQ2kMwCkrUgMRAbiLPBMmBRzXgBZQM9AzBQwGQYQhnKDFAQkC+wnSGGIYUhnyGZoZQhlyGVIY+hBMjOYUhkKAbCaAZDBgOGAqBYLEM1UKwIyMoEy6cy1DJwAfWWAlWlAlUkAkWzgWQ6kBcNFc0D8kFmFoN1JwNtyQHiIqBOBQZVg6sGKw0+G5wwWG3w0uAPTrOqwWaA3FIJpJMgelML4vm7JIK/E9SVC6RLGDIQuvDoSAKqxu+nEoY0BguwXzKBfisAi4B8mQwxv6xq+udgqyDVajWDRQavgf5baHDT4DDQh3llX5KXBqYGzWbgAkaQIXp0YDLCjPQMDfQMA02UHZygUcXBIM2gxKABjA9zBgcGD4YAhlCgvWsYLjHcZrjD5MtUzFTFVANRysQI1SPMgAKYWgG2WKZb</latexit> <latexit sha1_base64="7Al4njU+SPABk7vxC9kk7AP5JTg=">AAACy3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQu4BXvGGWpk6ihk6ShkayrYKsTk5KcraMSkFSUmVwcAJbI0QWQ2kMwCkrUgMRAbiLPBMmBRzXgBZQM9AzBQwGQYQhnKDFAQkC+wnSGGIYUhnyGZoZQhlyGVIY+hBMjOYUhkKAbCaAZDBgOGAqBYLEM1UKwIyMoEy6cy1DJwAfWWAlWlAlUkAkWzgWQ6kBcNFc0D8kFmFoN1JwNtyQHiIqBOBQZVg6sGKw0+G5wwWG3w0uAPTrOqwWaA3FIJpJMgelML4vm7JIK/E9SVC6RLGDIQuvDoSAKqxu+nEoY0BguwXzKBfisAi4B8mQwxv6xq+udgqyDVajWDRQavgf5baHDT4DDQh3llX5KXBqYGzWbgAkaQIXp0YDLCjPQMDfQMA02UHZygUcXBIM2gxKABjA9zBgcGD4YAhlCgvWsYLjHcZrjD5MtUzFTFVANRysQI1SPMgAKYWgG2WKZb</latexit> <latexit sha1_base64="7Al4njU+SPABk7vxC9kk7AP5JTg=">AAACy3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQu4BXvGGWpk6ihk6ShkayrYKsTk5KcraMSkFSUmVwcAJbI0QWQ2kMwCkrUgMRAbiLPBMmBRzXgBZQM9AzBQwGQYQhnKDFAQkC+wnSGGIYUhnyGZoZQhlyGVIY+hBMjOYUhkKAbCaAZDBgOGAqBYLEM1UKwIyMoEy6cy1DJwAfWWAlWlAlUkAkWzgWQ6kBcNFc0D8kFmFoN1JwNtyQHiIqBOBQZVg6sGKw0+G5wwWG3w0uAPTrOqwWaA3FIJpJMgelML4vm7JIK/E9SVC6RLGDIQuvDoSAKqxu+nEoY0BguwXzKBfisAi4B8mQwxv6xq+udgqyDVajWDRQavgf5baHDT4DDQh3llX5KXBqYGzWbgAkaQIXp0YDLCjPQMDfQMA02UHZygUcXBIM2gxKABjA9zBgcGD4YAhlCgvWsYLjHcZrjD5MtUzFTFVANRysQI1SPMgAKYWgG2WKZb</latexit> <latexit sha1_base64="dO1RVWgbv0QBXFnWGIM/DJuxBX8=">AAACdXichVG9TsJQGD1UVEQU3ExciARjHMhXHTROJi6O/MhPgoS05YINpW3aQoLEF2B1cHDSxMH4AD6Aiy/gwCMYR0xcHPwoJEaJ+jXtPffc73y3J0e1Dd31iAYBaSY4OzcfWggvRsJLy9FYpOBabUcTec0yLKekKq4wdFPkPd0zRMl2hNJSDVFUm4ej82JHOK5umcde1xaVltIw9bquKR5T6WosQSnyKz4N5AlIYFJW7AEnqMGChjZaEDDhMTagwOWnDBkEm7kKesw5jHT/XOAcYda2uUtwh8Jsk78N3pUnrMn70UzXV2t8i8Gvw8o4kvRMdzSkJ7qnF/r4dVbPnzH6ly6v6lgr7Gq0v5p7/1fV4tXD6ZfqD4XK3X978lDHnu9FZ2+2z4xcauP5nbPLYW4/m+xt0A29sr9rGtAjOzQ7b9ptRmSvEOZ85J9pTIPCdkqmlJwhhLCGdWxyDLs4wBHSyPN1NfRxIQWlLUke5ygFJoGu4FtJO59jaI2O</latexit> <latexit sha1_base64="E4InN5LL3f7ScZLfQ8Hrb0Y10wI=">AAACwHichVFNSxtRFD0ZbRtTW6Mrwc2gpCQg4U43SkEQBNGFkI9GA8aGmelL+jKTmWFmEojRP+AfcOFKwUWpe3+AG9eFLkL9A9KlhW5ceGcSKK3UvuHdd96599w3h2t4tgxCokFCGRt/9vxFciL1cvLV66n09OR24HZ8U1RM13b9qqEHwpaOqIQytEXV84XeNmyxY1hrUX6nK/xAus77sOeJvbbedGRDmnrIVD29Xt78oGXlotpaVK2cuqLWbLepZmsNXzf7BU60clG0OLY4HkZchHlbcSZmc/X0AuUpXupjoI3AAkar4KYvUcNHuDDRQRsCDkLGNnQE/O1CA8Fjbg995nxGMs4LHCLF2g5XCa7QmbU4Nvm2O2Idvkc9g1ht8is2b5+VKjL0jT7THV3TF7ql+3/26sc9on/p8WkMtcKrTx3Nln/9V9XmM8Sn36onFAZXP+0pRAPLsRfJ3ryYiVyaw/7d/eO78rtSpv+GzugH+zulAV2xQ6f70zwvitIJUjwg7e9xPAbbb/Ma5bUiIYk5zCPLY1jCKjZQQIWfu8BXfMeNsqUEyv5wlEpiNNMZ/LGUgweNd6Uk</latexit> <latexit sha1_base64="E4InN5LL3f7ScZLfQ8Hrb0Y10wI=">AAACwHichVFNSxtRFD0ZbRtTW6Mrwc2gpCQg4U43SkEQBNGFkI9GA8aGmelL+jKTmWFmEojRP+AfcOFKwUWpe3+AG9eFLkL9A9KlhW5ceGcSKK3UvuHdd96599w3h2t4tgxCokFCGRt/9vxFciL1cvLV66n09OR24HZ8U1RM13b9qqEHwpaOqIQytEXV84XeNmyxY1hrUX6nK/xAus77sOeJvbbedGRDmnrIVD29Xt78oGXlotpaVK2cuqLWbLepZmsNXzf7BU60clG0OLY4HkZchHlbcSZmc/X0AuUpXupjoI3AAkar4KYvUcNHuDDRQRsCDkLGNnQE/O1CA8Fjbg995nxGMs4LHCLF2g5XCa7QmbU4Nvm2O2Idvkc9g1ht8is2b5+VKjL0jT7THV3TF7ql+3/26sc9on/p8WkMtcKrTx3Nln/9V9XmM8Sn36onFAZXP+0pRAPLsRfJ3ryYiVyaw/7d/eO78rtSpv+GzugH+zulAV2xQ6f70zwvitIJUjwg7e9xPAbbb/Ma5bUiIYk5zCPLY1jCKjZQQIWfu8BXfMeNsqUEyv5wlEpiNNMZ/LGUgweNd6Uk</latexit> <latexit sha1_base64="+6ipzflaF06Un90/cKhew0DnaCU=">AAACy3ichVHBStxQFD1GrTq1ddSN4CZ0sMyADDduFEEQBdGFMDqOCo4dkvTN+CaZJCSZAR1duvEHXLhS6KLo3g/opuuCC2l/oLhUcOOiN5lAaaX2hXffeefec18O1/BsGYREt11Kd0/vq77+gdTrwTdvh9LDI5uB2/RNUTJd2/W3DT0QtnREKZShLbY9X+gNwxZbhrUY5bdawg+k62yE+57Ybeg1R1alqYdMVdJLxZUPWlZOqvVJ1cqpc2rZdmtqtlz1dbNd4EQ9F0WLY53jUcRFmLcVZ2I2V0lnKE/xUp8DLQEZJKvgpq9Rxke4MNFEAwIOQsY2dAT87UADwWNuF23mfEYyzgscIcXaJlcJrtCZtTjW+LaTsA7fo55BrDb5FZu3z0oVE3RDn+mevtIl/aSnf/Zqxz2if9nn0+hohVcZOhkrPv5X1eAzxN5v1QsKg6tf9hSiipnYi2RvXsxELs1O/9bB6X1xdn2i/Z4u6I79ndMtfWGHTuvB/LQm1s+Q4gFpf4/jOdicymuU19YoM7+QjKof43iHLM9jGvNYRgElfvcK3/AdP5RVJVAOlMNOqdKVaEbxx1KOfwG1GKZX</latexit> <latexit sha1_base64="7Al4njU+SPABk7vxC9kk7AP5JTg=">AAACy3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQu4BXvGGWpk6ihk6ShkayrYKsTk5KcraMSkFSUmVwcAJbI0QWQ2kMwCkrUgMRAbiLPBMmBRzXgBZQM9AzBQwGQYQhnKDFAQkC+wnSGGIYUhnyGZoZQhlyGVIY+hBMjOYUhkKAbCaAZDBgOGAqBYLEM1UKwIyMoEy6cy1DJwAfWWAlWlAlUkAkWzgWQ6kBcNFc0D8kFmFoN1JwNtyQHiIqBOBQZVg6sGKw0+G5wwWG3w0uAPTrOqwWaA3FIJpJMgelML4vm7JIK/E9SVC6RLGDIQuvDoSAKqxu+nEoY0BguwXzKBfisAi4B8mQwxv6xq+udgqyDVajWDRQavgf5baHDT4DDQh3llX5KXBqYGzWbgAkaQIXp0YDLCjPQMDfQMA02UHZygUcXBIM2gxKABjA9zBgcGD4YAhlCgvWsYLjHcZrjD5MtUzFTFVANRysQI1SPMgAKYWgG2WKZb</latexit> <latexit sha1_base64="7Al4njU+SPABk7vxC9kk7AP5JTg=">AAACy3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQu4BXvGGWpk6ihk6ShkayrYKsTk5KcraMSkFSUmVwcAJbI0QWQ2kMwCkrUgMRAbiLPBMmBRzXgBZQM9AzBQwGQYQhnKDFAQkC+wnSGGIYUhnyGZoZQhlyGVIY+hBMjOYUhkKAbCaAZDBgOGAqBYLEM1UKwIyMoEy6cy1DJwAfWWAlWlAlUkAkWzgWQ6kBcNFc0D8kFmFoN1JwNtyQHiIqBOBQZVg6sGKw0+G5wwWG3w0uAPTrOqwWaA3FIJpJMgelML4vm7JIK/E9SVC6RLGDIQuvDoSAKqxu+nEoY0BguwXzKBfisAi4B8mQwxv6xq+udgqyDVajWDRQavgf5baHDT4DDQh3llX5KXBqYGzWbgAkaQIXp0YDLCjPQMDfQMA02UHZygUcXBIM2gxKABjA9zBgcGD4YAhlCgvWsYLjHcZrjD5MtUzFTFVANRysQI1SPMgAKYWgG2WKZb</latexit> <latexit sha1_base64="7Al4njU+SPABk7vxC9kk7AP5JTg=">AAACy3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQu4BXvGGWpk6ihk6ShkayrYKsTk5KcraMSkFSUmVwcAJbI0QWQ2kMwCkrUgMRAbiLPBMmBRzXgBZQM9AzBQwGQYQhnKDFAQkC+wnSGGIYUhnyGZoZQhlyGVIY+hBMjOYUhkKAbCaAZDBgOGAqBYLEM1UKwIyMoEy6cy1DJwAfWWAlWlAlUkAkWzgWQ6kBcNFc0D8kFmFoN1JwNtyQHiIqBOBQZVg6sGKw0+G5wwWG3w0uAPTrOqwWaA3FIJpJMgelML4vm7JIK/E9SVC6RLGDIQuvDoSAKqxu+nEoY0BguwXzKBfisAi4B8mQwxv6xq+udgqyDVajWDRQavgf5baHDT4DDQh3llX5KXBqYGzWbgAkaQIXp0YDLCjPQMDfQMA02UHZygUcXBIM2gxKABjA9zBgcGD4YAhlCgvWsYLjHcZrjD5MtUzFTFVANRysQI1SPMgAKYWgG2WKZb</latexit> <latexit sha1_base64="7Al4njU+SPABk7vxC9kk7AP5JTg=">AAACy3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQu4BXvGGWpk6ihk6ShkayrYKsTk5KcraMSkFSUmVwcAJbI0QWQ2kMwCkrUgMRAbiLPBMmBRzXgBZQM9AzBQwGQYQhnKDFAQkC+wnSGGIYUhnyGZoZQhlyGVIY+hBMjOYUhkKAbCaAZDBgOGAqBYLEM1UKwIyMoEy6cy1DJwAfWWAlWlAlUkAkWzgWQ6kBcNFc0D8kFmFoN1JwNtyQHiIqBOBQZVg6sGKw0+G5wwWG3w0uAPTrOqwWaA3FIJpJMgelML4vm7JIK/E9SVC6RLGDIQuvDoSAKqxu+nEoY0BguwXzKBfisAi4B8mQwxv6xq+udgqyDVajWDRQavgf5baHDT4DDQh3llX5KXBqYGzWbgAkaQIXp0YDLCjPQMDfQMA02UHZygUcXBIM2gxKABjA9zBgcGD4YAhlCgvWsYLjHcZrjD5MtUzFTFVANRysQI1SPMgAKYWgG2WKZb</latexit> <latexit sha1_base64="7Al4njU+SPABk7vxC9kk7AP5JTg=">AAACy3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQu4BXvGGWpk6ihk6ShkayrYKsTk5KcraMSkFSUmVwcAJbI0QWQ2kMwCkrUgMRAbiLPBMmBRzXgBZQM9AzBQwGQYQhnKDFAQkC+wnSGGIYUhnyGZoZQhlyGVIY+hBMjOYUhkKAbCaAZDBgOGAqBYLEM1UKwIyMoEy6cy1DJwAfWWAlWlAlUkAkWzgWQ6kBcNFc0D8kFmFoN1JwNtyQHiIqBOBQZVg6sGKw0+G5wwWG3w0uAPTrOqwWaA3FIJpJMgelML4vm7JIK/E9SVC6RLGDIQuvDoSAKqxu+nEoY0BguwXzKBfisAi4B8mQwxv6xq+udgqyDVajWDRQavgf5baHDT4DDQh3llX5KXBqYGzWbgAkaQIXp0YDLCjPQMDfQMA02UHZygUcXBIM2gxKABjA9zBgcGD4YAhlCgvWsYLjHcZrjD5MtUzFTFVANRysQI1SPMgAKYWgG2WKZb</latexit> <latexit sha1_base64="7Al4njU+SPABk7vxC9kk7AP5JTg=">AAACy3icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQu4BXvGGWpk6ihk6ShkayrYKsTk5KcraMSkFSUmVwcAJbI0QWQ2kMwCkrUgMRAbiLPBMmBRzXgBZQM9AzBQwGQYQhnKDFAQkC+wnSGGIYUhnyGZoZQhlyGVIY+hBMjOYUhkKAbCaAZDBgOGAqBYLEM1UKwIyMoEy6cy1DJwAfWWAlWlAlUkAkWzgWQ6kBcNFc0D8kFmFoN1JwNtyQHiIqBOBQZVg6sGKw0+G5wwWG3w0uAPTrOqwWaA3FIJpJMgelML4vm7JIK/E9SVC6RLGDIQuvDoSAKqxu+nEoY0BguwXzKBfisAi4B8mQwxv6xq+udgqyDVajWDRQavgf5baHDT4DDQh3llX5KXBqYGzWbgAkaQIXp0YDLCjPQMDfQMA02UHZygUcXBIM2gxKABjA9zBgcGD4YAhlCgvWsYLjHcZrjD5MtUzFTFVANRysQI1SPMgAKYWgG2WKZb</latexit> SI2(i, j, k) = log( P(i, j, k) P(i)P(j)P(k) ) <latexit sha1_base64="l+q9r5nQhmo0Qbr6WmkITtIMIjU=">AAACuXichVHBSuNQFD1GHbWjY8fZCG6CRWmhlBsRlBFBdKO7aq0KVksSX+tr0yQkacEp/oA/MDCuFFyIH+BWcDPj3oWfMMzSgdnMwps0ICrqC++9c8+9574cruFa0g+I7rqU7p7eD339A4mPg0OfhpOfRzZ8p+mZomg6luNtGbovLGmLYiADS2y5ntAbhiU2jfpSmN9sCc+Xjr0eHLhip6FXbVmRph4wVU5OF1Z2p9Iyq9ayaj2jzqsly6mq6VLF0812nhO1bD1zGKJMPl3jzVGmnExRjqKlvgRaDFKIV95JXqKEPTgw0UQDAjYCxhZ0+PxtQwPBZW4HbeY8RjLKCxwiwdomVwmu0Jmt81nlaDtmbY7Dnn6kNvkVi7fHShUTdEvndE8/6YJ+0/9Xe7WjHuG/HPBtdLTCLQ8fjRb+vatq8B1g/1H1hsLg6rc9BahgNvIi2ZsbMaFLs9O/9e37feHr2kR7kk7pD/s7oTu6Zod26695tirWjpHgAWnPx/ESbEzlNMppq9OphcV4VP0YwzjSPI8ZLGAZeRT53R+4wi/cKHOKruwrtU6p0hVrvuDJUvwHdDqgxQ==</latexit> <latexit sha1_base64="l+q9r5nQhmo0Qbr6WmkITtIMIjU=">AAACuXichVHBSuNQFD1GHbWjY8fZCG6CRWmhlBsRlBFBdKO7aq0KVksSX+tr0yQkacEp/oA/MDCuFFyIH+BWcDPj3oWfMMzSgdnMwps0ICrqC++9c8+9574cruFa0g+I7rqU7p7eD339A4mPg0OfhpOfRzZ8p+mZomg6luNtGbovLGmLYiADS2y5ntAbhiU2jfpSmN9sCc+Xjr0eHLhip6FXbVmRph4wVU5OF1Z2p9Iyq9ayaj2jzqsly6mq6VLF0812nhO1bD1zGKJMPl3jzVGmnExRjqKlvgRaDFKIV95JXqKEPTgw0UQDAjYCxhZ0+PxtQwPBZW4HbeY8RjLKCxwiwdomVwmu0Jmt81nlaDtmbY7Dnn6kNvkVi7fHShUTdEvndE8/6YJ+0/9Xe7WjHuG/HPBtdLTCLQ8fjRb+vatq8B1g/1H1hsLg6rc9BahgNvIi2ZsbMaFLs9O/9e37feHr2kR7kk7pD/s7oTu6Zod26695tirWjpHgAWnPx/ESbEzlNMppq9OphcV4VP0YwzjSPI8ZLGAZeRT53R+4wi/cKHOKruwrtU6p0hVrvuDJUvwHdDqgxQ==</latexit> <latexit sha1_base64="l+q9r5nQhmo0Qbr6WmkITtIMIjU=">AAACuXichVHBSuNQFD1GHbWjY8fZCG6CRWmhlBsRlBFBdKO7aq0KVksSX+tr0yQkacEp/oA/MDCuFFyIH+BWcDPj3oWfMMzSgdnMwps0ICrqC++9c8+9574cruFa0g+I7rqU7p7eD339A4mPg0OfhpOfRzZ8p+mZomg6luNtGbovLGmLYiADS2y5ntAbhiU2jfpSmN9sCc+Xjr0eHLhip6FXbVmRph4wVU5OF1Z2p9Iyq9ayaj2jzqsly6mq6VLF0812nhO1bD1zGKJMPl3jzVGmnExRjqKlvgRaDFKIV95JXqKEPTgw0UQDAjYCxhZ0+PxtQwPBZW4HbeY8RjLKCxwiwdomVwmu0Jmt81nlaDtmbY7Dnn6kNvkVi7fHShUTdEvndE8/6YJ+0/9Xe7WjHuG/HPBtdLTCLQ8fjRb+vatq8B1g/1H1hsLg6rc9BahgNvIi2ZsbMaFLs9O/9e37feHr2kR7kk7pD/s7oTu6Zod26695tirWjpHgAWnPx/ESbEzlNMppq9OphcV4VP0YwzjSPI8ZLGAZeRT53R+4wi/cKHOKruwrtU6p0hVrvuDJUvwHdDqgxQ==</latexit> <latexit sha1_base64="l+q9r5nQhmo0Qbr6WmkITtIMIjU=">AAACuXichVHBSuNQFD1GHbWjY8fZCG6CRWmhlBsRlBFBdKO7aq0KVksSX+tr0yQkacEp/oA/MDCuFFyIH+BWcDPj3oWfMMzSgdnMwps0ICrqC++9c8+9574cruFa0g+I7rqU7p7eD339A4mPg0OfhpOfRzZ8p+mZomg6luNtGbovLGmLYiADS2y5ntAbhiU2jfpSmN9sCc+Xjr0eHLhip6FXbVmRph4wVU5OF1Z2p9Iyq9ayaj2jzqsly6mq6VLF0812nhO1bD1zGKJMPl3jzVGmnExRjqKlvgRaDFKIV95JXqKEPTgw0UQDAjYCxhZ0+PxtQwPBZW4HbeY8RjLKCxwiwdomVwmu0Jmt81nlaDtmbY7Dnn6kNvkVi7fHShUTdEvndE8/6YJ+0/9Xe7WjHuG/HPBtdLTCLQ8fjRb+vatq8B1g/1H1hsLg6rc9BahgNvIi2ZsbMaFLs9O/9e37feHr2kR7kk7pD/s7oTu6Zod26695tirWjpHgAWnPx/ESbEzlNMppq9OphcV4VP0YwzjSPI8ZLGAZeRT53R+4wi/cKHOKruwrtU6p0hVrvuDJUvwHdDqgxQ==</latexit> SI3(i, j, k) = log( P(i, j, k) P(i, k)P(j) ) <latexit sha1_base64="CRFW1YfM7pe7Cv4xPVG5YtdLyI8=">AAACt3ichVHBShtRFD1Obaup1qgboZvBoGQghDtasBYEsZt2F41RwWiYGV/iy0xmpjOTgA35gf6ABVcKLoof4M5NF/UHXPgJ0qWCGxfeTAZKK+od3rzzzr3nvne4pu/IMCK67FNe9L989XpgMPVmaPjtSHp0bC30moElSpbneMGGaYTCka4oRTJyxIYfCKNhOmLdtD918+stEYTSc1ejPV9sNYyaK6vSMiKmKumZ4pft2azMqfWcamvqglp2vJqaLVcDw2oXOFHP2VonRrZWyNa1jlZJZyhPcagPgZ6ADJIoeOlTlLEDDxaaaEDARcTYgYGQv03oIPjMbaHNXMBIxnmBDlKsbXKV4AqDWZv/NT5tJqzL527PMFZbfIvDK2Cliim6oJ90Ted0Qld092ivdtyj+5Y93s2eVviVke8TxdtnVQ3eI+z+VT2hMLn6aU8RqvgQe5HszY+Zrkur17/1bf+6+HFlqj1NR/SH/R3SJf1ih27rxjpeFisHSPGA9P/H8RCszeR1yuvL7zOLS8moBvAOk8jyPOawiM8ooMT3/sAZfuNcmVcqSlXZ7ZUqfYlmHP+E8vUeFR2gPQ==</latexit> <latexit sha1_base64="CRFW1YfM7pe7Cv4xPVG5YtdLyI8=">AAACt3ichVHBShtRFD1Obaup1qgboZvBoGQghDtasBYEsZt2F41RwWiYGV/iy0xmpjOTgA35gf6ABVcKLoof4M5NF/UHXPgJ0qWCGxfeTAZKK+od3rzzzr3nvne4pu/IMCK67FNe9L989XpgMPVmaPjtSHp0bC30moElSpbneMGGaYTCka4oRTJyxIYfCKNhOmLdtD918+stEYTSc1ejPV9sNYyaK6vSMiKmKumZ4pft2azMqfWcamvqglp2vJqaLVcDw2oXOFHP2VonRrZWyNa1jlZJZyhPcagPgZ6ADJIoeOlTlLEDDxaaaEDARcTYgYGQv03oIPjMbaHNXMBIxnmBDlKsbXKV4AqDWZv/NT5tJqzL527PMFZbfIvDK2Cliim6oJ90Ted0Qld092ivdtyj+5Y93s2eVviVke8TxdtnVQ3eI+z+VT2hMLn6aU8RqvgQe5HszY+Zrkur17/1bf+6+HFlqj1NR/SH/R3SJf1ih27rxjpeFisHSPGA9P/H8RCszeR1yuvL7zOLS8moBvAOk8jyPOawiM8ooMT3/sAZfuNcmVcqSlXZ7ZUqfYlmHP+E8vUeFR2gPQ==</latexit> <latexit sha1_base64="CRFW1YfM7pe7Cv4xPVG5YtdLyI8=">AAACt3ichVHBShtRFD1Obaup1qgboZvBoGQghDtasBYEsZt2F41RwWiYGV/iy0xmpjOTgA35gf6ABVcKLoof4M5NF/UHXPgJ0qWCGxfeTAZKK+od3rzzzr3nvne4pu/IMCK67FNe9L989XpgMPVmaPjtSHp0bC30moElSpbneMGGaYTCka4oRTJyxIYfCKNhOmLdtD918+stEYTSc1ejPV9sNYyaK6vSMiKmKumZ4pft2azMqfWcamvqglp2vJqaLVcDw2oXOFHP2VonRrZWyNa1jlZJZyhPcagPgZ6ADJIoeOlTlLEDDxaaaEDARcTYgYGQv03oIPjMbaHNXMBIxnmBDlKsbXKV4AqDWZv/NT5tJqzL527PMFZbfIvDK2Cliim6oJ90Ted0Qld092ivdtyj+5Y93s2eVviVke8TxdtnVQ3eI+z+VT2hMLn6aU8RqvgQe5HszY+Zrkur17/1bf+6+HFlqj1NR/SH/R3SJf1ih27rxjpeFisHSPGA9P/H8RCszeR1yuvL7zOLS8moBvAOk8jyPOawiM8ooMT3/sAZfuNcmVcqSlXZ7ZUqfYlmHP+E8vUeFR2gPQ==</latexit> <latexit sha1_base64="CRFW1YfM7pe7Cv4xPVG5YtdLyI8=">AAACt3ichVHBShtRFD1Obaup1qgboZvBoGQghDtasBYEsZt2F41RwWiYGV/iy0xmpjOTgA35gf6ABVcKLoof4M5NF/UHXPgJ0qWCGxfeTAZKK+od3rzzzr3nvne4pu/IMCK67FNe9L989XpgMPVmaPjtSHp0bC30moElSpbneMGGaYTCka4oRTJyxIYfCKNhOmLdtD918+stEYTSc1ejPV9sNYyaK6vSMiKmKumZ4pft2azMqfWcamvqglp2vJqaLVcDw2oXOFHP2VonRrZWyNa1jlZJZyhPcagPgZ6ADJIoeOlTlLEDDxaaaEDARcTYgYGQv03oIPjMbaHNXMBIxnmBDlKsbXKV4AqDWZv/NT5tJqzL527PMFZbfIvDK2Cliim6oJ90Ted0Qld092ivdtyj+5Y93s2eVviVke8TxdtnVQ3eI+z+VT2hMLn6aU8RqvgQe5HszY+Zrkur17/1bf+6+HFlqj1NR/SH/R3SJf1ih27rxjpeFisHSPGA9P/H8RCszeR1yuvL7zOLS8moBvAOk8jyPOawiM8ooMT3/sAZfuNcmVcqSlXZ7ZUqfYlmHP+E8vUeFR2gPQ==</latexit> SI4(i, j, k) = log( P(i, k|j) P(i|j)P(k|j) ) <latexit sha1_base64="3w78VI2wpCsqNoxi+SgL+TVRKjI=">AAACuXichVFNSxtRFD1O60fjV1o3hW4GgyUBCXckYFEKoW50F02jgtEwM77El0xmhplJII3+gf4BQVcKXYg/oFvBTeu+C3+CuFRw46J3JgOlinqHmXveuffcN4druJb0A6LLPuXV6/6BwaE3ieGR0bHx5Nt3q77T8kxRMh3L8dYN3ReWtEUpkIEl1l1P6E3DEmtGYyGsr7WF50vH/hp0XLHZ1Gu2rEpTD5iqJHPFpa1cWk6r9Wm1kVE/q2XLqanpctXTzW6BC43demYvRJwL6eiUqSRTlKUo1MdAi0EKcRSc5E+UsQ0HJlpoQsBGwNiCDp+fDWgguMxtosucx0hGdYE9JFjb4i7BHTqzDf7W+LQRszafw5l+pDb5Fotfj5UqpugPndAN/aJTuqL7J2d1oxnhv3Q4Gz2tcCvj398X715UNTkH2PmnekZhcPfzngJU8SnyItmbGzGhS7M3v/1t/6Y4tzLV/UjHdM3+juiSztmh3b41fyyLlUMkeEHaw3U8BqszWY2y2nIulf8Sr2oIHzCJNO9jFnksooAS33uAM/zGhTKv6MqOUu+1Kn2xZgL/heL/BcgCodg=</latexit> <latexit sha1_base64="3w78VI2wpCsqNoxi+SgL+TVRKjI=">AAACuXichVFNSxtRFD1O60fjV1o3hW4GgyUBCXckYFEKoW50F02jgtEwM77El0xmhplJII3+gf4BQVcKXYg/oFvBTeu+C3+CuFRw46J3JgOlinqHmXveuffcN4druJb0A6LLPuXV6/6BwaE3ieGR0bHx5Nt3q77T8kxRMh3L8dYN3ReWtEUpkIEl1l1P6E3DEmtGYyGsr7WF50vH/hp0XLHZ1Gu2rEpTD5iqJHPFpa1cWk6r9Wm1kVE/q2XLqanpctXTzW6BC43demYvRJwL6eiUqSRTlKUo1MdAi0EKcRSc5E+UsQ0HJlpoQsBGwNiCDp+fDWgguMxtosucx0hGdYE9JFjb4i7BHTqzDf7W+LQRszafw5l+pDb5Fotfj5UqpugPndAN/aJTuqL7J2d1oxnhv3Q4Gz2tcCvj398X715UNTkH2PmnekZhcPfzngJU8SnyItmbGzGhS7M3v/1t/6Y4tzLV/UjHdM3+juiSztmh3b41fyyLlUMkeEHaw3U8BqszWY2y2nIulf8Sr2oIHzCJNO9jFnksooAS33uAM/zGhTKv6MqOUu+1Kn2xZgL/heL/BcgCodg=</latexit> <latexit sha1_base64="3w78VI2wpCsqNoxi+SgL+TVRKjI=">AAACuXichVFNSxtRFD1O60fjV1o3hW4GgyUBCXckYFEKoW50F02jgtEwM77El0xmhplJII3+gf4BQVcKXYg/oFvBTeu+C3+CuFRw46J3JgOlinqHmXveuffcN4druJb0A6LLPuXV6/6BwaE3ieGR0bHx5Nt3q77T8kxRMh3L8dYN3ReWtEUpkIEl1l1P6E3DEmtGYyGsr7WF50vH/hp0XLHZ1Gu2rEpTD5iqJHPFpa1cWk6r9Wm1kVE/q2XLqanpctXTzW6BC43demYvRJwL6eiUqSRTlKUo1MdAi0EKcRSc5E+UsQ0HJlpoQsBGwNiCDp+fDWgguMxtosucx0hGdYE9JFjb4i7BHTqzDf7W+LQRszafw5l+pDb5Fotfj5UqpugPndAN/aJTuqL7J2d1oxnhv3Q4Gz2tcCvj398X715UNTkH2PmnekZhcPfzngJU8SnyItmbGzGhS7M3v/1t/6Y4tzLV/UjHdM3+juiSztmh3b41fyyLlUMkeEHaw3U8BqszWY2y2nIulf8Sr2oIHzCJNO9jFnksooAS33uAM/zGhTKv6MqOUu+1Kn2xZgL/heL/BcgCodg=</latexit> <latexit sha1_base64="3w78VI2wpCsqNoxi+SgL+TVRKjI=">AAACuXichVFNSxtRFD1O60fjV1o3hW4GgyUBCXckYFEKoW50F02jgtEwM77El0xmhplJII3+gf4BQVcKXYg/oFvBTeu+C3+CuFRw46J3JgOlinqHmXveuffcN4druJb0A6LLPuXV6/6BwaE3ieGR0bHx5Nt3q77T8kxRMh3L8dYN3ReWtEUpkIEl1l1P6E3DEmtGYyGsr7WF50vH/hp0XLHZ1Gu2rEpTD5iqJHPFpa1cWk6r9Wm1kVE/q2XLqanpctXTzW6BC43demYvRJwL6eiUqSRTlKUo1MdAi0EKcRSc5E+UsQ0HJlpoQsBGwNiCDp+fDWgguMxtosucx0hGdYE9JFjb4i7BHTqzDf7W+LQRszafw5l+pDb5Fotfj5UqpugPndAN/aJTuqL7J2d1oxnhv3Q4Gz2tcCvj398X715UNTkH2PmnekZhcPfzngJU8SnyItmbGzGhS7M3v/1t/6Y4tzLV/UjHdM3+juiSztmh3b41fyyLlUMkeEHaw3U8BqszWY2y2nIulf8Sr2oIHzCJNO9jFnksooAS33uAM/zGhTKv6MqOUu+1Kn2xZgL/heL/BcgCodg=</latexit> ' ∈ ℝ|+|×|-|×|+|
  10. Global Relation Vectors • ,&"* • ! "# ,$ %#

    pre-train  &'( !'% fix   • Residual variance(1 /) • . -   (in, on, etc.)  "#+0$(2 X j:Xijk>0 (rik · ˜ wj + ˜ bj SI(i, j, k))2 <latexit sha1_base64="N1KtAiosdxX8/56bMO65VQriN7o=">AAAC0HichVE9axRRFD2Z+BE3atbYCDaDS2SDyXInCEqKELRRbJJdN1nIxmFm9iW+nU9m3m6MwyC2+QMWVgoWYmXlD7DxDwRJp6VYRrCx8M7siGgw3uHNO/e8e+57h2tHnkwU0cGYNn7i5KnTE2cqk2fPnZ+qXpheS8JB7Ii2E3ph3LGtRHgyEG0llSc6USws3/bEuu3ezs/XhyJOZBjcV7uR2PSt7UBuScdSTJnVe91k4Jtpf7FjprLvZkuU6fWYsZvpXacXKr2rpNcT6U5m9vVrvzI7z+b11t26nNP7c7o7O/tgwazWqEFF6EeBUYIaylgJq+/QRQ8hHAzgQyCAYuzBQsLfBgwQIuY2kTIXM5LFuUCGCmsHXCW4wmLW5f82ZxslG3Ce90wKtcO3eLxiVuqYoX16TYf0gd7QF/rxz15p0SN/yy7v9kgrInNq71Lr+39VPu8KD3+rjlHYXH28J4Ut3Cy8SPYWFUzu0hn1Hz5+dthabM6kV+klfWV/L+iA3rPDYPjNebUqms9R4QEZf4/jKFhbaBjUMFav15ZvlaOawGVcQZ3ncQPLuIMVtPnet/iIT/isNbVH2hPt6ahUGys1F/FHaHs/Afd6rEk=</latexit> <latexit sha1_base64="N1KtAiosdxX8/56bMO65VQriN7o=">AAAC0HichVE9axRRFD2Z+BE3atbYCDaDS2SDyXInCEqKELRRbJJdN1nIxmFm9iW+nU9m3m6MwyC2+QMWVgoWYmXlD7DxDwRJp6VYRrCx8M7siGgw3uHNO/e8e+57h2tHnkwU0cGYNn7i5KnTE2cqk2fPnZ+qXpheS8JB7Ii2E3ph3LGtRHgyEG0llSc6USws3/bEuu3ezs/XhyJOZBjcV7uR2PSt7UBuScdSTJnVe91k4Jtpf7FjprLvZkuU6fWYsZvpXacXKr2rpNcT6U5m9vVrvzI7z+b11t26nNP7c7o7O/tgwazWqEFF6EeBUYIaylgJq+/QRQ8hHAzgQyCAYuzBQsLfBgwQIuY2kTIXM5LFuUCGCmsHXCW4wmLW5f82ZxslG3Ce90wKtcO3eLxiVuqYoX16TYf0gd7QF/rxz15p0SN/yy7v9kgrInNq71Lr+39VPu8KD3+rjlHYXH28J4Ut3Cy8SPYWFUzu0hn1Hz5+dthabM6kV+klfWV/L+iA3rPDYPjNebUqms9R4QEZf4/jKFhbaBjUMFav15ZvlaOawGVcQZ3ncQPLuIMVtPnet/iIT/isNbVH2hPt6ahUGys1F/FHaHs/Afd6rEk=</latexit> <latexit sha1_base64="N1KtAiosdxX8/56bMO65VQriN7o=">AAAC0HichVE9axRRFD2Z+BE3atbYCDaDS2SDyXInCEqKELRRbJJdN1nIxmFm9iW+nU9m3m6MwyC2+QMWVgoWYmXlD7DxDwRJp6VYRrCx8M7siGgw3uHNO/e8e+57h2tHnkwU0cGYNn7i5KnTE2cqk2fPnZ+qXpheS8JB7Ii2E3ph3LGtRHgyEG0llSc6USws3/bEuu3ezs/XhyJOZBjcV7uR2PSt7UBuScdSTJnVe91k4Jtpf7FjprLvZkuU6fWYsZvpXacXKr2rpNcT6U5m9vVrvzI7z+b11t26nNP7c7o7O/tgwazWqEFF6EeBUYIaylgJq+/QRQ8hHAzgQyCAYuzBQsLfBgwQIuY2kTIXM5LFuUCGCmsHXCW4wmLW5f82ZxslG3Ce90wKtcO3eLxiVuqYoX16TYf0gd7QF/rxz15p0SN/yy7v9kgrInNq71Lr+39VPu8KD3+rjlHYXH28J4Ut3Cy8SPYWFUzu0hn1Hz5+dthabM6kV+klfWV/L+iA3rPDYPjNebUqms9R4QEZf4/jKFhbaBjUMFav15ZvlaOawGVcQZ3ncQPLuIMVtPnet/iIT/isNbVH2hPt6ahUGys1F/FHaHs/Afd6rEk=</latexit> <latexit sha1_base64="N1KtAiosdxX8/56bMO65VQriN7o=">AAAC0HichVE9axRRFD2Z+BE3atbYCDaDS2SDyXInCEqKELRRbJJdN1nIxmFm9iW+nU9m3m6MwyC2+QMWVgoWYmXlD7DxDwRJp6VYRrCx8M7siGgw3uHNO/e8e+57h2tHnkwU0cGYNn7i5KnTE2cqk2fPnZ+qXpheS8JB7Ii2E3ph3LGtRHgyEG0llSc6USws3/bEuu3ezs/XhyJOZBjcV7uR2PSt7UBuScdSTJnVe91k4Jtpf7FjprLvZkuU6fWYsZvpXacXKr2rpNcT6U5m9vVrvzI7z+b11t26nNP7c7o7O/tgwazWqEFF6EeBUYIaylgJq+/QRQ8hHAzgQyCAYuzBQsLfBgwQIuY2kTIXM5LFuUCGCmsHXCW4wmLW5f82ZxslG3Ce90wKtcO3eLxiVuqYoX16TYf0gd7QF/rxz15p0SN/yy7v9kgrInNq71Lr+39VPu8KD3+rjlHYXH28J4Ut3Cy8SPYWFUzu0hn1Hz5+dthabM6kV+klfWV/L+iA3rPDYPjNebUqms9R4QEZf4/jKFhbaBjUMFav15ZvlaOawGVcQZ3ncQPLuIMVtPnet/iIT/isNbVH2hPt6ahUGys1F/FHaHs/Afd6rEk=</latexit>
  11. Global Relation Vectors • !     •

    ik ... k i ... ... ... ... ... !"# $"# %"# Rik = [rik; rki; sik; ski; tik; tki; wi; wk] <latexit sha1_base64="WfMTdaegGBF5TEAeFoeawxOO500=">AAACw3ichVFNS9xQFD2mfnX8GttNwU3oYHE13IigKC1SKbjUsaPCdAhJfI6PZJKQvJlih/kD/QNduFLqQvwB/gA34la78CcUlwpuuujNB5Qq6nvk3XPOu+e+XK4dejJWRFc92ovevv6BwZeFoeGR0bHi+Kv1OGhFjqg6gRdEm7YVC0/6oqqk8sRmGAmraXtiw3aXkvuNtohiGfif1W4o6k2r4ctt6ViKJbP4oWJ2pNvV3+u1KEULHFzZXYgzFmdMZUxlTP9qyuRw62axRGVKl/4QGDkoIV8rQfEEX7CFAA5aaELAh2LswULMuwYDhJC1OjqsRYxkei/QRYG9Lc4SnGGx6vLZYFbLVZ95UjNO3Q6/4vEXsVPHJP2iI7qhMzqm3/Tn0VqdtEbyL7sc7cwrQnPs+5u1u2ddTY4KO/9cTzhszn66J4VtzKW9SO4tTJWkSyer3/7242ZtvjLZeUcHdM397dMVnXKHfvvWOVwVlT0UeEDG/XE8BOvTZYPKxupMafFjPqpBTOAtpnges1jEMlZQ5Xd/4hwXuNQ+aa4WaSpL1Xpyz2v8t7TuX6y5qc0=</latexit> <latexit sha1_base64="WfMTdaegGBF5TEAeFoeawxOO500=">AAACw3ichVFNS9xQFD2mfnX8GttNwU3oYHE13IigKC1SKbjUsaPCdAhJfI6PZJKQvJlih/kD/QNduFLqQvwB/gA34la78CcUlwpuuujNB5Qq6nvk3XPOu+e+XK4dejJWRFc92ovevv6BwZeFoeGR0bHi+Kv1OGhFjqg6gRdEm7YVC0/6oqqk8sRmGAmraXtiw3aXkvuNtohiGfif1W4o6k2r4ctt6ViKJbP4oWJ2pNvV3+u1KEULHFzZXYgzFmdMZUxlTP9qyuRw62axRGVKl/4QGDkoIV8rQfEEX7CFAA5aaELAh2LswULMuwYDhJC1OjqsRYxkei/QRYG9Lc4SnGGx6vLZYFbLVZ95UjNO3Q6/4vEXsVPHJP2iI7qhMzqm3/Tn0VqdtEbyL7sc7cwrQnPs+5u1u2ddTY4KO/9cTzhszn66J4VtzKW9SO4tTJWkSyer3/7242ZtvjLZeUcHdM397dMVnXKHfvvWOVwVlT0UeEDG/XE8BOvTZYPKxupMafFjPqpBTOAtpnges1jEMlZQ5Xd/4hwXuNQ+aa4WaSpL1Xpyz2v8t7TuX6y5qc0=</latexit> <latexit sha1_base64="WfMTdaegGBF5TEAeFoeawxOO500=">AAACw3ichVFNS9xQFD2mfnX8GttNwU3oYHE13IigKC1SKbjUsaPCdAhJfI6PZJKQvJlih/kD/QNduFLqQvwB/gA34la78CcUlwpuuujNB5Qq6nvk3XPOu+e+XK4dejJWRFc92ovevv6BwZeFoeGR0bHi+Kv1OGhFjqg6gRdEm7YVC0/6oqqk8sRmGAmraXtiw3aXkvuNtohiGfif1W4o6k2r4ctt6ViKJbP4oWJ2pNvV3+u1KEULHFzZXYgzFmdMZUxlTP9qyuRw62axRGVKl/4QGDkoIV8rQfEEX7CFAA5aaELAh2LswULMuwYDhJC1OjqsRYxkei/QRYG9Lc4SnGGx6vLZYFbLVZ95UjNO3Q6/4vEXsVPHJP2iI7qhMzqm3/Tn0VqdtEbyL7sc7cwrQnPs+5u1u2ddTY4KO/9cTzhszn66J4VtzKW9SO4tTJWkSyer3/7242ZtvjLZeUcHdM397dMVnXKHfvvWOVwVlT0UeEDG/XE8BOvTZYPKxupMafFjPqpBTOAtpnges1jEMlZQ5Xd/4hwXuNQ+aa4WaSpL1Xpyz2v8t7TuX6y5qc0=</latexit> <latexit sha1_base64="WfMTdaegGBF5TEAeFoeawxOO500=">AAACw3ichVFNS9xQFD2mfnX8GttNwU3oYHE13IigKC1SKbjUsaPCdAhJfI6PZJKQvJlih/kD/QNduFLqQvwB/gA34la78CcUlwpuuujNB5Qq6nvk3XPOu+e+XK4dejJWRFc92ovevv6BwZeFoeGR0bHi+Kv1OGhFjqg6gRdEm7YVC0/6oqqk8sRmGAmraXtiw3aXkvuNtohiGfif1W4o6k2r4ctt6ViKJbP4oWJ2pNvV3+u1KEULHFzZXYgzFmdMZUxlTP9qyuRw62axRGVKl/4QGDkoIV8rQfEEX7CFAA5aaELAh2LswULMuwYDhJC1OjqsRYxkei/QRYG9Lc4SnGGx6vLZYFbLVZ95UjNO3Q6/4vEXsVPHJP2iI7qhMzqm3/Tn0VqdtEbyL7sc7cwrQnPs+5u1u2ddTY4KO/9cTzhszn66J4VtzKW9SO4tTJWkSyer3/7242ZtvjLZeUcHdM397dMVnXKHfvvWOVwVlT0UeEDG/XE8BOvTZYPKxupMafFjPqpBTOAtpnges1jEMlZQ5Xd/4hwXuNQ+aa4WaSpL1Xpyz2v8t7TuX6y5qc0=</latexit>
  12.  •  • Relation Induction: Google Analogy, DiffVec dataset

    • Relation Ranking: SemEval2012 Task2 • Relation Extraction (distant supervision) : NYT+Freebase •    • Diff:   • Conc:   • Avg:  "! [ravg ik ; ravg ki ; savg ik ; savg ki ; tavg ik ; tavg ki ; wi; wk] <latexit sha1_base64="ZjaWgedbNiL8kCIo/AghBAxg7LM=">AAAC3nicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQtERhfFV2dm18ZVJ5al11oDOdmZME4xskwxskwJskwJsoxCeXwmiMiOjRdQNtAzAAMFTIYhlKHMAAUB+QLbGWIYUhjyGZIZShlyGVIZ8hhKgOwchkSGYiCMZjBkMGAoAIrFMlQDxYqArEywfCpDLQMXUG8pUFUqUEUiUDQbSKYDedFQ0TwgH2RmMVh3MtCWHCAuAupUYFA1uGqw0uCzwQmD1QYvDf7gNKsabAbILZVAOgmiN7Ugnr9LIvg7QV25QLqEIQOhC4+OJKBq/H4qYUhjsAD7JRPotwKwCMiXyRDzy6qmfw62ClKtVjNYZPAa6L+FBjcNDgN9mFf2JXlpYGrQbAYuYAQZokcHJiPMSM/QQM8w0ETZwQkaVRwM0gxKDBrA+DBncGDwYAhgCAXau5/hKcM7hvdMCUxtTL1M/RClTIxQPcIMKIBpBgDHsLdZ</latexit> <latexit sha1_base64="ZjaWgedbNiL8kCIo/AghBAxg7LM=">AAAC3nicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQtERhfFV2dm18ZVJ5al11oDOdmZME4xskwxskwJskwJsoxCeXwmiMiOjRdQNtAzAAMFTIYhlKHMAAUB+QLbGWIYUhjyGZIZShlyGVIZ8hhKgOwchkSGYiCMZjBkMGAoAIrFMlQDxYqArEywfCpDLQMXUG8pUFUqUEUiUDQbSKYDedFQ0TwgH2RmMVh3MtCWHCAuAupUYFA1uGqw0uCzwQmD1QYvDf7gNKsabAbILZVAOgmiN7Ugnr9LIvg7QV25QLqEIQOhC4+OJKBq/H4qYUhjsAD7JRPotwKwCMiXyRDzy6qmfw62ClKtVjNYZPAa6L+FBjcNDgN9mFf2JXlpYGrQbAYuYAQZokcHJiPMSM/QQM8w0ETZwQkaVRwM0gxKDBrA+DBncGDwYAhgCAXau5/hKcM7hvdMCUxtTL1M/RClTIxQPcIMKIBpBgDHsLdZ</latexit> <latexit sha1_base64="ZjaWgedbNiL8kCIo/AghBAxg7LM=">AAAC3nicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQtERhfFV2dm18ZVJ5al11oDOdmZME4xskwxskwJskwJsoxCeXwmiMiOjRdQNtAzAAMFTIYhlKHMAAUB+QLbGWIYUhjyGZIZShlyGVIZ8hhKgOwchkSGYiCMZjBkMGAoAIrFMlQDxYqArEywfCpDLQMXUG8pUFUqUEUiUDQbSKYDedFQ0TwgH2RmMVh3MtCWHCAuAupUYFA1uGqw0uCzwQmD1QYvDf7gNKsabAbILZVAOgmiN7Ugnr9LIvg7QV25QLqEIQOhC4+OJKBq/H4qYUhjsAD7JRPotwKwCMiXyRDzy6qmfw62ClKtVjNYZPAa6L+FBjcNDgN9mFf2JXlpYGrQbAYuYAQZokcHJiPMSM/QQM8w0ETZwQkaVRwM0gxKDBrA+DBncGDwYAhgCAXau5/hKcM7hvdMCUxtTL1M/RClTIxQPcIMKIBpBgDHsLdZ</latexit> <latexit sha1_base64="ZjaWgedbNiL8kCIo/AghBAxg7LM=">AAAC3nicSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQtERhfFV2dm18ZVJ5al11oDOdmZME4xskwxskwJskwJsoxCeXwmiMiOjRdQNtAzAAMFTIYhlKHMAAUB+QLbGWIYUhjyGZIZShlyGVIZ8hhKgOwchkSGYiCMZjBkMGAoAIrFMlQDxYqArEywfCpDLQMXUG8pUFUqUEUiUDQbSKYDedFQ0TwgH2RmMVh3MtCWHCAuAupUYFA1uGqw0uCzwQmD1QYvDf7gNKsabAbILZVAOgmiN7Ugnr9LIvg7QV25QLqEIQOhC4+OJKBq/H4qYUhjsAD7JRPotwKwCMiXyRDzy6qmfw62ClKtVjNYZPAa6L+FBjcNDgN9mFf2JXlpYGrQbAYuYAQZokcHJiPMSM/QQM8w0ETZwQkaVRwM0gxKDBrA+DBncGDwYAhgCAXau5/hKcM7hvdMCUxtTL1M/RClTIxQPcIMKIBpBgDHsLdZ</latexit>
  13. Relation Induction • 6:/% +:"#  • 4 "#,. 508;

    • Google Analogy: 14*; • DiffVec: 36*; • "#, 7<)- • Reverse, random generating • 9'31 linear SVM 8!&($2
  14. Relation Ranking • ! # "  #  

    !  & • SemEval2012 Task2: 79( !%$   ' • E.g object-typical action • glass:break • soldier:fight • juggernaut:crush
  15. Relation Ranking • %*$7.   3 • 4/1(0 

    )68+2   • #'-)6train/dev/test5" linear SVM regression !, • &920~30
  16. Relation Extraction • Distant supervision RE: • NYT corpusFreebase. •

    NYT corpusFreebase'5330 3 %(,   / # • '5  " !1*$+'53Freebase SVM04&$+ 2-)
  17.  • GloVe B=2"D5) @ • =2,/: RE0( • *<

    • #$%9  • ;>13F 6word pair×)*+4G • Latent Relational AnalysisUniversal Schema=2+.- ! &AE 8C7?' • cf. window-context vs. dependency-context [Levy&Goldberg, 2014]