Slide 39
Slide 39 text
) Attention Context
!39
Attention ( ) ...
(mixture modeling...)
m
X
i=1
↵ifi(x)
AAACyHichVFNL8RQFD3qa3wPNhKbYULYTF6HhEgkwkasfA0SQ9PWG170K21nGM1sbCT+gIUViYWILT/Axh+w8BPEksTGwm2niSC4TfvOO/ee+87r1RxDeD5jjzVSbV19Q2Oiqbmlta29I9nZteLZRVfnOd02bHdNUz1uCIvnfOEbfM1xuWpqBl/VdmfC/GqJu56wrWW/7PANU922REHoqk+UkuzLe0VTCcSkXNk0U3nVcHZURaQKihjKa2awXxlWkmmWYVGkfgI5BmnEMW8n75HHFmzoKMIEhwWfsAEVHj3rkMHgELeBgDiXkIjyHBU0k7ZIVZwqVGJ36btNu/WYtWgf9vQitU6nGPS6pExhgD2wS/bC7tkVe2Lvv/YKoh6hlzKtWlXLHaXjuGfp7V+VSauPnU/Vn559FDAeeRXk3YmY8BZ6VV86OHlZmlgcCAbZOXsm/2fskd3RDazSq36xwBdP//CjkZff/1iYjytohPL3gf0EK9mMPJLJLoymp6bjYSbQi34M0cTGMIVZzCNHJxzhGje4leYkR9qTytVSqSbWdONLSIcfaXWkfQ==
m
X
i=1
↵i(x)fi(x)
AAAC0HichVE9LwRRFD3G9/paNBLNxobQbN4gIRLJhkbJrkVimcyMt7yYr8zMbjDZiFajVKhIFKLWUmj8AcX+BFGSaBTuzE4iCO5k5p137j33nTdXcwzh+YzVGqTGpuaW1rb2REdnV3dPsrdvxbPLrs4Lum3Y7pqmetwQFi/4wjf4muNy1dQMvqrtzof51Qp3PWFby/6+wzdMddsSJaGrPlFKcqzolU0lELNyddNMFVXD2VEVMVrUzGCvOpYqfWIlmWYZFkXqJ5BjkEYci3byAUVswYaOMkxwWPAJG1Dh0bMOGQwOcRsIiHMJiSjPUUWCtGWq4lShErtL323arcesRfuwpxepdTrFoNclZQrD7JFdsRf2wK7ZE3v/tVcQ9Qi97NOq1bXcUXqOB/Jv/6pMWn3sfKr+9OyjhOnIqyDvTsSEt9Dr+srB6Ut+JjccjLAL9kz+z1mN3dMNrMqrfrnEc2d/+NHIy+9/LMzHFTRC+fvAfoKV8Yw8kRlfmkxn5+JhtmEQQxiliU0hiwUsokAnnOAGt7iTctKedCgd1UulhljTjy8hHX8An3GnuQ==
context
(query,key,value)
context /subtype ...
( )
"Attention Weights"
m
X
i=1
↵(x, xi)yi
AAAC0HichVE9LwRRFD3G9/eikWgmNoRENm+QEIlEaJQsi8QymRlveTFfmZndWJONaDVKhYpEIWothcYfUPgJoiTRKNyZnUQQ3Jc3975z77nvvLm6awo/YOyxRqqtq29obGpuaW1r7+hMdXWv+E7RM3jOcEzHW9M1n5vC5rlABCZfcz2uWbrJV/XduSi/WuKeLxx7OSi7fMPStm1REIYWEKSmhvN+0VJDMa1UNi05r5nujjaU161wrzIiV70qhuWyKtRUmmVYbPLPQEmCNBJbcFL3yGMLDgwUYYHDRkCxCQ0+rXUoYHAJ20BImEeRiPMcFbQQt0hVnCo0Qnfpu02n9QS16Rz19GO2QbeYtD1iyhhgD+ySvbB7dsWe2PuvvcK4R6SlTF6vcrmrdh71Lr39y7LIB9j5ZP2pOUABk7FWQdrdGIleYVT5pf2Tl6Wp7EA4yM7ZM+k/Y4/sjl5gl16Ni0WePf1Dj05afv9jUT6poBEq3wf2M1gZzShjmdHF8fTMbDLMJvShH0M0sQnMYB4LyNENx7jGDW6lrLQnHUiH1VKpJuH04ItJRx+/26fH
↵(x, xi) =
k(x, xi)
P
j
k(x, xj)
AAAC93icSyrIySwuMTC4ycjEzMLKxs7BycXNw8vHLyAoFFacX1qUnBqanJ+TXxSRlFicmpOZlxpaklmSkxpRUJSamJuUkxqelO0Mkg8vSy0qzszPCympLEiNzU1Mz8tMy0xOLAEKxQsExCTmFGQkasQk5VZX1OooQOj4TE0FBVuFmLSixOTqbJgkXK62Oqa4NDc+SwFdKkuzNl5A2UDPAAwUMBmGUIYyAxQE5AucYIhhSGHIZ0hmKGXIZUhlyGMoAbJzGBIZioEwmsGQwYChACgWy1ANFCsCsjLB8qkMtQxcQL2lQFWpQBWJQNFsIJkO5EVDRfOAfJCZxWDdyUBbcoC4CKhTgUHV4KrBSoPPBicMVhu8NPiD06xqsBkgt1QC6SSI3tSCeP4uieDvBHXlAukShgyELrxuLmFIY7AAuzUT6PYCsAjIF8kQ/WVV0z8HWwWpVqsZLDJ4DXT/QoObBoeBPsgr+5K8NDA1aDYe9yQB3YI7xEDyUBXAKDREjzBMRpiRnqGxnlGgibKDEzQyORikGZQYNIAxZs7gwODBEMAQCrRhG8MdhqcMz5gqmeYzLWNaAVHKxAjVI8yAApg2AgA4s7gO
query key
value
( )
?
ࢀߟ) Attention Models in Graphs: A Survey (2018, arXiv:1807.07984)