Share
Embed
論文紹介: Incorporating Syntactic and Semantic Information in Word Embeddings using Graph Convolutional Networks
- TMU Komachi lab
- paper reading (ACL2019)
- Incorporating Syntactic and Semantic Information in
Word Embeddings using Graph Convolutional Networks. Vashishth et al., ACL2019
- paper URL: https://www.aclweb.org/anthology/P19-1320.pdf
Satoru Katsumata
December 11, 2023
Transcript
15 slides
-
*ODPSQPSBUJOH4ZOUBDUJDBOE 4FNBOUJD*OGPSNBUJPO JO8PSE&NCFEEJOHT VTJOH(SBQI$POWPMVUJPOBM/FUXPSLT 4IJLIBS7BTIJTIUI .BOJL#IBOEBSJ 1SBUFFL:BEBW 1JZVTI3BJ $IJSBOKJC#IBUUBDIBSZZB
1BSUIB5BMVLEBS "$- 5.6,PNBDIJMBC.4BUPSV,BUTVNBUB -
"CTUSBDU w 5IJTQBQFSQSPQPTFT4ZO($/ B(SBQI$POWPMVUJPOCBTFE NFUIPEGPSMFBSOJOHXPSEFNCFEEJOHT w 6OMJLFQSFWJPVTNFUIPET 4ZO($/VUJMJ[FTTZOUBDUJD DPOUFYUGPSMFBSOJOHXPSESFQSFTFOUBUJPOTXJUIPVU JODSFBTJOHWPDBCVMBSZTJ[F
w 5IFZBMTPQSFTFOU4FN($/ BGSBNFXPSLGPSJODPSQPSBUJOH EJWFSTFTFNBOUJDLOPXMFEHFJOMFBSOFEXPSEFNCFEEJOHT w 5IFZEFNPOTUSBUFUIBUPVSQSPQPTFENFUIPETPCUBJO TVCTUBOUJBMJNQSPWFNFOUPWFSTUBUFPGUIFBSUBQQSPBDIFT !2 -
(SBQI$POWPMVUJPOBM /FUXPSLTGPS/-1 w %F⒎FSSBSEFUBM /*14,JQGBOE8FMMJOH *$-3.BSDIFHHJBOJBOE5JUPW &./-1 w ($/TPO%JSFDUFE-BCFMFE(SBQIT
w &EHF-BCFM(BUJOH.FDIBOJTN !3 v OPEF XPSE FEHF JOQVUOPEFGFBUVSFT d EJNFOTJPO n = |v| f = ReLU hk+1 v = f ∑ u∈𝒩+ (v) (Wk luv hk u + bk luv ) 𝒩+ (v) = 𝒩(v) ∪ {v} G = (V, E, X) E X 2 Rn⇥d hk+1 v = f ∑ u∈𝒩+ (v) gk luu × (Wk luv hk u + bk luv ) gk luv = σ ( ̂ Wk luv hk u + ̂ bk luv ) -
(SBQI$POWPMVUJPOBM /FUXPSLTGPS/-1 w %F⒎FSSBSEFUBM /*14,JQGBOE8FMMJOH *$-3.BSDIFHHJBOJBOE5JUPW &./-1 w ($/TPO%JSFDUFE-BCFMFE(SBQIT
w &EHF-BCFM(BUJOH.FDIBOJTN !4 v OPEF XPSE FEHF JOQVUOPEFGFBUVSFT d EJNFOTJPO n = |v| GPS%JSFDUFE(SBQI ˠ6TFUIFPQQPTJUFEJSFDUJPO GSPNEFQFOEFOUTUPIFBET GPS-BCFMFE(SBQI ˠBOEMBCFMTQFDJpDNPEFMQBSBNFUFST Wk luv bk luv EJTDPWFS XBUFS PO W0 self W0 self W0 self W0 self W0 self h0 W0 subj W0 subj−1 W0 obj−1 W0 obj W0 case W0 nmod W0 nmod−1 4DJFOUJTUT .BST DBTF TVCK PCK ONPE -PPLVQUBCMF G Є G Є G Є G Є G Є h0 h0 h0 h0 W0 case−1 h1 h1 h1 h1 h1 h0 hk+1 v = f ∑ u∈𝒩+ (v) (Wk luv hk u + bk luv ) 𝒩+ (v) = 𝒩(v) ∪ {v} -
.PUJWBUJPO w 4LJQHSBN (MP7F FUDʜ ˠ5IFTFNFUIPETIBWFBSFTUSJDUJPOPGUIFXJOEPXTJ[F ˠ6TJOH%FQFOEFODZCBTFEDPOUFYUIFMQTUPBMMFWJBUFUIJTQSPCMFN w "MMFYJTUJOHTZOUBDUJDDPOUFYUCBTFENFUIPETFYQBOEWPDBCTJ[F ʢlTDJFOUJTUT@TVCKz
lXBUFS@PCKz lNBST@ONPEzOFFETUPCFJODMVEFEBTBQBSUPG WPDBCGPSlEJTDPWFSzʣ ˠ5IJTQBQFSQSPQPTFT4ZO($/ XIJDIFNQMPZT($/T 5IJTNFUIPEEPFTOPUFYQBOEWPDBCTJ[F w &OSJDIJOHXPSEFNCFEEJOHTXJUITFNBOUJDLOPXMFEHFIFMQTUPJNQSPWFUIFJS RVBMJUZGPSTFWFSBM/-1UBTLT ˠ&YJTUJOHBQQSPBDIFTBSFFJUIFSJODBQBCMFPGVUJMJ[JOHUIFTFEJWFSTFSFMBUJPOTPS OFFEUPCFFYQMJDJUMZNPEFMFEGPSFYQMPJUJOHUIFN ˠ5IJTQBQFSQSPQPTFT4FN($/ XIJDIMFBSOTUPVUJMJ[FNVMUJQMFTFNBOUJD DPOTUSBJOTCZNPEFMJOHUIFNBTEJ⒎FSFOUFEHFUZQFT !5 3FUSPpU FUDʜ -
4ZO($/ w JOQVUEFQFOEFODZQBSTFETFOUFODF w -JLFB$#08NPEFM w 4ZO($/EPFTOPUJODMVEFTFMGMPPQT w ($/TMBZFSJTPOF w
USBJOJOH/FHBUJWF4BNQMJOH !6 4DJFOUJTUT .BST DBTF TVCK PCK ONPE -PPLVQUBCMF G Є G Є G Є G Є G Є h0 h0 h0 h0 W0 case−1 h1 h1 h1 h1 h1 EJTDPWFS XBUFS PO h0 W0 case W0 subj W0 subj−1 W0 obj−1 W0 obj W0 nmod W0 nmod−1 UBSHFUXBUFS $POUFYU FNCFEEJOH 0VUQVUMBZFS E = |V| ∑ t=1 log P (wt |wt 1 , wt 2 …wt Nt ) P (wt |wt 1 , wt 2 …wt Nt ) = exp (vT wt ht) ∑ |V| i=1 exp (vT wi ht) E = |V| ∑ t=1 ( vT wt ht − log |V| ∑ i=1 exp (vT wi ht)) hk+1 i = f ∑ j∈N(i) gk lij × (Wk lij hk j + bk lij ) -
4FN($/ w JOQVUDPSQVTBOETFNBOUJDLOPXMFEHF GSPN8PSE/FUPS11%# w *ODPSQPSBUJOHTFNBOUJDLOPXMFEHFJOQSFUSBJOFEXPSEFNCFEEJOHT w &YJTUJOHBQQSPBDIFTTZOPOZNZBOEBOUPOZNZ TZNNFUSJDJOGPSNBUJPO
ˠ5IJTNFUIPEDBOJODPSQPSBUFTZNNFUSJDBOEBTZNNFUSJD JOGPSNBUJPOKPJOUMZ w 1SFTFSWFUIFTFNBOUJDTFODPEFEJOQSFUSBJOFEFNCFEEJOHT ˠJOJUJBMJ[JOHCPUIUBSHFUBOEDPOUFYUFNCFEEJOHT ˠLFFQJOHUBSHFUFNCFEEJOHTpYFEEVSJOHUSBJOJOH w 4FN($/JODMVEFTTFMGMPPQTUPQSFTFSWFUIFJOJUJBMMFBSOFESFQSFTFOUBUJPO w USBJOJOH/FHBUJWF4BNQMJOH !7 -
4FN($/ w JOQVUDPSQVTBOETFNBOUJDLOPXMFEHF GSPN8PSE/FUPS11%# w *ODPSQPSBUJOHTFNBOUJDLOPXMFEHFJOQSFUSBJOFEXPSEFNCFEEJOHT w &YJTUJOHBQQSPBDIFTTZOPOZNZBOEBOUPOZNZ TZNNFUSJDJOGPSNBUJPO
ˠ5IJTNFUIPEDBOJODPSQPSBUFTZNNFUSJDBOEBTZNNFUSJD JOGPSNBUJPOKPJOUMZ w 1SFTFSWFUIFTFNBOUJDTFODPEFEJOQSFUSBJOFEFNCFEEJOHT ˠJOJUJBMJ[JOHCPUIUBSHFUBOEDPOUFYUFNCFEEJOHT ˠLFFQJOHUBSHFUFNCFEEJOHTpYFEEVSJOHUSBJOJOH w 4FN($/JODMVEFTTFMGMPPQTUPQSFTFSWFUIFJOJUJBMMFBSOFESFQSFTFOUBUJPO w USBJOJOH/FHBUJWF4BNQMJOH !8 -
&YQFSJNFOUTFUUJOHT w %BUB8JLJQFEJB NJMMJPOTFOUT CJMMJPOUPLFOTCJMMJPOTZOUBDUJDEFQFOEFODJFT w #BTFMJOFT w WT4ZO($/
8PSEWFD $#08 (MP7F%FQT EFQFOEFODZ &95 TFDPOEPSEFS EFQFOEFODZ w WT4FN($/ 3FUSPpU$PVOUFSpU BOUPOZN TZOPOZN +PJOU3FQT KPJOUMFBSOJOH w &WBMVBUJPO w *OUSJOTJD5BTLT8PSE4JNJMBSJUZ$PODFQU$BUFHPSJ[BUJPO8PSE"OBMPHZ w &YUSJOTJD5BTLT/&32"104UBHHJOH$PSFGFSFODF3FTPMVUJPO !9 -
4FN($/SFTVMUT VTJOHWBSJPVTSFMBUJPOT !11 /PUBUJPO FH4FN($/ 4ZO($/ ˠJOJU4ZO($/QPTUQSPDFTTJOH4FN($/
OVNPGSFMBUJPOT 3FMBUJPO 3ˠTZOPOZN 3ˠTZOPOZNBOEBOUPOZN 3ˠTZOPOZN BOUPOZN IZQFSOZNBOEIZQPOZN 4FN($/XJUI4ZO($/HJWFT UIFCFTUQFSGPSNBODF -
$PODMVTJPO w 5IJTQBQFSIBTQSPQPTFE4ZO($/ w (SBQIDPOWPMVUJPOCBTFEBQQSPBDIVTJOHTZOUBDUJDDPOUFYUGPSMFBSOJOHXPSE SFQSFTFOUBUJPOT w 4ZO($/PWFSDPNFTUIFQSPCMFNPGWPDBCVMBSZFYQMPTJPO 5IJTNFUIPEPVUQFSGPSNTTUBUFPGUIFBSUXPSEFNCFEEJOHBQQSPBDIFTPO TFWFSBMJOUSJOTJDBOEFYUSJOTJDUBTLT
w 5IJTQBQFSBMTPQSPQPTFT4FN($/ w KPJOUMZJODPSQPSBUJOHEJWFSTFTFNBOUJDJOGPSNBUJPOJOQSFUSBJOFEXPSE FNCFEEJOHT w 5IFDPNCJOBUJPOPG4ZO($/BOE4FN($/HJWFTUIFCFTUPWFSBMM QFSGPSNBODF !14 -
3FMBUFEXPSLT w $POWPMVUJPOBMOFVSBMOFUXPSLTPOHSBQITXJUIGBTU MPDBMJ[FETQFDUSBMpMUFSJOH%F⒎FSSBSEFUBM /*14 w 4FNJTVQFSWJTFEDMBTTJpDBUJPOXJUIHSBQIDPOWPMVUJPOBM OFUXPSLT,JQGBOE8FMMJOH *$-3 w
&ODPEJOHTFOUFODFTXJUIHSBQIDPOWPMVUJPOBMOFUXPSLT GPSTFNBOUJDSPMFMBCFMJOH.BSDIFHHJBOJBOE5JUPW &./-1 w 4PVSDFDPEFIUUQTHJUIVCDPNNBMMMBCJJTD8PSE($/ !15