ACL 2020 に採択された Heterogeneous Graph Neural Networks for Extractive Document Summarization を読んでいます。
Heterogeneous Graph Neural Networks forExtractive Document Summarization
View Slide
ڮ ݎࢤuchi_k @__uchi_k__About meyuni, inc. දnlpaper.challenge ӡӦFreelance Machine LearningɹɹɹɹɹEngineer / Researcherformer ژେใӃ, ະ౿16FreakOut Machine Learning Engineer
nlpaper.challengeࣗવݴޠॲཧͷΛ͍Ζ͍Ζ͢ΔࣾձਓɾֶੜɾݚڀऀͷίϛϡχςΟʢϘϥϯςΟΞத৺ͰӡӦʣ"$-ͷશཏΛࢦͯ͠ɺ"$-ެࣜʹ͋Δʹै͍ɺͷΛઃఆͯ͠ɺͦΕͧΕͷνʔϜʹ͔ΕͯαʔϕΠຊఔͷจΛಡΈɺٞ-5ձͳͲΛ͍ͯ͠·ͨ͠
ACL2020ੜܥɺάϥϑܥͷจ͕͔ͳΓ૿͑ͨҹ#&35 3P#&35BͷࣄલֶशݴޠϞσϧʹؔ͢Δݴٴ͕΄΅ඞͣ͋Δ࠶ݱੑͷࢹ࣮ͷԠ༻͔Βɺࢦඪͷݟ͕͠ਐΜͩϕετϖʔύʔɺ/-1λεΫͷςετέʔεΈ͍ͨͳͷΛఆٛͯ͠௨աΛݟΑ͏Έ͍ͨͳΛ͍ͯͨ͠Γ,OPXMFEHFHSBQIʹճؼͯ͠ɺάϥϑ্ͰͷԋࢉάϥϑߏɺֶशΛߦ͏Α͏ͳ͕૿ՃҎ্ɺࢲݟͰͨ͠
)FUFSPHFOFPVT(SBQI/FVSBM/FUXPSLTGPS&YUSBDUJWF%PDVNFOU4VNNBSJ[BUJPO#abstractจॻཁͰɺηϯςϯεؒͷؔੑͷϞσϧԽ͕ඇৗʹॏཁɻैདྷɺ3//ϕʔεͷख๏ͰܥྻͰϞσϧԽ͍ͯͨ͠%BORJOH8BOH 4IBOHIBJ,FZ-BCPSBUPSZPG*OUFMMJHFOU*OGPSNBUJPO1SPDFTTJOH 'VEBO6OJWFSTJUZ FUBM "$-நग़తจॻཁͰηϯςϯεؒͷؔੑΛදݱ͢ΔͨΊʹIFUFSPHFOFPVTHSBQIΛಋೖ͠ɺ4P5"Λୡ֦ுੑͳͲʹ͍ͭͯݕূͨ͠ɻจॻͷҙຯߏܥྻΑΓάϥϑߏͷํ͕ద͍ͯ͠Δ͜ͱ͕࠷ۙͷݚڀͰΘ͔͖͍ͬͯͯΔ͕ɺྑ͍άϥϑߏ·ͩఏҊ͞Ε͍ͯͳ͔ͬͨ୯ޠϊʔυͱจϊʔυΛ࣋ͭIFUFSPͳHSBQIߏΛఏҊ͠ɺ୯จॻɾଟจॻཁͦΕͧΕͰ4P5"Λୡɻ֦ுੑʹ͍ͭͯٞͨ͠
#abstract #extractive document summarizationݩͷจॻ͔Βؔ࿈͢ΔจॻΛऔΓग़ͯ͠ɺཁͱͯ͠࠶ߏ͢ΔλεΫநग़తจॻཁ୯ޠΛܦ༝ͨ͠จͷؔੑΛදݱ͢ΔIFUFSPHSBQIΛఆٛυΩϡϝϯτͷ֤ηϯςϯεΛ#JEJSFDUJPOBM-45.ͰϕΫτϧԽɻ͜ΕʹΑͬͯηϯςϯεͷҙຯΛଊ͑ͨϕΫτϧ͕࡞ΒΕΔʢXPSEMBZFSʣநग़ܕͱɺදݱΛநԽͯ͠θϩ͔ΒཁจΛ࡞ΔੜܕɺͦΕΒͷࠞ߹ͷύλʔϯ͕͋Δ͞Βʹ͜ͷϕΫτϧಉ࢜ͷؔੑΛ#JEJSFDUJPOBM-45.Ͱֶश͢ΔʢTFOUFODFMBZFSʣηϯςϯεΛநग़͢Δ֬Λग़ྗ4VNNB3V//FS ॳظͷݚڀ
)FUFSPHFOFPVT(SBQI࣮ੈքͷάϥϑIFUFSPHFOFPVTͳͷ͕ଟ͍࣮ੈքͷάϥϑɺҟͳΔಛۭؒͷ༷ʑͳλΠϓͷϊʔυɾΤοδͰߏ͞Ε͍ͯΔ#abstract #heterogeneous graph
#model overviewηϯςϯεͷΈΛϊʔυͱͯ͠άϥϑΛߏங͢ΔͷͰͳ͘ɺηϯςϯεΛͭͳ͙հͷΑ͏ͳϊʔυΛՃ1SPQPTFE(SBQI୯ޠΛܦ༝ͨ͠จͷؔੑΛදݱ͢ΔIFUFSPHSBQIΛఆٛจใͰ୯ޠϊʔυΛߋ৽Ͱ͖Δ ଞͷϊʔυλΠϓΛՃ͢ΔͳͲͷ֦ுੑ͕͋ΔɺͳͲͷར͜ͷจͰɺ࠷খҙຯ୯ҐΛ୯ޠʹ͍ͯ͠Δɻྫ͑ɺΑΓநԽͯ͠୯ޠͷҙຯ֓೦ΛϊʔυλΠϓͱ͢Δ͜ͱ໘നͦ͏HSBQIJOJUJBMJ[Fˠ("5Ͱߋ৽ˠηϯςϯεಛ͔ΒཁจʹՃ͢Δ͔൱͔ͷྨΛղ͘ɺͱ͍͏खॱ
#model overview #learning stepHSBQIJOJUJBMJ[FSͰɺจʹΧʔωϧαΠζͷҟͳΔ$//Λద༻ͯ͠OHSBNಛΛநग़ʢہॴಛʣɺ࣍ʹ#J-45.ͰηϯςϯεϨϕϧͷಛΛநग़ʢେҬಛʣ1SPQPTFE(SBQIֶशखॱͱNPEFMPWFSWJFX୯ޠϊʔυͱจϊʔυͷؔੑʹؔ͢Δใͱͯ͠ɺUGJEGΛΤοδಛͰ༻͢Δάϥϑಛ(SBQI"UUFOUJPO/FUXPSLͰߋ৽
#model overview #graph attention networkࣗͱपғʹͦΕͧΕॏΈΛ͔͚ͨϕΫτϧ͔ΒBUUFOUJPOΛܭࢉ͠ɺपลϊʔυ͔ΒͷBHHSFHBUJPOʹར༻(SBQI"UUFOUJPO/FUXPSLάϥϑ্ͰͷBUUFOUJPOΛఆٛ"UUFOUJPOྡϊʔυ"UUFOUJPOΛܭࢉ͢Δؔ"UUFOUJPOΛߟྀͨ͠BHHSFHBUJPOάϥϑूͷڑؔΛɺάϥϑߏʹґଘ͠ͳ͍BUUFOUJPOͱͯ͠ఆֶٛ͠शϕʔεͰٻΊΔɺΈ͍ͨͳϊʔυಛ
#dataset #train test split%BUBTFU୯จॻཁͰͭɺෳจॻཁͰͭͷσʔληοτͰ࣮ݧ• ୯จॻཁͰ࠷͘ར༻͞Ε͍ͯΔϕϯνϚʔΫσʔληοτ• USBJO WBMJE UFTUσʔλͦΕͧΕ $//%BJMZ.BJM2"σʔλ• /FX:PSL5JNFT"OOPUBUFE$PSQVT 4BOEIBVT ͔Βऩू͞Εͨ୯จॻཁσʔληοτ• USBJO WBMJE UFTUσʔλͦΕͧΕ ݅/:5.VMUJ/FXT• ෳจॻཁσʔληοτ• ͦΕͧΕʙͷจॻʹର͠ɺਓ͕ؒॻ͍ͨཁ͕͋Δ• USBJO WBMJE UFTUσʔλͦΕͧΕ
#experiment #setting #hyper-parameter #preprocessing4FUUJOH)ZQFSQBSBNFUFSTલॲཧάϥϑ࣮ݧετοϓϫʔυ۟ಡͷআڈ ೖྗจॻͷ࠷େΛจʹઃఆ UGJEGԼҐΛআڈ ޠኮΛʹ੍ݶ ࣍ݩͷ(MP7FͰຒΊࠐΈจϕΫτϧαΠζͰॳظԽ Τοδಛྔ࣍ݩͰॳظԽ IFBEόοναΠζ ֶशF "EBN FQPDIͰMPTT͕Լ͕Βͳ͍߹FBSMZTUPQQJOH ୯จॻཁͰ্Ґจ ෳจॻཁͰ্ҐจΛબ
#methods #extractor• &YU#J-45.◦$//#J-45.◦จॻΛจͷܥྻͱΈͳ͠จؔΛֶश͢Δ• &YU5SBOTGPSNFS◦5SBOTGPSNFSUSBOTGPSNFS◦શจͷϖΞϫΠζ૬ޓ࡞༻Λֶश◦จϨϕϧͷશ࿈݁άϥϑͱΈͳͤΔ• )4( )FUFS4VN(SBQI◦ఏҊख๏ɻจ୯ޠจͷؔੑΛάϥϑͰϞσϧԽ◦)4(ͰϊʔυྨʹΑͬͯཁจΛબ͠ɺ͞ΒʹUSJHSBNCMPDLJOHʹΑͬͯUSJHSBN͕ࣅ͍ͯΔจΛআ֎͠ੑΛ͑ͨόʔδϣϯ࣮ݧ.FUIPET
#result #CNN/DailyMail3FTVMUʢ୯จॻཁɿ$//%BJMZ.BJMʣ$//%BJMZ.BJMͰͷ୯จॻཁͷ݁Ռɻطଘख๏ͯ͢Λ্ճΔείΞ͕ಘΒΕͨɻ-&"%͕ϕʔεϥΠϯɺ03"$-&͕VQQFSCPVOEMBCFMQSFWJPVTTUVEZQSPQPTFENFUIPEจ຺όϯσΟουͱͯ͠ఆٛͨ͠)&3ʹؔͯ͠ಛʹϙϦγʔ͋Γͳ࣮͠ݧ͠ɺ͍ͣΕউͪʢ#&35Λ͍ͬͯͳ͍ʣશͯͷطଘख๏ΑΓߴ͍είΞ͕ಘΒΕͨ306(& -ͰධՁɻͦΕͧΕHSBN HSBN Ұக͢Δ࠷ܥྻͷྨࣅͷείΞ
#result #CNN/DailyMail3FTVMUʢ୯จॻཁɿ$//%BJMZ.BJMʣจܥྻશଓάϥϑΛར༻ͨ͠ख๏ͱൺΔ͜ͱͰɺIFUFSPHSBQIߏͷ༗༻ੑ͕ࣔ͞Εͨɻ&YUNFUIPEQSPQPTFENFUIPEจܥྻɺશଓάϥϑΛͬͨ&YU#J-45. &YU5SBOTGPSNFSΑΓߴ͍είΞIFUFSPHSBQIΛ͏͜ͱͰɺηϯςϯεؒͷෆཁͳ݁߹ΛޮՌతʹআڈͰ͖͍ͯΔ
#result #NYT503FTVMUʢ୯จॻཁɿ/:5ʣ/:5Ͱͷ୯จॻཁͷ࣮ݧ݁Ռɻ$//%BJMZ.BJMͱجຊతʹಉ͕͡ݟΒΕͨɻجຊతʹ$//%BJMZ.BJMͱಉ͡ͰɺఏҊख๏͕طଘख๏Λ্ճ͍ͬͯΔQSPQPTFENFUIPEUSJHSBNCMPDLJOH͋Γόʔδϣϯ͕ҐͰͳ͍ͷͳͥɾɾɾʁˠ$//%BJMZ.BJMͰॏෳͷগͳ͍Օॻ͖Λ࿈݁͢Δܗ͕ࣜͩɺ/:5ͰΩʔϑϨʔζ͕ෳճొ͢ΔͳͲॏෳ͕͋ΔɻͳͷͰɺUSJHSBNCMPDLJOHͰ/:5ͰείΞΛग़ͮ͠Β͍ͷͰ
#ablation #CNN/DailyMail୯ޠϑΟϧλϦϯάͷআͰ3 3-είΞݮগ 3είΞ૿Ճ"CMBUJPO$//%BJMZ.BJMͰBCMBUJPO͠ϞδϡʔϧͷߩݙΛௐͨɻ୯ޠϑΟϧλϦϯάʹΑΓɺಛʹॏཁͳ୯ޠϊʔυʹϑΥʔΧεͰ͖Δར͕CJHSBNใΛࣦ͏σϝϦοτΛ্ճ͍ͬͯΔͷͰͳ͍͔("5ؒͷSFTJEVBMDPOOFDUJPOΛআ͢Δ͜ͱͰείΞ͕େ͖͘ݮগ("5ͷSFTJEVBMDPOOFDUJPOɺIFUFSPHSBQIʹ͓͚ΔผλΠϓͷϊʔυ͔ΒͷूͰཧతʹॏཁͳͷͰ୯ͳΔ݁߹Ͱஔ͖͑Ͱ͖ͳ͍
#result #multidocument)4( )%4(ڞʹطଘख๏Λ্ճΔείΞ͕ಘΒΕ͍ͯͯɺಛʹ)%4(ͰείΞ্ঢ͕େ͖͍3FTVMUʢଟจॻཁʣଟจॻཁͰจॻϊʔυΛՃͨ͠ఏҊख๏ͰݕূจॻϊʔυͷՃ͕ଟจॻཁʹޮՌతͰ͋Δ͜ͱ͕ࣔࠦUSJHSBNCMPDLJOH͕ޮ͍͍ͯͳ͍ͷɺ͓ͦΒ͖ͬ͘͞ͱಉ͡ཧ༝ఏҊख๏Ͱ୯ʹϊʔυλΠϓΛՃ͢Δ͚ͩͰผλεΫʹԠ༻Ͱ͖͓ͯΓɺൃలੑ͕ߴ͍QSPQPTFENFUIPE
#qualitative analysis #degree୯ޠϊʔυͷ͕ߴ͍ͱɺͦͷ୯ޠͷग़ݱ͕ଟ͍ͱ͍͏͜ͱʹͳΓจॻͷΛʢଟগʣද͢2VBMJUBUJWF"OBMZTJT୯ޠϊʔυͷ͕༩͑ΔӨڹΛௐࠪ୯ޠϊʔυ͕͋Δ͜ͱͰɺจใͷूͱେҬදݱͷ͕ߦΘΕ͍ͯΔՄೳੑ͕ࣔࠦ͞ΕΔ୯ޠͷͱ306(&͕ൺྫˠੑͷߴ͍จॻ΄Ͳཁ͠қ͍͕ߴ͍ͱෳͷจͷใΛू͢Δ͜ͱ͕Ͱ͖ɺϞσϧͷԸܙΛΑΓڧ͘ड͚Δ͜ͱ͕Ͱ͖Δͱߟ͑ΒΕΔ
#qualitative analysis #sourceจॻ͕૿Ճ͢Δ͜ͱͰɺϕʔεϥΠϯ্ঢ͢Δ͕ఏҊख๏ͰԼ͠ จͰฒͿ2VBMJUBUJWF"OBMZTJTଟจॻཁͰɺจॻͷͷӨڹΛௐࠪจॻͷ૿ՃͰ)&5&346.(3"1)ͱ)&5&3%0$46.(3"1)ͷੑೳ͕֦ࠩେจॻͱจॻͷ͕ؔෳࡶʹͳΔ΄Ͳɺจॻϊʔυͷར͕ΑΓେ͖͘ͳΔ'JSTUɺΧόϨοδΛ֬อͰ͖ΔจষΛ֤จॻ͔Βڧ੍తʹநग़Ͱ͖Δจॻͷ૿Ճʹ͍ɺશจͷओࢫΛΧόʔͰ͖ΔݶΒΕͨͷจΛநग़͢Δ͜ͱ͕ࠔʹͳ͍ͬͯͨ͘Ί
#key points·ͱΊIFUFSPHSBQIΛ͏͜ͱͰɺจॻཁʹpOFHSBJOFEͳҙຯ୯ҐΛಋೖ͢Δ͜ͱ͕Ͱ͖ɺจɾจষؒͷؔੑͷϞσϦϯάͷ༗ޮੑ͕͔֬ΊΒΕͨख๏ͷ֦ுੑߴ͘ɺ୯จॻཁ͔ΒϊʔυλΠϓͷՃͷΈͰଟจॻཁʹରԠՄೳIFUFSPHSBQIʹಛԽͨ͠ख๏ʢϝλύεΛͬͨαϒάϥϑͷఆٛɺIFUFSPHSBQIʹର͢ΔBUUFOUJPOʣΛࢼ͢ͱ໘ന͍͔ࠓޙ#&35ࣄલֶशϞσϧΛ͍Ζ͍Ζݕ౼͍ͨ͠ͱͷ͜ͱචऀܰ͘৮Ε͍͕ͯͨɺ୯ޠϊʔυʹͨΔ෦͕ҙຯϊʔυ·ͰநԽ͞ΕͨΓͨ͠Βख๏ͷ༏Ґੑ͕ΑΓ׆͔͞ΕΔͱࢥ͏ɻͦ͏Ͱͳͯ͘ɺϊʔυλΠϓͷՃ͍Ζ͍Ζࢼͤͦ͏