前処理が単語埋め込みに与える影響 A Comprehensive Analysis of Preprocessing for Word Representation Learning in Affective Tasks

ACL2020 に採択された A Comprehensive Analysis of Preprocessing for Word Representation Learning in Affective Tasks という論文を読んでいます。特に感情認識系のタスクにおいて前処理が単語埋め込みに与える影響を調べ、よく行われる実験設定が本当に正しいのかを検証しています。

Transcript

1. 15min LT: A Comprehensive Analysis of Preprocessing for Word Representation Learning in Affective Tasks
2. ಺ڮ ݎࢤ uchi_k @__uchi_k__ About me yuni, inc. ୅ද nlpaper.challenge ӡӦ Freelance Machine Learning ɹɹɹɹɹEngineer / Researcher former ژେ৘ใӃ, ະ౿16 FreakOut Machine Learning Engineer
3. About yuni େاۀ͔ΒελʔτΞοϓɺݚڀػؔͳͲ͔Βػցֶशؔ࿈ͷडୗ։ൃ Λߦ͖ͬͯ·ͨ͠ σʔλυϦϒϯͳ΋ͷͮ͘Γࣄۀͱͯ͠ɺΦϯϥΠϯ໰਍ʹΑΔύʔι φϥΠζ৸۩ͷ੡࡞΋΍͍ͬͯͨΓ͠·͢ ࡢ೥຤૑ۀͨ͠ɺγʔυظʹ͋ΔελʔτΞοϓʢࣾһ໊ʣͰ͢ ػցֶशºϚʔέςΟϯάྖҬͰαʔϏε։ൃΛ͍ͯ͠·͢ 6($
   ͷղੳ͕ओͳ࢓ࣄͰɺࠓ೔ͷ࿦จ΋ͦΕʹؔ܎ͨ͠ײ৘ղੳʹͭ ͍ͯ
4. #distributional hypothesis #word embedding ෼෍Ծઆʹجͮ͘୯ޠຒΊࠐΈͷݶք ʮ޾ͤʯͱʮ൵͠ΈʯͷϖΞ͕ʮ޾ͤʯͱʮتͼʯͷϖΞΑΓྨࣅ౓ ͕ߴ͘ͳΔɺͳͲ௚ײʹ൓͢Δྨࣅ౓͕ಘΒΕΔ͜ͱ΋͋ΓɺλεΫ ͝ͱʹ୯ޠຒΊࠐΈΛௐ੔͢Δඞཁ͕͋Δ The Distributional Hypothesis is that words that occur in the same contexts tends to have similar meanings [Harris, 1954]. ࣅͨจ຺Ͱසൟʹग़ݱ͢Δ୯ޠಉ࢜͸ҙຯతʹྨࣅ͍ͯ͠Δͱߟ͑ͯɺ ຒΊࠐΈۭؒͰ΋ۙ͘ͳΔͱ͍͏Ծઆ ୯ޠͷҙຯΛܾΊΔͨΊͷҰͭͷํ๏ͱͯ͠ɺ෼෍Ծઆ͕͋Δɻ ౷ܭతʹ୯ޠͷҙຯΛಘΔͨΊͷํ๏ͰɺXPSEWFD
   ͷΑ͏ͳਪ࿦ ϕʔεͷϞσϧ΍୯ʹ౷ܭ৘ใΛ࣍ݩ࡟ݮ͢ΔΧ΢ϯτϕʔεͷख๏΋ ͋Δ
5. "
   $PNQSFIFOTJWF
   "OBMZTJT
   PG
   1SFQSPDFTTJOH
   
   GPS
   8PSE
   3FQSFTFOUBUJPO
   -FBSOJOH
   JO
   "⒎FDUJWF
   5BTLT #abstract /BTUBSBO
   #BCBOFKBE %FQBSUNFOU
   PG
   &MFDUSJDBM
   &OHJOFFSJOH
   BOE
   $PNQVUFS
   4DJFODF
   FU
   BM
   "$-
    ಛʹײ৘ೝࣝܥͷλεΫʹ͓͍ͯલॲཧ͕୯ޠຒΊࠐΈʹ༩͑ΔӨڹΛௐ΂ɺ
   Α͘ߦΘΕΔ࣮ݧઃఆ͕ຊ౰ʹਖ਼͍͠ͷ͔ݕূ͢Δ ֶशࡁΈͷ୯ޠຒΊࠐΈΛ࢖͍͕͚ͪͩͲɺྫ͑͹ʮ޾ͤʯͱʮ൵͠Έʯͷ ϖΞ͕ʮ޾ͤʯͱʮتͼʯͷϖΞΑΓྨࣅ౓͕ߴ͘ͳΔΑ͏ͳຒΊࠐΈ͕ଘ ࡏ͢Δͷʹײ৘ೝ͕ࣝຊ౰ʹղ͚Δʁ 4UPQXPSET
   OFHBUJPO
   104
   MFNNBUJ[BUJPO
   ͳͲͷલॲཧΛͲ͏࢖͏͔ ͕ຊ࣭తʹॏཁͳͷͰ͸ʁ લॲཧ͕୯ޠຒΊࠐΈʹ༩͑ΔӨڹͷେ͖͞Λݕূ͠ɺैདྷͷ࣮ݧઃఆͷݟ ௚͠Λߦ͍͍ͨ
6. ؔ࿈ݚڀʢXPSEWFD
   6($ʣ • *NQSPWJOH
   %JTUSJCVUJPOBM
   4JNJMBSJUZ
   XJUI
   -FTTPOT
   GSPN
   8PSE
   &NCFEEJOHT
   ◦ 0NFS
   -FWZ #BS*MBO
   6OJWFSTJUZ
   FU
   BM
   "$-
   
   
   ◦ 8PSE
   FNCFEEJOH
   ʹ͓͍ͯɺΧ΢ϯτϕʔεͷख๏Ͱ΋ϋΠύʔύϥϝʔ λௐ੔࣍ୈͰ
   XPSEWFD
   ͳͲͷਪ࿦ϕʔεͷख๏ʹউͯΔ͜ͱΛࣔͨ͠
   • /-'**5
   BU
   *&45
   &NPUJPO
   3FDPHOJUJPO
   VUJMJ[JOH
   /FVSBM
   /FUXPSLT
   BOE
   .VMUJMFWFM
   1SFQSPDFTTJOH
   ◦ 4BNVFM
   1FDBS 4MPWBL
   6OJWFSTJUZ
   PG
   5FDIOPMPHZ
   FU
   BM
   &./-1
   
   ◦ 6TFS
   HFOFSBUFE
   DPOUFOUT
   Λ࢖༻͢Δ৔߹ͷલॲཧͷॏཁੑʹ͍ͭͯௐ΂ ͍ͯΔɻಛʹإจࣈ΍ֆจࣈͷೝࣝΛৄ͘͠ߦ͍είΞΛ্͛Δ͜ͱʹ੒ޭ #recent study #ugc #word2vec
7. ؔ࿈ݚڀʢલॲཧʣ • 0O
   TUPQXPSET
   pMUFSJOH
   BOE
   EBUB
   TQBSTJUZ
   GPS
   TFOUJNFOU
   BOBMZTJT
   PG
   UXJUUFS
   ◦
   )BTTBO
   4BJG ,OPXMFEHF
   .FEJB
   *OTUJUVUF
   5IF
   0QFO
   6OJWFSTJUZ
   FU
   BM
   -3&$
   
   ◦ ετοϓϫʔυͷআڈ͕༗ޮ͔ͦ͏Ͱͳ͍͔͸ϫʔυϦετͷ࡞Γํ΍λε ΫͰେ͖͘ҟͳΔ͕ɺUXJUUFS
   TFOUJNFOU
   Ͱ͸Ұൠతͳํ๏ͩͱ֐ͷํ͕େ ͖͍͜ͱΛࣔͨ͠
   • "
   DPNQBSBUJWF
   FWBMVBUJPO
   PG
   QSFQSPDFTTJOH
   UFDIOJRVFT
   BOE
   UIFJS
   JOUFSBDUJPOT
   GPS
   UXJUUFS
   TFOUJNFOU
   BOBMZTJT
   ◦ 4ZNFPO
   4ZNFPOJEJT
   &YQFSU
   4ZTUFNT
   XJUI
   "QQMJDBUJPOT
   
   ◦ લॲཧͷςΫχοΫΛ৭ʑࢼͯ͠ΈͨΒɺײ৘෼ੳͰ͸
   MFNNBUJ[BUJPO
   ͱ ਺ࣈͷআڈɺ୹ॖܗͷஔ׵͕࠷΋είΞʹد༩ #recent study #preprocessing #emotion
8. "
   $PNQSFIFOTJWF
   "OBMZTJT
   PG
   1SFQSPDFTTJOH
   
   GPS
   8PSE
   3FQSFTFOUBUJPO
   -FBSOJOH
   JO
   "⒎FDUJWF
   5BTLT #abstract λεΫݻ༗ͷඍௐ੔΍Ϟσϧͷվળ΋ॏཁͰ͸͋Δ͕ɺઌߦݚڀ͔Β͸લॲ ཧ΍ϋΠύʔύϥϝʔλͷӨڹ͕ແࢹͰ͖ͳ͍͜ͱ͕ಡΈऔΕΔ ֶश༻σʔλͷ୯ޠຒΊࠐΈΛߦ͏લɾޙͦΕͧΕͷλΠϛϯάͰલॲཧΛ ߦͬͨΓɺςετσʔλͷલॲཧͱ߹ΘͤͨΓ߹Θͤͳ͔ͬͨΓΛࢼ͢ /BTUBSBO
   #BCBOFKBE %FQBSUNFOU
   PG
   &MFDUSJDBM
   &OHJOFFSJOH
   BOE
   $PNQVUFS
   4DJFODF
   FU
   BM
   "$-
    ಛʹײ৘ೝࣝܥͷλεΫʹ͓͍ͯલॲཧ͕୯ޠຒΊࠐΈʹ༩͑ΔӨڹΛௐ΂ɺ
   Α͘ߦΘΕΔ࣮ݧઃఆ͕ຊ౰ʹਖ਼͍͠ͷ͔ݕূ͢Δ
9. #key points ͜ͷ࿦จΛ঺հ͢Δཧ༝ ໘ന͍৽نख๏΋ͨ͘͞Μ͋Δ͕ɺ࣮ӡ༻Ͱਫ਼౓͕ग़ͤΔ΋ͷ͕ͳ͔ ͳ͔ͳ͍ͱײ͍ͯͨ͡ ݁ہલॲཧͷબͼํ΍ख๏ͷҧ͍͕େ͖͘είΞʹӨڹ͍ͯ͠Δ͕ɺ ࿦จͰͦΕΛ࿦͍ͯ͡Δ΋ͷ͕΄ͱΜͲͳ͍ ҉໧஌తͳલॲཧͷ஌ࣝΛ·ͱΊΔ͍͍ػձʹͳΕ͹͍͍͔ͳͱࢥͬ ͨ
10. #key points ΍ͬͨ͜ͱ લॲཧΛ୯ޠຒΊࠐΈʹ౷߹͢ΔͱͲΜͳޮՌ͕͋Δ͔ʁ Ͳͷલॲཧ͕ײ৘෼ੳܥͷλεΫʹޮՌ͕͋Δͷ͔ʁ ࣄલֶश͞Εͨ΋ͷΑΓվળ͞Ε͍ͯΔ͔ʁ ͭͷֶशσʔλɺͭͷςετσʔλΛ࢖༻ͨ͠ײ৘ܥλεΫͰɺֶ शσʔλɺ෼ྨσʔλɺ྆ํɺͦΕͧΕʹલॲཧΛద༻ͨ͠৔߹Ͱൺ ֱ ݕূͨ͜͠ͱ
11. #preprocessing #pipeline /-1
    ʹ͓͚ΔલॲཧͷྲྀΕ ΫϦʔχϯά ෼ׂ ਖ਼نԽ ѹॖ ϕΫτϧԽ λά
    ه߸ͳͲͷআڈ
    QVODUVBUJPO ܗଶૉղੳ
    ࣙॻͷ௥Ճ
    ܎Γड͚ղੳ ਺ࣈͷஔ͖׵͑
    إจࣈͳͲͷೝࣝ
    TQFMMDIFDL
    
    දهΏΕ
    MPXFSDBTJOH
    ୅දޠ΁ͷஔ͖׵͑
    লུޠ
    
    MFNNBUJ[BUJPO
    TUFNNJOH
    OFHBUJPO
    Φϯτϩδʔ 4UPQXPSE
    ͷআڈ
    104 $#08
    TLJQHSBN
    #&35
    DPWFSBHF
    ͷௐࠪ
    ෼ྨσʔλͱޠኮΛ͚ۙͮΔ
    FUD
12. #preprocessing #negation /FHBUJPO • ൓ҙޠࣙॻͷ࡞੒
    ◦8PSE/FU
    ίʔύεͰ൓ҙޠࣙॻΛ࡞੒
    ◦൓ҙޠ͕ݟ͔ͭΒͳ͍
    PS
    ͭͰ͋Ε͹ͦͷ··ɺෳ਺͋Δ৔߹͸
    VL8BD
    ίʔύεͷதͰ࠷େͷස౓Λ࣋ͭ൓ҙޠͱͨ͠Γ୯ʹϥϯμϜ ʹબ୒ͨ͠Γ
    • ൱ఆޠͷ൓ҙޠ΁ͷஔ׵
    ◦൱ఆޠ͕ݟ͔ͭͬͨ৔߹ɺଓ͘୯ޠΛநग़͠ɺ൓ҙޠࣙॻͰ൓ҙޠΛ ݕࡧɻ൓ҙޠ͕ݟ͔ͭͬͨ৔߹ɺ൱ఆޠͱ൱ఆ͞ΕͨޠΛͦΕʹஔ͖ ׵͑Δ
    ◦ྫ͑͹ɺ<b*
    BN
    OPU
    IBQQZ
    bUPEBZ`>ͱ͍͏จͰ͸ɺ൱ఆޠʢ`OPUʣ ͱͦΕʹରԠ͢Δ୯ޠʢIBQQZʣΛಛఆɻ൓ҙޠࣙॻͰ
    bIBQQZ`
    ͷ൓ ҙޠʢ`TBE`ʣΛ୳͠ɺOPU
    IBQQZ`
    Λ
    bTBE`
    ʹஔ͖׵͑Δ
13. #corpus #training #dataset /FXT શମͱͯ͠ɺ4UPQ
    XPSE
    ͷআڈ΍
    104
    Ͱ͸
    WPDBC
    TJ[F
    ͸͋·Γม ΘΒͳ͍͕
    DPSQVT
    TJ[F
    ͕େ͖͘ݮগ ʙ೥ͷΞϝϦΧͷͷग़ ൛෺͔Βͷ ݅ͷهࣄ 8JLJQFEJB 8JLJQFEJB
    ͷهࣄ  ݅Ͱ ߏ੒͞ΕΔɺ/FXT
    ΑΓ໿ഒେ͖ ͍ίʔύε 5SBJOJOH
    $PSQVT ͭͷαΠζɾੑ࣭ͷҟͳΔίʔύεʹͭͷલॲཧΛߦ͏
14. #corpus #evaluation #dataset &WBMVBUJOH
    $PSQVT 4FOUJNFOU
    BOBMZTJT
    FNPUJPO
    DMBTTJpDBUJPO
    
    TBSDBTN
    EFUFDUJPO
    ͷͭͷλεΫͰධՁɻ • *.%#
    ◦
    ݅ͷөըϨϏϡʔɻϙδωΨൺ
    • 4FN&WBM
    ◦
    ໿πΠʔτɻϙδωΨൺ
    • "JSMJOF
    ◦
    ߤۭձࣾࣾʹؔ͢Δ໿݅πΠʔτɻ
    4FOUJNFOU
    BOBMZTJT
    ײ৘ϙδωΨ • *4&"3
    ◦
    ໿݅ͷɺײ৘Λשى͢Δݸਓతͳ࿩
    • "MN
    ◦
    ໿݅ͷ͓ͱ͗࿩
    • 44&$
    ◦
    4FN&WBM
    Λ࠶Ξϊςʔγϣϯͨ͠໿݅ͷπ Πʔτ
    &NPUJPO
    %FUFDUJPO
    ײ৘Ϋϥε෼ྨ 4BSDBTN
    %FUFDUJPO
    ൽ೑ͷݕग़ • 0OJPO
    ◦
    ൽ೑Λѻ͏ϝσΟΞͱͦ͏Ͱͳ͍ϝσΟΞ͔Βऩू ͨ͠໿݅ͷχϡʔεϔουϥΠϯ
    • *"$
    ◦
    ໿݅ͷൃ࿩Ԡ౴
    • 3FEEJU
    ◦
    ஶऀ͕ϥϕϧ෇͚ͨ͠໿ສ݅ͷ
    3FEEJU
    ౤ߘ
    
15. #result /FHBUJPO
    ͕શͯͷσʔληοτʹ͓͍ͯ࠷΋ޮՌతͩͬͨ /FXT
    ίʔύεʹલॲཧΛߦͬͨ͋ͱ୯ޠຒΊࠐΈΛ࡞੒ͨ͠৔߹ͷ
    GTDPSF /FHBUJPO
    Ҏ֎ͷલॲཧͷෳ߹ΛؚΊͯ΋ɺOFHBUJPO
    ͷΈͷ৔߹͕ৗʹ ൪໨ʹείΞ͕ߴ͔ͬͨ
16. #result ʢҰൠతͳʣ4UPQXPSET
    TUFNNJOH
    ͸ɺ୯ମͰ͸ҙຯ͕͋Γͦ͏ʹݟ͑ ͯ΋ଞͷલॲཧͱಉ࣌ͩͱείΞʹد༩͍ͯ͠ͳ͍͜ͱ͕Θ͔Δ શͯͷલॲཧΛద༻ͯ͠΋
    OFHBUJPO
    ͷΈͷ৔߹ͱมΘΒͳ͍͔গ͠Լ͕Δ ͘Β͍ʢ0OJPO
    3FEEJU
    44&$ʣ 4UPQXPSET
    ΍
    104
    ͸ίʔύεαΠζΛେ͖͘ݮগͤ͞Δ͕ɺ104
    Ͱ͸ είΞݮগ͕ͳ͍
17. #result XJLJQFEJB
    DPSQVT
    Λ࢖ͬͨ
    $#08
    4LJQHSBN
    #&35
    ͷ
    'TDPSF
    ൺֱ 8JLJQFEJB
    Ͱ΋ಉ༷ͷ܏޲͕ΑΓڧ·ͬͨ
18. #result #preprocess #postprocess ֶशίʔύεʹલॲཧΛద༻͢Δ৔߹ʢQSFʣͱɺ
    ෼ྨσʔληοτʹલॲཧΛద༻͢Δ৔߹ʢQPTUʣͷൺֱ ௚ײ௨ΓɺQPTU
    ͷΈ͕͍ͣΕͷ৔߹Ͱ΋࠷΋είΞ͕௿͘ͳͬͨ QSF
    ͱ
    CPUI
    ͰείΞʹେ͖ͳ͕ࠩͳ͘ɺQSF
    ͕࠷΋ॏཁͰ͋Δ͜ͱ͕ࣔ ͞Εͨ Ұൠతʹɺ୯ޠຒΊࠐΈ͕༩͑ΒΕͨ৔߹͸෼ྨσʔλΛ߹Θͤʹ͍͘͜ͱ ͕ଟ͍Α͏ʹࢥ͏ͷͰɺҙ֎ͳ݁Ռ
19. #result #compare with SoTA 4P5"
    ϕʔεϥΠϯʹର͢ΔఏҊϞσϧͷධՁ
    શͯͷλεΫͰఏҊख๏܈͕
    4P5"
    Λ্ճΔ #&35
    ͕Ұ൪ڧ͍ͷ͸౰વͳͷͰগͣ͠Δ͍͕ɺఏҊख๏શମͱͯ͠উͬͯ ͍Δ΋ͷ͕ଟ͍ʢ*.%#
    *"$
    0OJPO
    3FEEJU
    44&$ʣ 4P5"
    ͳࣄલֶशϞσϧ͸ఏҊख๏ΑΓང͔ʹେ͖͍ίʔύεΛ࢖͍ͬͯΔ ͷͰɺQSF
    ͷॏཁੑ͕Θ͔Δ
20. #result #relative improvement GTDPSF
    ͷઈର஋ͱجຊతͳલॲཧ͔Βͷ૬ରతͳվળ TFOUJNFOU
    BOBMZTJT
    ͱ
    TBSDBTN
    EFUFDUJPO
    ͷͭͷόΠφϦλεΫΑ ΓϚϧνΫϥε෼ྨλεΫͰͷվળ෯ͷ΄͏͕एׯେ͖͍ ΑΓଟ͘ͷσʔληοτͰൺֱ͠ͳ͍ͱ·ͩ·ͩඍົͳࠩͰ͔͠ͳ͍Α͏ͳ ؾ͸͢Δ
21. #key points ·ͱΊ ୯ޠຒΊࠐΈ࣌఺Ͱͷલॲཧ͕࠷΋λΠϛϯάͱͯ͠༗ޮͰ͋Δ͜ͱ ͕ࣔ͞Εͨ ୯ମͱͯ͠͸
    OFHBUJPO
    ͕࠷΋ޮՌ͕͋ΓɺҰൠతͳ
    TUPQXPSET
    ΍
    TUFNNJOH
    ͸είΞΛԼ͛Δ͜ͱ͕ଟ͍ ڊେͳίʔύεͰֶशࡁΈͷ୯ޠຒΊࠐΈΛ࢖͏ΑΓɺλεΫʹ߹Θ ͤͨલॲཧΛద੾ͳλΠϛϯάͰߦ͏͜ͱͰείΞͰ্ճΕΔ ҉໧஌తʹ஌ΒΕ͍ͯͨ஌ݟ͕ଟ͍͕ɺ͔ͬ͠Γͨ͠ݕূΛߦ͏͜ͱ Ͱମܥతͳ஌ࣝʹͨ͠