Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[SNLP2021] Prefix-Tuning: Optimizing Continuous Prompts for Generation

191b51e394766638990831da6691d089?s=47 junya-takayama
September 09, 2021

[SNLP2021] Prefix-Tuning: Optimizing Continuous Prompts for Generation

191b51e394766638990831da6691d089?s=128

junya-takayama

September 09, 2021
Tweet

Transcript

  1. Prefix-Tuning: Optimizing Continuous Prompts for Generation Xiang Lisa Li, Percy

    Liang. ACL-IJCNLP2021  ୈճ࠷ઌ୺ /-1ษڧձ ಡΉਓ ߴࢁ ൏໼ʢࡕେʣ ਤද͸஫ऍͷͳ͍ݶΓશͯݩ࿦จ͋Δ͍͸ஶऀεϥΠυ͔ΒͷҾ༻Ͱ͢
  2. ͲΜͳ࿦จʁ ØࣄલֶशࡁΈݴޠੜ੒ϞσϧͷܰྔͳλεΫసҠ๏ͷఏҊ ØλεΫݻ༗ͷύϥϝʔλྻʢ1SFpYʣΛϞσϧ֤૚ʹ 1SPNQUతʹ ೖྗ͢Δ͜ͱͰɼϞσϧΛ࠶ֶशͤͣʹλεΫ΁ͷసҠΛߦ͏ ֤λεΫ͝ͱͷอଘ͢΂͖ύϥϝʔλͷαΠζΛܶతʹ࡟ݮ✌✌ Øཁ໿ͱ 5BCMFUP5FYUͰ͸௨ৗͷϑΝΠϯνϡʔχϯάͱ΄΅ޓ֯ͷ ੑೳΛୡ੒ɽ֎ૠ΍গࢿݯઃఆʹڧ͍ 1

  3. ຊ࿦จͰѻ͏λεΫ $POEJUJPOBMOBUVSBMMBOHVBHFHFOFSBUJPO ೖྗܥྻ 9ʹج͍ͮͯग़ྗจ :Λੜ੒͢ΔλεΫ܈ Ø5BCMFUP5FYU 95BCMFσʔλ :9Λઆ໌͢Δจষ Øཁ໿ 9

    ௕͍ هࣄจॻ :9Λཁ໿ͨ͠จষ 2 Table: name[Clowns] customer-rating[1 out of 5] eatType[coffee shop] area[riverside] food[Chinese] near[Clere Hall] Description: Clowns is a coffee shop in the riverside area near Clare Hall that has a rating 1 out of 4. They serve Chinese food. Article: Scientists at University College London discovered people tend to think that their hands are wider and their fingers are shorter than they truly are. They say the confusion may lie in the way the brain… Summary: The brain naturally distorts body image – a finding which could explain disorders like anorexia, say experts.
  4. എܠ 'VMM 'JOFUVOJOH ࣄલֶशࡁΈݴޠੜ੒Ϟσϧʢ(15 #"35ͳͲʣͷશମΛ࠶ֶश ☺ ଟ͘ͷݴޠੜ੒λεΫʹ͓͍ͯ EFGBDUPTUBOEBSEͳํ๏ ͭΑ͍ "

    ֤λεΫ͝ͱʹϞσϧશମΛอଘ͢Δඞཁ͕͋Γɼߴίετ ࣄલֶशͷڧΈ͸ੜ͔ͭͭ͠ɼετϨʔδʹ༏͍͠ख๏͕ཉ͍͠ʂ 3
  5. എܠ*ODPOUFYUMFBSOJOH λεΫͷઆ໌จͱগ਺ͷ࣮ྫͰ 1SPNQUJOH ☺ ύϥϝʔλΛߋ৽͠ͳ͍ͨΊλεΫ͝ͱʹϞσϧΛ༻ҙ͢Δඞཁͳ͠ " Ϟσϧ΁ͷೖྗ௕੍͕ݶ͞ΕΔͨΊɼେ͖ͳ܇࿅σʔλΛѻ͑ͳ͍ " (15ͷΑ͏ͳ௒ڊେϞσϧͰͷΈద༻Մೳ "

    1SPNQUΛਓखͰ࡞੒͢Δඞཁ͕͋Γɼඇৗʹߴίετ 4
  6. Ґஔ෇͚-JHIUXFJHIUpOFUVOJOH ݴޠੜ੒Ϟσϧͷ΄ͱΜͲͳ͍͠શ෦ͷύϥϝʔλΛݻఆ͠ɼ Ұ෦ͷ૚΍৽ͨʹ௥Ճͨ͠αϒωοτϫʔΫͷΈΛֶश͢Δख๏܈ 5 Ø্૚ L૚͚ͩΛ࠶ֶश ! 'VMMpOFUVOJOHʹൺ΂ੑೳ͕௿Լ Øখ͍͞αϒωοτϫʔΫΛ௥Ճʢ"EBQUFS<>ʣ "

    'VMMpOFUVOJOHͷ ͷύϥϝʔλ਺ ☺ 'VMMpOFUVOJOHʹඖఢ͢Δੑೳ ØఏҊख๏ʢ1SFpYUVOJOHʣ ☺ 'VMMpOFUVOJOHͷ  ͷύϥϝʔλ਺ ☺ 'VMMpOFUVOJOHʹඖఢ͢Δੑೳ <>)PVMTCZ FUBM 1BSBNFUFSF⒏DJFOUUSBOTGFSMFBSOJOHGPS/-1 *$.-
  7. .FUIPE 6

  8. 1SPNQUJOH͓͞Β͍ ৚݅ͱͳΔτʔΫϯྻʢ1SPNQUʣΛݴޠϞσϧʹೖྗͯ͠࢒ΓͷτʔΫϯ Λੜ੒ͤͯ͞໰୊Λղ͘ํ๏ɽύϥϝʔλߋ৽ΛߦΘͳ͍ 1SPNQUΛ্ख͘࡞Ε͹ )BSSZ1PUUFSHSBEVBUFEGSPN  )PHXBSUTͷΑ͏ͳ 2"ͳͲɼ༷ʑͳλεΫΛղ͘͜ͱ͕Ͱ͖Δ ৚݅෇͖ੜ੒λεΫͰ΋ 1SPNQUJOH͍ͨ͠ʂ

    7
  9. 1SPNQUJOHͰ /-(λεΫΛղ͚ͳ͍͔ʁ Ø͍͍ײ͡ʹ໰୊Λղ͚ΔΑ͏ͳ 1SPNQUʢ୯ޠྻʣΛݟ͚ͭΔʁ ! ཭ࢄ࠷దԽ͸೉͍͠ ! Ϟσϧ͕࣋ͭ୯ޠ &NCFEEJOH͚ͩͰ͸දݱྗ͕଍Γͳ͍ ØλεΫݻ༗ͷ

    &NCFEEJOHྻΛֶश͢Δʁ ! දݱྗɾੑೳ͕ʢ1SFpYUVOJOHΑΓ͸ʣ௿͍ʢஶऀΒͷݕূ݁Ռʣ ØશϨΠϠʔʹରͯ͠λεΫݻ༗ͷ࿈ଓ஋ύϥϝʔλʢ1SFpYʣΛೖྗ " දݱྗ͕ߴ͍ 8
  10. 1SFGJY5VOJOHGPS"VUPSFHSFTTJWFNPEFM (15 ࣌ࠁ ! ʹ͓͚Δதؒঢ়ଶ ℎ# ʹ͍ͭͯɼ ℎ# = %

    &' !, : *+, -# , ℎ.# , if ! ∈ &234 , otherwise. ଛࣦؔ਺͸ΫϩεΤϯτϩϐʔʢͨͩ͠ੜ੒Ϟσϧ͸ϑϦʔζʣ 9 Prefix
  11. 1SFpY5VOJOHGPS&ODPEFSEFDPEFSNPEFM #"35 ࣌ࠁ ! ʹ͓͚Δதؒঢ়ଶ ℎ# ʹ͍ͭͯɼ ℎ# = %

    &' !, : *+, -# , ℎ.# , if ! ∈ &234 , otherwise. ଛࣦؔ਺͸ΫϩεΤϯτϩϐʔʢͨͩ͠ੜ੒Ϟσϧ͸ϑϦʔζʣ 10 Prefix
  12. &YQFSJNFOUT 11

  13. 5BCMFUP5FYU (15 12 σʔλ8FC/-( L USBJOΧςΰϦ UFTU ΧςΰϦ Ø'JOFUVOJOHͷ αΠζͷ

    ύϥϝʔλ਺Ͱ 'JOFUVOJOHͱ ίϯύϥͳੑೳΛୡ੒ Øಉֶ͡शՄೳύϥϝʔλ਺Ͱ ൺֱ͢Δͱɼ"EBQUFSUVOJOH ΑΓ͸Δ͔ʹྑ͍ੑೳΛୡ੒
  14. 5BCMFUP5FYU (15 ֎ૠͰͷ݁Ռ σʔλ8FC/-( L USBJOΧςΰϦ UFTU ΧςΰϦ USBJOʹؚ·Εͳ͍ΧςΰϦ͚ͩͰͷ݁Ռ ֎ૠͰ͸

    'JOFUVOJOHΛ྇կɽಉ͡ύϥϝʔλ਺Ͱ͸ "EBQUFSͱಉ౳ 1SFpY  ͩͱͲ͏ͳΔΜͩΖ͏ʜʁ 13
  15. ཁ໿ #"35 σʔλ946. L 8FC/-( ͷ໿ഒ  ࣮ݧ݁Ռ ֎ૠઃఆ 14

    Ø8FC/-( ͱ͸ҟͳΓ 'JOFUVOJOH ʹͦͦ͜͜ෛ͚Δ • λεΫͷੑ࣭ʹΑΔ΋ͷʁ • σʔλͷྔͷҧ͍ʁޙͰݕূ Ø֎ૠͰ͸ 'JOFUVOJOHͱಉ౳ ʢ΍΍উͪʁʣ
  16. ཁ໿ #"35 গࢿݯઃఆ σʔλ946. L 8FC/-( ͷ໿ഒ  2σʔλ͕ଟ͍ʢগͳ͍ʣͱ 'JOFUVOJOHʹෛ͚Δʢউͭʣͷ͔ʁ

    USBJOσʔλ਺Λม࣮͑ͯݧ " ͦ͏Β͍͠ ͳΜͰ݅·ͰʁͲ͜·Ͱߦ͘ͱෛ͚Δͷ͔஌Γͨ͘ͳ͍ʁ 15
  17. %JTDVTTJPO 16

  18. ٞ࿦ Øͳͥ֎ૠɾগࢿݯͰڧ͍ͷ͔ʁ • "EBQUFSͱ 1SFpYUVOJOH͕ಉ༷ͷ܏޲Λࣔͨ͜͠ͱ͔Βɼ -.ͷύϥϝʔλ͕อ࣋͞Ε͍ͯΔ͜ͱ͕ݪҼͩͱߟ͑ΒΕΔɽ • ഁ໓త๨٫͕ݪཧ্ى͖ͳ͍͜ͱʹخ͠Έ͕͋Γͦ͏ʁ Øͳͥۃ୺ʹখ͞ͳύϥϝʔλ਺Ͱ্ख͍͘͘ͷ͔ʁ •

    ͋Δσʔληοτʹ͓͍ͯҰఆͷੑೳʹୡ͢Δͷʹඞཁͳ࠷খͷ ύϥϝʔλ਺ʢ*OUSJOTJDEJNFOTJPOʣ͸ɼݴޠϞσϧ͕େ͖͘ɼ ͔ͭࣄલֶशΛଟ͘͢Δ΄Ͳখ͘͞ͳΔ <> Øͳͥඞཁͳύϥϝʔλ਺͸ 1SFpYUVOJOH  "EBQUFS ͳͷ͔ • 1SFpYUVOJOH͸ "EBQUFSͱҟͳΓ -.Λͦͷ··อ࣋͢ΔͨΊɼ ΑΓ -.Λ্ख͘׆༻Ͱ͖ΔͷͰ͸ʁ • ؾ࣋ͪ͸ͳΜͱͳ͘Θ͔Δ͚ͲͪΐͬͱΏΔ͍ʜʁ 17 <>"HIBKBOZBO FUBM *OUSJOTJDEJNFOTJPOBMJUZFYQMBJOTUIFFGGFDUJWFOFTTPGMBOHVBHFNPEFM GJOFUVOJOH "$-*+$/-1
  19. 'PMMPXVQXPSL <>-FTUFSFUBM 5IFQPXFSPGTDBMFGPSQBSBNFUFSF⒏DJFOUQSPNQUUVOJOH &./-1 1SFpYUVOJOHΛ୯७Խͨ͠ 1SPNQUUVOJOHΛ 5ʹద༻ͯ͠ৄࡉͳ෼ੳΛߦͬͨ࿦จ ØݩͷݴޠϞσϧͷαΠζ͕େ͖͍΄Ͳ 1SPNQUUVOJOHͱ 'JOFUVOJOHͷੑೳ͕ࠩ

    খ͘͞ͳΔ͜ͱΛ֬ೝ • ͜Ε͸ "HIBKBOZBO FUBMͷ࿩ͱҰ؏ੑ͕ ͋Δؾ͕͢Δʂ ØҟͳΔॳظ஋͔Β܇࿅͞Εͨෳ਺ͷ 1SPNQU Λ༻͍ͯΞϯαϯϒϧ͢Δ͜ͱͰਪ࿦ੑೳ͕ ޲্͢Δ͜ͱΛ֬ೝ ͳͲ 18 <>ΑΓҾ༻
  20. ·ͱΊ ØλεΫݻ༗ͷܰྔͳύϥϝʔλΛݴޠੜ੒Ϟσϧ֤૚ͷೖྗʹ෇༩͠ɼ ϞσϧͷύϥϝʔλΛมߋͤͣʹ໨తλεΫ΁ͷసҠΛߦ͏ख๏ʢ1SFpY UVOJOHʣΛఏҊ Ø'JOFUVOJOHͱಉ౳ͷੑೳΛୡ੒ͭͭ͠ɼλεΫݻ༗ͷύϥϝʔλ਺Λ 'JOFUVOJOHͷ·Ͱ࡟ݮ ॴײ Ø֎ૠ΍গࢿݯʹڧ͍ͷ͸خ͍͠ Ø"EBQUFSͱຊ࣭తʹԿ͕ҧ͏ͷ͔Θ͔Γ͔ͨͬͨ Ø#"35ͱ

    (15ͷൺֱΛݟ͔ͨͬͨɽཁ໿Ͱ (15 "VUPSFHSFTTJWF -. Λ࢖͏͜ͱ΋ 5BCMFUP5FYUͰ #"35 &ODPEFS%FDPEFS Λ࢖͏ ͜ͱ΋Ͱ͖Δ͸ͣͰ͸ʁ 19