[SNLP2021] Prefix-Tuning: Optimizing Continuous Prompts for Generation

Prefix-Tuning: Optimizing Continuous Prompts for Generation Xiang Lisa Li, Percy
Liang. ACL-IJCNLP2021 ୈճ࠷ઌ୺ /-1ษڧձ ಡΉਓ ߴࢁ ൏໼ʢࡕେʣ ਤද͸஫ऍͷͳ͍ݶΓશͯݩ࿦จ͋Δ͍͸ஶऀεϥΠυ͔ΒͷҾ༻Ͱ͢

ͲΜͳ࿦จʁ ØࣄલֶशࡁΈݴޠੜ੒ϞσϧͷܰྔͳλεΫసҠ๏ͷఏҊ ØλεΫݻ༗ͷύϥϝʔλྻʢ1SFpYʣΛϞσϧ֤૚ʹ 1SPNQUతʹ ೖྗ͢Δ͜ͱͰɼϞσϧΛ࠶ֶशͤͣʹλεΫ΁ͷసҠΛߦ͏ ֤λεΫ͝ͱͷอଘ͢΂͖ύϥϝʔλͷαΠζΛܶతʹ࡟ݮ✌✌ Øཁ໿ͱ 5BCMFUP5FYUͰ͸௨ৗͷϑΝΠϯνϡʔχϯάͱ΄΅ޓ֯ͷ ੑೳΛୡ੒ɽ֎ૠ΍গࢿݯઃఆʹڧ͍ 1

ຊ࿦จͰѻ͏λεΫ $POEJUJPOBMOBUVSBMMBOHVBHFHFOFSBUJPO ೖྗܥྻ 9ʹج͍ͮͯग़ྗจ :Λੜ੒͢ΔλεΫ܈ Ø5BCMFUP5FYU 95BCMFσʔλ :9Λઆ໌͢Δจষ Øཁ໿ 9
௕͍ هࣄจॻ :9Λཁ໿ͨ͠จষ 2 Table: name[Clowns] customer-rating[1 out of 5] eatType[coffee shop] area[riverside] food[Chinese] near[Clere Hall] Description: Clowns is a coffee shop in the riverside area near Clare Hall that has a rating 1 out of 4. They serve Chinese food. Article: Scientists at University College London discovered people tend to think that their hands are wider and their fingers are shorter than they truly are. They say the confusion may lie in the way the brain… Summary: The brain naturally distorts body image – a finding which could explain disorders like anorexia, say experts.

എܠ 'VMM 'JOFUVOJOH ࣄલֶशࡁΈݴޠੜ੒Ϟσϧʢ(15 #"35ͳͲʣͷશମΛ࠶ֶश ☺ ଟ͘ͷݴޠੜ੒λεΫʹ͓͍ͯ EFGBDUPTUBOEBSEͳํ๏ ͭΑ͍ "
֤λεΫ͝ͱʹϞσϧશମΛอଘ͢Δඞཁ͕͋Γɼߴίετ ࣄલֶशͷڧΈ͸ੜ͔ͭͭ͠ɼετϨʔδʹ༏͍͠ख๏͕ཉ͍͠ʂ 3

എܠ*ODPOUFYUMFBSOJOH λεΫͷઆ໌จͱগ਺ͷ࣮ྫͰ 1SPNQUJOH ☺ ύϥϝʔλΛߋ৽͠ͳ͍ͨΊλεΫ͝ͱʹϞσϧΛ༻ҙ͢Δඞཁͳ͠ " Ϟσϧ΁ͷೖྗ௕੍͕ݶ͞ΕΔͨΊɼେ͖ͳ܇࿅σʔλΛѻ͑ͳ͍ " (15ͷΑ͏ͳ௒ڊେϞσϧͰͷΈద༻Մೳ "
1SPNQUΛਓखͰ࡞੒͢Δඞཁ͕͋Γɼඇৗʹߴίετ 4

Ґஔ෇͚-JHIUXFJHIUpOFUVOJOH ݴޠੜ੒Ϟσϧͷ΄ͱΜͲͳ͍͠શ෦ͷύϥϝʔλΛݻఆ͠ɼ Ұ෦ͷ૚΍৽ͨʹ௥Ճͨ͠αϒωοτϫʔΫͷΈΛֶश͢Δख๏܈ 5 Ø্૚ L૚͚ͩΛ࠶ֶश ! 'VMMpOFUVOJOHʹൺ΂ੑೳ͕௿Լ Øখ͍͞αϒωοτϫʔΫΛ௥Ճʢ"EBQUFS<>ʣ "
'VMMpOFUVOJOHͷ ͷύϥϝʔλ਺ ☺ 'VMMpOFUVOJOHʹඖఢ͢Δੑೳ ØఏҊख๏ʢ1SFpYUVOJOHʣ ☺ 'VMMpOFUVOJOHͷ ͷύϥϝʔλ਺ ☺ 'VMMpOFUVOJOHʹඖఢ͢Δੑೳ <>)PVMTCZ FUBM 1BSBNFUFSF⒏DJFOUUSBOTGFSMFBSOJOHGPS/-1 *$.-

.FUIPE 6

1SPNQUJOH͓͞Β͍ ৚݅ͱͳΔτʔΫϯྻʢ1SPNQUʣΛݴޠϞσϧʹೖྗͯ͠࢒ΓͷτʔΫϯ Λੜ੒ͤͯ͞໰୊Λղ͘ํ๏ɽύϥϝʔλߋ৽ΛߦΘͳ͍ 1SPNQUΛ্ख͘࡞Ε͹ )BSSZ1PUUFSHSBEVBUFEGSPN )PHXBSUTͷΑ͏ͳ 2"ͳͲɼ༷ʑͳλεΫΛղ͘͜ͱ͕Ͱ͖Δ ৚݅෇͖ੜ੒λεΫͰ΋ 1SPNQUJOH͍ͨ͠ʂ
7

1SPNQUJOHͰ /-(λεΫΛղ͚ͳ͍͔ʁ Ø͍͍ײ͡ʹ໰୊Λղ͚ΔΑ͏ͳ 1SPNQUʢ୯ޠྻʣΛݟ͚ͭΔʁ ! ཭ࢄ࠷దԽ͸೉͍͠ ! Ϟσϧ͕࣋ͭ୯ޠ &NCFEEJOH͚ͩͰ͸දݱྗ͕଍Γͳ͍ ØλεΫݻ༗ͷ
&NCFEEJOHྻΛֶश͢Δʁ ! දݱྗɾੑೳ͕ʢ1SFpYUVOJOHΑΓ͸ʣ௿͍ʢஶऀΒͷݕূ݁Ռʣ ØશϨΠϠʔʹରͯ͠λεΫݻ༗ͷ࿈ଓ஋ύϥϝʔλʢ1SFpYʣΛೖྗ " දݱྗ͕ߴ͍ 8

1SFGJY5VOJOHGPS"VUPSFHSFTTJWFNPEFM (15 ࣌ࠁ ! ʹ͓͚Δதؒঢ়ଶ ℎ# ʹ͍ͭͯɼ ℎ# = %
&' !, : *+, -# , ℎ.# , if ! ∈ &234 , otherwise. ଛࣦؔ਺͸ΫϩεΤϯτϩϐʔʢͨͩ͠ੜ੒Ϟσϧ͸ϑϦʔζʣ 9 Preﬁx

1SFpY5VOJOHGPS&ODPEFSEFDPEFSNPEFM #"35 ࣌ࠁ ! ʹ͓͚Δதؒঢ়ଶ ℎ# ʹ͍ͭͯɼ ℎ# = %
&' !, : *+, -# , ℎ.# , if ! ∈ &234 , otherwise. ଛࣦؔ਺͸ΫϩεΤϯτϩϐʔʢͨͩ͠ੜ੒Ϟσϧ͸ϑϦʔζʣ 10 Preﬁx

&YQFSJNFOUT 11

5BCMFUP5FYU (15 12 σʔλ8FC/-( L USBJOΧςΰϦ UFTU ΧςΰϦ Ø'JOFUVOJOHͷ αΠζͷ
ύϥϝʔλ਺Ͱ 'JOFUVOJOHͱ ίϯύϥͳੑೳΛୡ੒ Øಉֶ͡शՄೳύϥϝʔλ਺Ͱ ൺֱ͢Δͱɼ"EBQUFSUVOJOH ΑΓ͸Δ͔ʹྑ͍ੑೳΛୡ੒

5BCMFUP5FYU (15 ֎ૠͰͷ݁Ռ σʔλ8FC/-( L USBJOΧςΰϦ UFTU ΧςΰϦ USBJOʹؚ·Εͳ͍ΧςΰϦ͚ͩͰͷ݁Ռ ֎ૠͰ͸
'JOFUVOJOHΛ྇կɽಉ͡ύϥϝʔλ਺Ͱ͸ "EBQUFSͱಉ౳ 1SFpY ͩͱͲ͏ͳΔΜͩΖ͏ʜʁ 13

ཁ໿ #"35 σʔλ946. L 8FC/-( ͷ໿ഒ ࣮ݧ݁Ռ ֎ૠઃఆ 14
Ø8FC/-( ͱ͸ҟͳΓ 'JOFUVOJOH ʹͦͦ͜͜ෛ͚Δ • λεΫͷੑ࣭ʹΑΔ΋ͷʁ • σʔλͷྔͷҧ͍ʁޙͰݕূ Ø֎ૠͰ͸ 'JOFUVOJOHͱಉ౳ ʢ΍΍উͪʁʣ

ཁ໿ #"35 গࢿݯઃఆ σʔλ946. L 8FC/-( ͷ໿ഒ 2σʔλ͕ଟ͍ʢগͳ͍ʣͱ 'JOFUVOJOHʹෛ͚Δʢউͭʣͷ͔ʁ
USBJOσʔλ਺Λม࣮͑ͯݧ " ͦ͏Β͍͠ ͳΜͰ݅·ͰʁͲ͜·Ͱߦ͘ͱෛ͚Δͷ͔஌Γͨ͘ͳ͍ʁ 15

%JTDVTTJPO 16

ٞ࿦ Øͳͥ֎ૠɾগࢿݯͰڧ͍ͷ͔ʁ • "EBQUFSͱ 1SFpYUVOJOH͕ಉ༷ͷ܏޲Λࣔͨ͜͠ͱ͔Βɼ -.ͷύϥϝʔλ͕อ࣋͞Ε͍ͯΔ͜ͱ͕ݪҼͩͱߟ͑ΒΕΔɽ • ഁ໓త๨٫͕ݪཧ্ى͖ͳ͍͜ͱʹخ͠Έ͕͋Γͦ͏ʁ Øͳͥۃ୺ʹখ͞ͳύϥϝʔλ਺Ͱ্ख͍͘͘ͷ͔ʁ •
͋Δσʔληοτʹ͓͍ͯҰఆͷੑೳʹୡ͢Δͷʹඞཁͳ࠷খͷ ύϥϝʔλ਺ʢ*OUSJOTJDEJNFOTJPOʣ͸ɼݴޠϞσϧ͕େ͖͘ɼ ͔ͭࣄલֶशΛଟ͘͢Δ΄Ͳখ͘͞ͳΔ <> Øͳͥඞཁͳύϥϝʔλ਺͸ 1SFpYUVOJOH "EBQUFS ͳͷ͔ • 1SFpYUVOJOH͸ "EBQUFSͱҟͳΓ -.Λͦͷ··อ࣋͢ΔͨΊɼ ΑΓ -.Λ্ख͘׆༻Ͱ͖ΔͷͰ͸ʁ • ؾ࣋ͪ͸ͳΜͱͳ͘Θ͔Δ͚ͲͪΐͬͱΏΔ͍ʜʁ 17 <>"HIBKBOZBO FUBM *OUSJOTJDEJNFOTJPOBMJUZFYQMBJOTUIFFGGFDUJWFOFTTPGMBOHVBHFNPEFM GJOFUVOJOH "$-*+$/-1

'PMMPXVQXPSL <>-FTUFSFUBM 5IFQPXFSPGTDBMFGPSQBSBNFUFSF⒏DJFOUQSPNQUUVOJOH &./-1 1SFpYUVOJOHΛ୯७Խͨ͠ 1SPNQUUVOJOHΛ 5ʹద༻ͯ͠ৄࡉͳ෼ੳΛߦͬͨ࿦จ ØݩͷݴޠϞσϧͷαΠζ͕େ͖͍΄Ͳ 1SPNQUUVOJOHͱ 'JOFUVOJOHͷੑೳ͕ࠩ
খ͘͞ͳΔ͜ͱΛ֬ೝ • ͜Ε͸ "HIBKBOZBO FUBMͷ࿩ͱҰ؏ੑ͕ ͋Δؾ͕͢Δʂ ØҟͳΔॳظ஋͔Β܇࿅͞Εͨෳ਺ͷ 1SPNQU Λ༻͍ͯΞϯαϯϒϧ͢Δ͜ͱͰਪ࿦ੑೳ͕ ޲্͢Δ͜ͱΛ֬ೝ ͳͲ 18 <>ΑΓҾ༻

·ͱΊ ØλεΫݻ༗ͷܰྔͳύϥϝʔλΛݴޠੜ੒Ϟσϧ֤૚ͷೖྗʹ෇༩͠ɼ ϞσϧͷύϥϝʔλΛมߋͤͣʹ໨తλεΫ΁ͷసҠΛߦ͏ख๏ʢ1SFpY UVOJOHʣΛఏҊ Ø'JOFUVOJOHͱಉ౳ͷੑೳΛୡ੒ͭͭ͠ɼλεΫݻ༗ͷύϥϝʔλ਺Λ 'JOFUVOJOHͷ·Ͱ࡟ݮ ॴײ Ø֎ૠ΍গࢿݯʹڧ͍ͷ͸خ͍͠ Ø"EBQUFSͱຊ࣭తʹԿ͕ҧ͏ͷ͔Θ͔Γ͔ͨͬͨ Ø#"35ͱ
(15ͷൺֱΛݟ͔ͨͬͨɽཁ໿Ͱ (15 "VUPSFHSFTTJWF -. Λ࢖͏͜ͱ΋ 5BCMFUP5FYUͰ #"35 &ODPEFS%FDPEFS Λ࢖͏ ͜ͱ΋Ͱ͖Δ͸ͣͰ͸ʁ 19

[SNLP2021] Prefix-Tuning: Optimizing Continuous Prompts for Generation

[SNLP2021] Prefix-Tuning: Optimizing Continuous Prompts for Generation

junya-takayama

More Decks by junya-takayama

Other Decks in Research

Featured

Transcript

Prefix-Tuning: Optimizing Continuous Prompts for Generation Xiang Lisa Li, Percy

ຊ࿦จͰѻ͏λεΫ $POEJUJPOBMOBUVSBMMBOHVBHFHFOFSBUJPO ೖྗܥྻ 9ʹج͍ͮͯग़ྗจ :Λੜ੒͢ΔλεΫ܈ Ø5BCMFUP5FYU 95BCMFσʔλ :9Λઆ໌͢Δจষ Øཁ໿ 9

എܠ 'VMM 'JOFUVOJOH ࣄલֶशࡁΈݴޠੜ੒Ϟσϧʢ(15 #"35ͳͲʣͷશମΛ࠶ֶश ☺ ଟ͘ͷݴޠੜ੒λεΫʹ͓͍ͯ EFGBDUPTUBOEBSEͳํ๏ ͭΑ͍ "

എܠ*ODPOUFYUMFBSOJOH λεΫͷઆ໌จͱগ਺ͷ࣮ྫͰ 1SPNQUJOH ☺ ύϥϝʔλΛߋ৽͠ͳ͍ͨΊλεΫ͝ͱʹϞσϧΛ༻ҙ͢Δඞཁͳ͠ " Ϟσϧ΁ͷೖྗ௕੍͕ݶ͞ΕΔͨΊɼେ͖ͳ܇࿅σʔλΛѻ͑ͳ͍ " (15ͷΑ͏ͳ௒ڊେϞσϧͰͷΈద༻Մೳ "

Ґஔ෇͚-JHIUXFJHIUpOFUVOJOH ݴޠੜ੒Ϟσϧͷ΄ͱΜͲͳ͍͠શ෦ͷύϥϝʔλΛݻఆ͠ɼ Ұ෦ͷ૚΍৽ͨʹ௥Ճͨ͠αϒωοτϫʔΫͷΈΛֶश͢Δख๏܈ 5 Ø্૚ L૚͚ͩΛ࠶ֶश ! 'VMMpOFUVOJOHʹൺ΂ੑೳ͕௿Լ Øখ͍͞αϒωοτϫʔΫΛ௥Ճʢ"EBQUFS<>ʣ "

.FUIPE 6

1SPNQUJOH͓͞Β͍ ৚݅ͱͳΔτʔΫϯྻʢ1SPNQUʣΛݴޠϞσϧʹೖྗͯ͠࢒ΓͷτʔΫϯ Λੜ੒ͤͯ͞໰୊Λղ͘ํ๏ɽύϥϝʔλߋ৽ΛߦΘͳ͍ 1SPNQUΛ্ख͘࡞Ε͹ )BSSZ1PUUFSHSBEVBUFEGSPN )PHXBSUTͷΑ͏ͳ 2"ͳͲɼ༷ʑͳλεΫΛղ͘͜ͱ͕Ͱ͖Δ ৚݅෇͖ੜ੒λεΫͰ΋ 1SPNQUJOH͍ͨ͠ʂ

1SPNQUJOHͰ /-(λεΫΛղ͚ͳ͍͔ʁ Ø͍͍ײ͡ʹ໰୊Λղ͚ΔΑ͏ͳ 1SPNQUʢ୯ޠྻʣΛݟ͚ͭΔʁ ! ཭ࢄ࠷దԽ͸೉͍͠ ! Ϟσϧ͕࣋ͭ୯ޠ &NCFEEJOH͚ͩͰ͸දݱྗ͕଍Γͳ͍ ØλεΫݻ༗ͷ

1SFGJY5VOJOHGPS"VUPSFHSFTTJWFNPEFM (15 ࣌ࠁ ! ʹ͓͚Δதؒঢ়ଶ ℎ# ʹ͍ͭͯɼ ℎ# = %

1SFpY5VOJOHGPS&ODPEFSEFDPEFSNPEFM #"35 ࣌ࠁ ! ʹ͓͚Δதؒঢ়ଶ ℎ# ʹ͍ͭͯɼ ℎ# = %

&YQFSJNFOUT 11

5BCMFUP5FYU (15 12 σʔλ8FC/-( L USBJOΧςΰϦ UFTU ΧςΰϦ Ø'JOFUVOJOHͷ αΠζͷ

5BCMFUP5FYU (15 ֎ૠͰͷ݁Ռ σʔλ8FC/-( L USBJOΧςΰϦ UFTU ΧςΰϦ USBJOʹؚ·Εͳ͍ΧςΰϦ͚ͩͰͷ݁Ռ ֎ૠͰ͸

ཁ໿ #"35 σʔλ946. L 8FC/-( ͷ໿ഒ ࣮ݧ݁Ռ ֎ૠઃఆ 14

ཁ໿ #"35 গࢿݯઃఆ σʔλ946. L 8FC/-( ͷ໿ഒ 2σʔλ͕ଟ͍ʢগͳ͍ʣͱ 'JOFUVOJOHʹෛ͚Δʢউͭʣͷ͔ʁ

%JTDVTTJPO 16

ٞ࿦ Øͳͥ֎ૠɾগࢿݯͰڧ͍ͷ͔ʁ • "EBQUFSͱ 1SFpYUVOJOH͕ಉ༷ͷ܏޲Λࣔͨ͜͠ͱ͔Βɼ -.ͷύϥϝʔλ͕อ࣋͞Ε͍ͯΔ͜ͱ͕ݪҼͩͱߟ͑ΒΕΔɽ • ഁ໓త๨٫͕ݪཧ্ى͖ͳ͍͜ͱʹخ͠Έ͕͋Γͦ͏ʁ Øͳͥۃ୺ʹখ͞ͳύϥϝʔλ਺Ͱ্ख͍͘͘ͷ͔ʁ •

'PMMPXVQXPSL <>-FTUFSFUBM 5IFQPXFSPGTDBMFGPSQBSBNFUFSF⒏DJFOUQSPNQUUVOJOH &./-1 1SFpYUVOJOHΛ୯७Խͨ͠ 1SPNQUUVOJOHΛ 5ʹద༻ͯ͠ৄࡉͳ෼ੳΛߦͬͨ࿦จ ØݩͷݴޠϞσϧͷαΠζ͕େ͖͍΄Ͳ 1SPNQUUVOJOHͱ 'JOFUVOJOHͷੑೳ͕ࠩ