Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Finetuned Language Models Are Zero-Shot Learners(最先端NLP2022)

Masaru Isonuma
September 20, 2022

Finetuned Language Models Are Zero-Shot Learners(最先端NLP2022)

Masaru Isonuma

September 20, 2022
Tweet

More Decks by Masaru Isonuma

Other Decks in Technology

Transcript

  1. ࿦จ঺հ: Finetuned Language Models Are Zero-Shot Learners Jason Wei, Maarten

    Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, Quoc V. Le Google Research ಡΈखɿүপ େ @ICLR2022 ஫ه͕ͳ͍ݶΓɺਤද͸ຊ࿦จ͔ΒͷҾ༻
  2. • JOTUSVDUJPOUVOJOHλεΫͷࢦࣔ΍ྫࣔΛϓϩϯϓτͱͯ͠Ճ͑ͨϚϧνλεΫֶश • ֶश࣌ʹ͸༩͍͑ͯͳ͍λεΫͱͦͷࢦࣔΛ༻͍ͯධՁ͢Δ͜ͱͰɺ Ϟσϧ͕ະ஌ͷࢦࣔͱλεΫʹ൚ԽͰ͖Δ͔ධՁ λεΫͷࢦࣔͱͦΕʹର͢Δճ౴Λֶश͢ΔJOTUSVDUJPOUVOJOHΛఏҊ 5 5SBOTMBUFUIFGPMMPXJOH UFYUJOUP+BQBOFTF )PXBSFZPV

    8IBUJTUIFTFOUJNFOU PGUIJTNPWJFSFWJFX *MJLFTQPSUT 5IJTNPWJFJTGVO *EPO`UMJLFUIJTGJMN ͓ݩؾͰ͔͢ʁ ࢲ͸ӡಈ͕޷͖Ͱ͢ɻ QPTJUJWF OFHBUJWF ֶश ධՁ ௨ৗͷϚϧνλεΫֶश 1BSBQISBTFUIF GPMMPXJOHUFYU *MPWFGSVJUT *HPUPTDIPPM *FOKPZFBUJOHGSVJU *BUUFOETDIPPM ຋༁ ײ৘෼ੳ ݴ͍׵͑ + + + + + + ⋮ ⋮ ⋮ ਺ेݸͷ λεΫ ֶश͍ͯ͠ ͳ͍λεΫ ϓϩϯϓτ Ϟσϧ
  3. ۙ೥ྨࣅͷऔΓ૊Έ͕૬͙࣍ 6 '-"/ ঺հ࿦จ 5<> /BUVSBM*OTUSVDUJPOT W<> W<> .FUB*$-<> த৺૊৫

    ൃදձٞ ϓϩϯϓτ ͷ಺༰ #JH4DJFODF XPSLTIPQ *$-3 *$-3 W"$- WBS9JW ౤ߘத /""$- ࢦࣔ ˎྫࣔΛՃ࣮͑ͨݧ༗Γ ࢦࣔ ࢦࣔ ྫࣔ ˎෛྫ౳Ճ࣮͑ͨݧ΋༗Δ͕ɺ λεΫͷࢦࣔ ྫ͕ࣔ࠷ߴੑೳ ྫࣔ ˎࢦࣔΛՃ࣮͑ͨݧ༗Γ Ϟσϧ -B.%"15 EFDPEFSPOMZ # 5-. FODPEFSEFDPEFS ## W#"35 FODPEFSEFDPEFS . W5-. (15 EFDPEFSPOMZ . [1] Sanh et al., Multitask Prompted Training Enables Zero-Shot Task Generalization. ICLR, 2022. [2] Mishra et al., Cross-Task Generalization via Natural Language Crowdsourcing Instructions. ACL, 2022. [3] Wang et al., Benchmarking Generalization via In-Context Instructions on 1,600+ Language Tasks. arXiv, 2022. [4] Min et al., MetaICL: Learning to Learn In Context. NAACL, 2022. (JUIVCͷϨϙδτϦ΍ϓϩϯϓτ ऩूαΠτͷυϝΠϯΛ΋ͱʹಉఆ
  4. • ैདྷͷσʔληοτʹՃ͑ɺϓϩϯϓτΛऩू͢ΔྲྀΕ͕࢝·͍ͬͯΔ ϓϩϯϓτΛऩू͢ΔϓϥοτϑΥʔϜ͕ొ৔ 7 [1] Bach et al., PromptSource: An

    Integrated Development Environment and Repository for Natural Language Prompts. ACL demo, 2022. https://github.com/bigscience-workshop/promptsource [2] Wang et al., Benchmarking Generalization via In-Context Instructions on 1,600+ Language Tasks. arXiv, 2022. https://instructions.apps.allenai.org/ 1SPNQUTPVSDFʢ5ͷֶशධՁʹ࢖༻ʣ<> /BUVSBM*OTUSVDUJPOT<>
  5. • طଘͷσʔληοτΛ༻ҙ͠ɺ֤σʔληοτΛΫϥελʹ෼ྨ • Ұ෦ͷΫϥελͰֶशΛߦ͍ɺֶशʹ࢖༻͠ͳ͍ΫϥελͰධՁ͢Δ͜ͱͰݫີʹ[FSPTIPUੑೳΛଌΔ σʔληοτͷऩूɾ෼ྨ 9 /BUVSBM-BOHVBHF6OEFSTUBOEJOH ෼ྨλεΫ /BUVSBM-BOHVBHF(FOFSBUJPO ੜ੒λεΫ

    * ߋʹɺ3FBEDPNQXDPNNPOTFOTFΫϥελͰධՁ͢Δࡍ͸3FBEJOHDPNQͱ$PNNPOTFOTFΫϥελ͸ֶशʹ༻͍ͣɺ /BUVSBMMBOHVBHFJOGFSFODFΫϥελͰධՁ͢Δࡍ͸1BSBQISBTFΫϥελ͸ֶशʹ༻͍ͳ͍ɻٯ΋ಉ༷ɻ
  6. • (15# #SPXOFUBM  • (-B. #& %VFUBM  •

    -B.%"15# 5IPQQJMBO FUBM  – XFC্ͷจষɺର࿩σʔλɺXJLJQFEJBͰ ࣄલֶशͨ͠EFDPEFSPOMZݴޠϞσϧ – ࣄલֶशίʔύεʹ͸ίʔυ΍ӳޠҎ֎ͷ จষ΋ؚ·ΕΔ • '-"/# – ࣄલֶशࡁΈ-B.%"15Λ JOTUSVDUJPOUVOJOH ࣮ݧઃఆ 13 ϕʔεϥΠϯͱఏҊ๏ σʔληοτ ϕʔε ϥΠϯ ఏҊ๏ ֶश ධՁ • ධՁͰ࢖༻͢ΔΫϥελҎ֎ͷશσʔληο τΛࠞ߹ • ֤σʔληοτͷσʔλ਺Λ࠷େສ݅ʹ੍ ݶ͠ɺσʔληοτؒͷෆۉߧΛܰݮ • ςϯϓϨʔτ͸σʔλຖʹϥϯμϜબ୒ • ԼهͷΫϥελͰධՁ – OBUVSBMMBOHVBHFJOGFSFODF SFBEJOH DPNQSFIFOTJPO DMPTFECPPL2"  USBOTMBUJPO DPNNPOTFOTFSFBTPOJOH  DPSFGFSFODFSFTPMVUJPO TUSVDUUPUFYU • ֤ςϯϓϨʔτͰಘͨੑೳͷฏۉΛଌఆ – ධՁࢦඪ͸BDDVSBDZ #-&6ͳͲ λεΫʹΑΓҟͳΔ * '-"/ͷςϯϓϨʔτΛೖྗͯ͠΋ੑೳ͕ஶ͘͠௿͍ͨΊɺ શͯͷϕʔεϥΠϯϞσϧ͸(15ͷϓϩϯϓτͰධՁ
  7. • JOTUSVDUJPOUVOFͨ͠Ϟσϧ ͷํ͕ɺ UVOF͍ͯ͠ͳ͍Ϟσϧ ΑΓߴ͍ੑೳ • ଟ͘ͷσʔληοτͰ(15΍(-B. Λ্ճΔ – େྔͷจষΛಡΈࠐΜͰֶश͢ΔΑΓɺ

    JOTUSVDUJPOUVOJOHͷํ͕༗ޮͱࣔࠦ instruction tuning͕ػೳͨ͠λεΫ 14 (accuracy) (accuracy/F1) (accuracy) (BLEU)
  8. • Ϟσϧ͕େ͖͘ͳΔ΄Ͳੑೳ͕޲্ɻશͯͷ࿦จͰಉ༷ͷ܏޲͕ใࠂɻ • '-"/Ͱ͸#ҎԼͷ৔߹JOTUSVDUJPOUVOJOHʹΑΓΉ͠Ζੑೳ͕௿Լ͢Δ͜ͱΛใࠂ͍ͯ͠Δ͕ɺ ଞͷ࿦จͰ͸ΑΓখ͍͞ϞσϧͰJOTUSVDUJPOUVOJOHͷ༗ޮੑΛใࠂ – 5 /BUVSBM*OTUSVDUJPOTW5-. # –

    .FUB*$-(15MBSHF . – /BUVSBM*OTUSVDUJPOTW#"35CBTF . • ϓϩϯϓτͷҧ͍ʢԼͭ͸ྫࣔΛՃ͍͑ͯΔʣ ࣄલֶशͷҧ͍ʢ$BVTBM-.WT.BTLFE-.ʣ ͕ݪҼʁ Ϟσϧͷେ͖͞ 20 A100/V100 ͰֶशՄೳ
  9. • /BUVSBM*OTUSVDUJPOTͰ͸਺छྨͷϓϩϯϓτΛ࡞੒ – %FGλεΫͷఆٛʢࢦࣔʣ 1PTλεΫͷਖ਼ྫʢྫࣔʣ /FHλεΫͷෛྫ &YQMਖ਼ྫͱෛྫͷࠜڌͷઆ໌ • /FHʢෛྫʣ΍&YQMʢࠜڌͷઆ໌ʣΛՃ͑ͯ΋ੑೳ͸޲্ͤͣɺ%FG 1PTʢࢦࣔʴྫࣔʣͰे෼

    λεΫͷࢦࣔͱྫࣔҎ֎ʹ༗ޮͳϓϩϯϓτ͸͋Δ͔ʁ 22 ग़య: Wang et al., Benchmarking Generalization via In-Context Instructions on 1,600+ Language Tasks. arXiv, 2022. ֶश࣌ʹ ࢖༻ͨ͠ ϓϩϯϓτ ධՁ࣌ʹ࢖༻ͨ͠ϓϩϯϓτ දதͷ਺஋͸ෳ਺λεΫͷฏۉੑೳʢ306(&-ʣ
  10. • ϓϩϯϓτΛՃ͑ͯϚϧνλεΫֶशΛߦ͏JOTUSVDUJPOUVOJOHΛఏҊ͠ɺ ϓϩϯϓτΛՃ͑ͳ͍৔߹ʹൺ΂ͯɺ൚Խੑೳ͕޲্͢Δ͜ͱΛ֬ೝ – େن໛ϞσϧͰେྔͷจষΛಡΈࠐΉํ๏ʢ(15ͳͲʣͱ͸ҟͳΔΞϓϩʔν͕ൃݟ͞Εͨ – ͲͪΒ͕༏Ε͍ͯΔ͔͸ެฏʹධՁͰ͖͍ͯͳ͍ • ൚Խੑೳʹର͠༩͑ΔӨڹ͸ҎԼͷ௨Γ –

    λεΫσʔληοτ਺ɿଟ͍΄͏͕ੑೳ͸ߴ·Δ͕ɺੑೳ޲্΁ͷد༩౓͸λεΫʹΑΓ·ͪ·ͪ – Ϟσϧͷେ͖͞ɿେ͖͍΄Ͳੑೳ͸ߴ͍ɻҰఆҎ্ͷେ͖͕͞ඞཁ͔͸ෆಁ໌ – ϓϩϯϓτͷ಺༰ɿݱঢ়Ͱ͸ɺλεΫͷࢦࣔͱྫࣔʢਖ਼ྫʣ͕࠷ߴੑೳ • ࠓޙͷݚڀ՝୊ – ຊ౰ʹϓϩϯϓτΛཧղͰ͖͍ͯΔ͔ʁ ࣍ͷ৿Լ͞Μͷ࿦จ঺հΛָ͓͠Έʹʂ > 8FCTPO 1BWMJDL l%P1SPNQU#BTFE.PEFMT3FBMMZ6OEFSTUBOEUIF.FBOJOHPG5IFJS1SPNQUT z/""$-  – ൚Խੑೳ޲্ʹ༗༻ͳࢦࣔ΍λεΫ͸Կ͔ʁ – ೚ҙͷࢦࣔΛཧղͰ͖Δ͜ͱΛͲ͏ධՁ͢Δ͔ʁ ݁࿦ 23