ハンドメイド作品を扱うECサイトに特化したBERTを用いた言語モデル構築に向けた取り組み/ipsj-NL250-05

2021.9.28 情報処理学会 第250回自然言語処理研究会(NL研)
https://nl-ipsj.or.jp/2021/09/10/nl250-program/

Transcript

1. ञҪහ඙(GMOϖύϘגࣜձࣾ ϖύϘݚڀॴ/۝भେֶେֶӃ),
   

   ࡾ୐༔հ(GMOϖύϘגࣜձࣾ ϖύϘݚڀॴ), ܀ྛ݈ଠ࿠(GMOϖύϘגࣜձࣾ ϖύϘݚڀॴ)
   ൃදऀɿञҪහ඙ / Pepabo R&D Institute, GMO Pepabo, Inc. / Kyushu University
   
   
   2021.09.28 ৘ใॲཧֶձ ୈ250ճࣗવݴޠॲཧݚڀձ
   ϋϯυϝΠυ࡞඼Λѻ͏ECαΠτʹಛԽͨ͠
   BERTΛ༻͍ͨݴޠϞσϧߏஙʹ޲͚ͨऔΓ૊Έ
   

   View Slide

2. 1. ݚڀͷഎܠ
   
   
   2. ՝୊ͱղܾࡦ
   
   
   3. ࣮ݧ
   
   
   4. ࣮ݧ݁Ռͱߟ࡯
   
   
   5. ·ͱΊͱࠓޙͷ՝୊
   2
   ໨࣍
   

   View Slide

3. 1.
   ݚڀͷഎܠ
   

   View Slide

4. • ฐࣾͰ͸ɺϋϯυϝΠυ࡞඼Λର৅ͱͨ͠CtoCͷECαΠτ(ʮϋϯυϝΠυ࡞
   ඼Λѻ͏ECαΠτʯͱݺͿ)Ͱ͋ΔminneΛӡӦ
   
   
   4
   ݚڀͷഎܠ(1/3)
   ˞NJOOF࡞Ոొ࿥ϚχϡΞϧΑΓ
   ࡞඼Λൢച ࡞඼Λߪೖ
   

   View Slide

5. • ֤࡞඼ʹ͸ɺλΠτϧ΍આ໌จͳͲ
   ͷςΩετ৘ใ͕෇༩͞Ε͍ͯΔ
   
   
   • ֤छλεΫʹར༻͍ͨ͠
   
   
   • ࡞඼ͷΧςΰϦ෼ྨ
   
   
   • λΠτϧɾઆ໌จͷࣗಈੜ੒
   
   
   • ࣭໰Ԡ౴
   
   
   ͳͲ
   5
   ݚڀͷഎܠ(2/3)
   େখΧςΰϦ
   ࡞඼λΠτϧ
   ࡞඼આ໌จ
   ˙࡞඼ͷྫ
   

   View Slide

6. • ͜ΕΒͷλεΫΛਓखͰղ͘ͷ͸ࠔ೉ͳͨΊػցతͳΞϓϩʔν͕ඞཁͱͳΔ
   
   
   • ҰํɺʮϋϯυϝΠυ࡞඼Λѻ͏ECαΠτʯ͸ҎԼͷΑ͏ͳಛ௃Λ࣋ͭ
   
   
   • ࡞඼͕ଟ༷Ͱ͋ΔͨΊɺ֤࡞඼Λਖ਼͘͠ಛ௃͚ͮΔ͜ͱ͕ࠔ೉
   
   
   • αʔϏεͷมԽʹԠͨ͡ߏ଄తͳมԽʹ௥ै͢Δ͜ͱ͸ࠔ೉
   
   
   → ্هͷಛ௃Λଊ͑ͯɺ໰୊ΛղܾͰ͖Δख๏͕ඞཁ
   6
   ݚڀͷഎܠ(3/3)
   

   View Slide

7. • ࣗવݴޠॲཧͰ͸ɺBERT[1]Λ࢖͏͜ͱ͕ओྲྀʹͳ͖͍ͬͯͯΔ
   
   
   • BERTͰ͸ɺࣄલֶशࡁΈϞσϧΛ࢖͏͜ͱͰɺ൚༻తͳ஌ࣝΛ࣋ͭݴޠϞσ
   ϧΛར༻͢Δ͜ͱ͕Ͱ͖Δ
   
   
   • ࣄલֶशࡁΈϞσϧΛɺλεΫʹԠͨ͡গͳ͍ڭࢣ༗ΓσʔλͰ
   fi
   ne-tuning͢
   Δ͜ͱͰɺߴ͍ੑೳΛಘΔ͜ͱ͕Մೳ(ຊݚڀͰ͸ɺBERT+
   fi
   ne-tuningͱݺͿ)
   
   
   • ௥ՃֶशʢBERTͷࣄલֶशࡁΈϞσϧ͔Βେن໛ͳίʔύεͰ࠶ࣄલֶशʣ
   Λߦ͏͜ͱͰɺίʔύεʹಛԽͨ͠஌ࣝ֫ಘ͕Մೳ [4],[6]
   7
   BERT
   [1] %FWMJO + $IBOH .8 -FF ,BOE5PVUBOPWB ,#&351SFUSBJOJOHPG%FFQ#JEJSFDUJPOBM5SBOTGPSNFSTGPS-BOHVBHF6OEFSTUBOEJOH
   
   <>-FF + :PPO 8 ,JN 4 ,JN % ,JN 4 4P $)BOE,BOH +#JP#&35BQSFUSBJOFECJPNFEJDBMMBOHVBHFSFQSFTFOUBUJPONPEFMGPSCJPNFEJDBMUFYUNJOJOH #JPJOGPSNBUJDT 7PM /P QQr
   
   <>/55
   デ
   ʔλۚ༥ۀք޲͚ࣗવݴޠॲཧٕज़ͷݕূ։࢝dۚ༥൛#&35Ϟ
   デ
   ϧͷ։ൃdɼIUUQTXXXOUUEBUBDPNKQKBOFXTSFMFBTF
   

   View Slide

8. • ʮϋϯυϝΠυ࡞඼Λѻ͏ECαΠτʯͰͷࣗવݴޠॲཧͷλεΫΛղ͖͍ͨ
   
   
   → BERT+
   fi
   ne-tuningΛ࢖͏͜ͱͰɺ֤՝୊ΛղܾͰ͖ΔͷͰ͸ͳ͍͔
   
   
   • ࣗવݴޠॲཧͷλεΫͷ಺ɺࠓճ͸ɺ࡞඼ͷΧςΰϦ෼ྨʹऔΓ૊Ή
   
   
   • ຊݚڀͰ͸ɺ࡞඼ͷΧςΰϦ෼ྨʹ͓͍ͯϕʔεϥΠϯख๏ͱBERT+
   fi
   ne-
   tuningͷϞσϧΛൺֱɾධՁ͢Δɻ݁ՌΛ౿·͑ͯɺࠓޙͷํ޲ੑΛܾΊ͍ͨ
   8
   ຊݚڀͷશମ૾
   ˞ϕʔεϥΠϯख๏͸จॻϕΫτϧԽख๏ɿ5'*%'ɺ෼ྨثɿ47.ɺϩδεςΟοΫճؼͱͨ͠
   

   View Slide

9. 2.
   ՝୊ͱղܾࡦ
   

   View Slide

10. • ࡞඼Λग़඼͢Δࡍʹ͸ɺ࡞Ո͕࡞඼ͷେ/খΧςΰϦΛొ࿥͍ͯ͠Δ
    
    
    

    

    

    

    → ొ࿥͢ΔࡍʹɺࣗಈͰେ/খΧςΰϦΛਪଌ͠ɺ࡞Ո΁ఏ͍ࣔͨ͠
    

    ɹɹࣄલʹΧςΰϦΛఏࣔ͢Δ͜ͱͰɺΧςΰϦઃఆޡΓΛ๷ࢭ͍ͨ͠
    
    
    • ͔͠͠ɺ࡞඼ͷΧςΰϦ෼ྨΛӡӦଆ͕ਓखͰߦ͏͜ͱ͸ݱ࣮తͰͳ͍
    

    → ՝୊ᶃʹର͢Δղܾࡦɿػցతʹ෼ྨ͢Δ͜ͱ͕ඞཁ
    10
    ՝୊ᶃɿਓखͰλεΫΛղ͘ͷ͸ࠔ೉
    ˙େΧςΰϦͷҰཡ
    ΞΫηαϦʔ
    ࢦྠɾϦϯά ϐΞε
    ɾɾɾ
    ɾɾɾ
    ˙େখΧςΰϦͷྫ
    

    View Slide

11. • ʮϋϯυϝΠυ࡞඼Λѻ͏ECαΠτʯͰ͸࡞Ո͕࡞ͬͨ࡞඼Λߪೖऀ͕ߪೖ
    

    → ࡞Ո͕ҟͳΔ৔߹ɺಉ͡࡞඼͸ଘࡏ͠ͳ͍
    

    ಉ͡࡞඼͕ଘࡏ͠ͳ͍͜ͱ͕େ͖ͳಛ௃Ͱ͋Δ
    
    
    • औΓѻ͏࡞඼ͷछྨ͕ଟ༷ͱͳΓɺ࡞඼Λద੾ͳΧςΰϦ΁෼ྨ͢ΔͨΊʹ͸
    ෯޿͍࡞඼஌͕ࣝඞཁͱͳΔ
    

    → ѻ͏࡞඼͕ଟ༷Ͱ͋ͬͯ΋ɺ࡞඼ͷಛ௃Λଊ͑ΒΕΔख๏͕ඞཁ
    11
    ՝୊ᶄɿʮϋϯυϝΠυ࡞඼Λѻ͏ECαΠτʯͷ࡞඼͕ଟ༷
    

    View Slide

12. • BERTͰ͸ɺࣄલֶशࡁΈϞσϧΛར༻͢Δ͜ͱ͕Ͱ͖ɺ൚༻తͳ஌ࣝΛ࣋ͭ
    ݴޠϞσϧͷύϥϝʔλΛར༻Ͱ͖Δ
    
    
    • λεΫʹԠͨ͡গͳ͍ڭࢣ༗ΓσʔλͰ
    fi
    ne-tuning͢Δ͜ͱͰɺߴ͍ੑೳΛಘ
    Δ͜ͱ͕Մೳ
    
    
    • ͞ΒʹɺBERT௥ՃֶशʹΑΓɺଟ༷ͳ࡞඼ͷ஌ࣝ֫ಘ΋ظ଴Ͱ͖Δ
    
    
    → BERT(ࣄલֶशࡁΈ or ௥Ճֶश)+
    fi
    ne-tuningʹΑΓɺ
    

    ɹ࡞඼ͷଟ༷ੑΛଊ͑ΒΕΔͱߟ͑Δ
    12
    ՝୊ᶄʹର͢Δղܾࡦ
    

    View Slide

13. • ߏ଄తͳมԽͷྫɿΧςΰϦͷ৽ن࡞੒΍࡟আɺ౷ഇ߹ͳͲ
    
    
    • ࡞඼ͷ෼ྨΛߦ͏৔߹ɺ࡞඼จॻʹରͯ͠ɺจॻϕΫτϧԽख๏ɾલॲཧɾద
    ੾ͳ෼ྨثͷબఆ͕ඞཁʹͳͬͯ͘Δ
    
    
    

    → ߏ଄తͳมԽ͕ى͖ͨࡍɺ౎౓ߟྀ͠ͳ͚Ε͹ͳΒͣɺ௥ै͢Δͷ͸ࠔ೉
    13
    ՝୊ᶅɿʮϋϯυϝΠυ࡞඼Λѻ͏ECαΠτʯͷߏ଄తͳมԽ΁ͷ௥ै͕ࠔ೉
    จॻϕΫτϧԽख๏ લॲཧ ෼ྨثʹΑΔֶश
    ࠷దͳจॻϕΫτϧख๏͸Կʁ ࠷దͳલॲཧ͸Կʁ ࠷దͳ෼ྨث͸Կʁ
    ه߸Λআ͘ɺ୯ޠස౓͕ߴස
    ౓ɺ௿ස౓͸আ͘ͳͲ
    47. ϩδεςΟοΫճؼ
    

    ϥϯμϜϑΥϨετ ܾఆ໦ͳͲ
    #P8 5'*%' %PD7FD
    

    48&. 4$%7ɹͳͲ
    

    View Slide

14. • BERTͷ৔߹ɺ൚༻తͳϞσϧ͔Β
    fi
    ne-tuningͷΈΛ࣮ࢪ͢Δ͜ͱͰ௥ैՄ
    

    ྫʣΧςΰϦͷߏ଄͕มΘͬͨͱͯ͠΋ɺมΘͬͨΧςΰϦϥϕϧͷΈΛ
    fi
    ne-tuning͢Ε͹ରԠͰ͖Δ
    14
    ՝୊ᶅʹର͢Δղܾࡦ
    #&35ʢࣄલֶशࡁΈϞσϧʣPS#&35௥Ճֶश 'JOFUVOJOH
    จॻϕΫτϧԽख๏ લॲཧ ෼ྨثʹΑΔֶश
    ه߸Λআ͘ɺ୯ޠස౓͕ߴස
    ౓ɺ௿ස౓͸আ͘ͳͲ
    47. ϩδεςΟοΫճؼ
    

    ϥϯμϜϑΥϨετ ܾఆ໦ͳͲ
    #P8 5'*%' %PD7FD
    

    48&. 4$%7ɹͳͲ
    ϙΠϯτᶃɿࣄલʹ൚༻తͳϞσϧΛ࡞੒Ͱ͖Δ
    ϙΠϯτᶄɿߏ଄ͷมԽ͕ى͖ͨͱͯ͠΋ɺ
    fi
    OFUVOJOHͷΈͰ௥ैՄೳ
    

    View Slide

15. • ຊݚڀͰ͸ɺ࡞඼෼ྨʹ͓͍ͯϕʔεϥΠϯख๏ͱBERT+
    fi
    ne-tuningͷϞσϧ
    ΛൺֱɾධՁ͢Δɻ
    
    
    • ՝୊ͱղܾࡦٴͼޮՌͷԾઆΛҎԼʹࣔ͢
    
    
    15
    ຊݚڀͷ՝୊ͱղܾࡦͷରԠ
    ϕʔεϥΠϯख๏˞ #&35
    fi
    OFUVOJOH
    #&35௥Ճֶश
    fi
    OF
    UVOJOH
    ՝୊ᶃɿਓखͰ͸ࠔ೉ ˓ ˓ ˓
    ՝୊ᶄɿ࡞඼͕ଟ༷ ˚ ˓ ˕
    ՝୊ᶅɿมԽʹ௥ै º ˓ ˓
    ຌྫɹºɿѱ͍ɹ˓ɿྑ͍ɹ˕ɿ͔ͳΓྑ͍
    ˞ϕʔεϥΠϯख๏͸จॻϕΫτϧԽख๏ɿ5'*%'ɺ෼ྨثɿ47.ɺϩδεςΟοΫճؼ
    ຊݚڀͰऔΓ૊Ήൣғ
    

    View Slide

16. 3.
    ࣮ݧ
    

    View Slide

17. • ຊݚڀͷλεΫ͸ɺ֤࡞඼ʹ෇༩͞Ε͍ͯΔେ/খΧςΰϦΛ෼ྨ͢Δ͜ͱͱ
    ͨ͠ɻ࡞඼จॻ͸λΠτϧɺઆ໌จΛ࢖༻ͨ͠ɻ
    
    
    • จॻ෼ྨੑೳΛҎԼ3छྨͷख๏Ͱൺֱ
    
    
    • ϕʔεϥΠϯख๏ᶃɿ࡞඼จॻΛTF-IDFʹΑΔϕΫτϧԽɺ෼ྨثɿSVM
    
    
    • ϕʔεϥΠϯख๏ᶄɿ࡞඼จॻΛTF-IDFʹΑΔϕΫτϧԽɺ෼ྨثɿϩδε
    ςΟοΫճؼ
    
    
    • BERT+
    fi
    ne-tuningख๏ɿࣄલֶशࡁΈBERTΛ
    fi
    ne-tuning͢Δ
    17
    ࣮ݧ֓ཁ(1/2)
    

    View Slide

18. • σʔληοτ͸֤ΧςΰϦʹ͓͍ͯਫ਼౓ͷ֬อ͕े෼ɺ͔ͭɺܭࢉϦιʔε಺ͰֶशՄೳ
    ͳ݅਺ΛԾͱͯ͠10ສ݅ͱઃఆ
    

    → minneͷ࡞඼܈͔Β104,161݅Λநग़(2021೥6݄29೔࣌఺)
    
    
    • খΧςΰϦͷΫϥε਺͸239, େΧςΰϦͷΫϥε਺͸19
    
    
    • લॲཧͱͯ͠ɺׅހه߸ɺશۭ֯നɺURLΛ࡟আ
    
    
    • ֶशσʔλͱςετσʔλͷൺ཰͸9:1ͱͨ͠
    
    
    • ධՁࢦඪ͸Accuracy, Precision, Recall, F1-scoreΛ༻͍ͨ
    
    
    • ຊݚڀͰ͸ɺ෼ྨੑೳΛ૯߹తʹධՁ͢ΔͨΊɺF1-scoreΛॏࢹ͢Δ
    18
    ࣮ݧ֓ཁ(2/2)
    

    View Slide

19. • BERTͷࣄલֶशࡁΈϞσϧ͸ɺ౦๺େֶ͕ެ։͍ͯ͠Δ೔ຊޠֶशϞσϧ
    (Tohoku-BERTͱݺͿ)Λ༻͍ͨ[11]
    
    
    • Tokenizer͸Tohoku-BERT(MeCab<10>+NEologd<9>)Λ༻͍ɺ
    fi
    ne-tuningʹ͸
    BertForSequenceClassi
    fi
    cation<3>Λ࢖༻ͨ͠
    
    
    • ֶशύϥϝʔλ͸ɺଛࣦؔ਺͸ަࠩΤϯτϩϐʔޡࠩɺ࠷దԽؔ਺͸
    AdamW<5>ɺֶश཰͸2e-5
    19
    BERT+
    fi
    ne-tuning(1/2)
    [3] Huggingface, Transformers library, https://huggingface.co/transformers
    [5] Loshchilov, I. and Hutter, F.: Decoupled weight decay regularization, arXiv preprint arXiv:1711.05101 (2017).
    [9] Toshinori, S.: Neologism dictionary based on the language resources on the Web for Mecab (2015).
    [10] ޻౻୓ : MeCab, https://taku910.github.io/
    

    [11] ౦๺େֶެ։ͷ೔ຊޠࣄલֶशࡁΈBERT: https://github.com/cl-tohoku/bert-japanese
    

    View Slide

20. • จॻͷ࠷େ௕͕512Λ௒͑ͨ৔߹͸ɺઆ໌จͷઌ಄ͱ຤ඌΛ࢒͢ํࣜΛద༻[8]
    
    
    • fine-tuningͷֶशঢ়گΛ౿·͑ɺࠓճ͸epoch10·Ͱͱͨ͠ɻ෼ྨʹ͸F1-
    score͕Ұ൪ߴ͔ͬͨepochͷϞσϧΛ༻͍ͨ
    20
    BERT+
    fi
    ne-tuning(2/2)
    <8> Sun, C., Qiu, X., Xu, Y. and Huang, X.: How to fine-tune bert for text classification?, China National Conference on Chinese Computational Linguistics, Springer, pp. 194–206 (2019)
    ਤ1ɹখΧςΰϦ෼ྨλεΫʹ͓͚ΔBERT+fine-tuningͷֶशঢ়گ(ೖྗ͸λΠτϧ) ਤɹখΧςΰϦ෼ྨλεΫʹ͓͚ΔBERT+fine-tuningͷֶशঢ়گ(ೖྗ͸λΠτϧઆ໌จ)
    

    View Slide

21. • จॻϕΫτϧ࡞੒ʹ͓͍ͯɺϕʔεϥΠϯख๏ͱͯ͠ɺTF-IDFΛ༻͍ͨʢ඼ࢺ
    ͸໊ࢺͷΈʣ
    
    
    • ֤࡞඼จॻΛscikit-learnͷt
    fi
    dfvectorizerʹΑΓϕΫτϧԽ
    

    ೖྗ͕ʮλΠτϧ+આ໌จʯͷޠኮ਺ΛʮλΠτϧʯͱ߹Θͤͨ
    • 3෼ׂަࠩݕূͰ࠷దͳϋΠύʔύϥϝʔλΛٻΊͨޙɺςετσʔλΛධՁ
    
    
    21
    ϕʔεϥΠϯख๏ɿTF-IDF
    

    View Slide

22. 4.
    ࣮ݧ݁Ռͱߟ࡯
    

    View Slide

23. • ᶃ PrecisionҎ֎ɿ
    

    BERT+
    fi
    ne-tuning͕ྑ͍݁Ռ
    
    
    • ᶄ Precisionɿ
    

    TF-IDF, SVM͕ྑ͍݁Ռ
    • ೖྗจॻ͸ʮλΠτϧʯΑΓʮλ
    Πτϧ+આ໌จʯ͕ྑ͍݁Ռ
    

    → આ໌จ΋෼ྨੑೳ޲্ʹد༩
    23
    େΧςΰϦͷ෼ྨ݁Ռͱߟ࡯
    ᶃ
    ᶃ
    ᶄ
    ᶄ
    ˞֤ࢦඪʹ͓͍ͯҰ൪ੑೳ͕ߴ͍஋Λଠࣈͱͨ͠
    

    View Slide

24. • ᶃ PrecisionҎ֎ɿ
    

    BERT+
    fi
    ne-tuning͕ྑ͍݁Ռ
    
    
    • ᶄ Precisionɿ
    

    TF-IDF, SVM͕ྑ͍݁Ռ
    • େΧςΰϦͷ෼ྨ݁ՌΑΓ΋શମ
    తʹੑೳ͕௿͍
    

    → Ϋϥε਺͕ଟ͍ͨΊɺશମ͔Β
    খΧςΰϦΛ෼ྨ͢Δ͜ͱ͸ࠔ೉
    Ͱ͸ͳ͍͔ 24
    খΧςΰϦͷ෼ྨ݁Ռͱߟ࡯
    ᶃ
    ᶃ
    ᶄ
    ᶄ
    ˞֤ࢦඪʹ͓͍ͯҰ൪ੑೳ͕ߴ͍஋Λଠࣈͱͨ͠
    

    View Slide

25. • BERT+
    fi
    ne-tuningϞσϧͰͷʮλΠ
    τϧ+આ໌จʯͷ෼ྨޡΓ݁Ռ
    
    
    • ಉ͡େΧςΰϦ಺ͷখΧςΰϦΛਖ਼
    ͘͠෼ྨͰ͖ͳ͔ͬͨ
    
    
    • ࣮ࡍͷ࡞඼ςΩετΛ໨ࢹͰ֬ೝ
    
    
    25
    খΧςΰϦͷ෼ྨޡΓͷ݁Ռ
    ຌྫ
    খΧςΰϦ
    ʢେΧςΰϦʣ
    

    View Slide

26. • ਖ਼ղͷΧςΰϦࣗମ͕ਖ਼͘͠ͳ͍৔߹΋͋Γ͏Δ
    
    
    • ྫ͑͹ɺϔΞΞΫηαϦʔͷΧςΰϦ͕ઌʹ࡞੒͞ΕɺͦͷޙɺϔΞΰϜͷΧςΰϦ
    ͕࡞੒͞Εͨɺͱ͍͏ܦҢ͕͋ΔʢଞͷΧςΰϦͰ΋ಉ༷ͷมԽ͕ߟ͑ΒΕΔʣ
    
    
    • ݱঢ়ͷσʔλʹجͮ͘ΧςΰϦߏ଄ͷมԽΛݕग़Ͱ͖ΔՄೳੑ͕͋Δ
    26
    খΧςΰϦͷ෼ྨʹ͓͚Δ෼ྨޡΓͷߟ࡯ᶃ
    • ໨ࢹͰ֬ೝ
    

    ϔΞΞΫηαϦʔɿ7݅ɺϔΞΰϜɿ64݅
    

    ϔΞΞΫηαϦʔɿ0݅ɺόϨολɾϔΞΫϦοϓ
    ɿ47݅
    

    όοάɿ12݅ɺτʔτόοάɿ20݅
    
    
    

    View Slide

27. 27
    খΧςΰϦͷ෼ྨʹ͓͚Δ෼ྨޡΓͷߟ࡯ᶄ
    • ʮΠϠϦϯάɺϐΞεʯͲͪΒʹ΋౰ͯ͸·Δ࡞඼͕ଟ͍
    

    → Ұҙʹ෼ྨ͢Δඞཁ͕͋Δ͔Ͳ͏͔ɺݕ౼͕ඞཁ
    
    
    • ҰํɺϐΞεͷΈɾΠϠϦϯάͷΈɺͷ࡞඼͸ਖ਼͘͠෼ྨ͍ͨ͠
    • ໨ࢹͰ֬ೝ
    

    ΠϠϦϯάɿ2݅ɺϐΞεɿ13݅ɺ྆ํɿ55݅
    

    ϐΞεɿ2݅ɺΠϠϦϯάɿ10݅ɺ྆ํɿ22݅ɺ
    

    ɹͲͪΒͱ΋ݴ͑ͳ͍ɿ5݅
    

    View Slide

28. 28
    খΧςΰϦͷ෼ྨੑೳ޲্ʹ޲͚ͨࠓޙͷऔΓ૊Έ
    • େΧςΰϦΛ෼ྨޙʹɺখΧςΰϦΛ෼ྨ͢ΔϞσϧΛ࡞Δ
    
    
    • ϝϦοτɹɿখΧςΰϦͷ෼ྨੑೳͷ޲্͕ݟࠐΊΔ
    
    
    • σϝϦοτɿେΧςΰϦ୯ҐͰͷϞσϧߏங͕ඞཁ
    
    
    ɾɾ
    STEP1:େΧςΰϦΛ෼ྨ
    ˙ࠓޙͷऔΓ૊Έ
    ˙ݱঢ়
    STEP2:֤େΧςΰϦ಺͔ΒখΧςΰϦΛ෼ྨ
    ௚઀ɺখΧςΰϦΛ෼ྨ͍ͯ͠Δ
    ΞΫηαϦʔ ϑΝογϣϯ ϚεΫ ͓΋ͪΌ
    ΞΫηαϦʔ
    ࢦྠɾϦϯά ϐΞε
    ɾɾɾ
    ɾɾɾ
    খ
    খ খ খ খ
    େ
    

    View Slide

29. 5.
    ·ͱΊͱࠓޙͷ՝୊
    

    View Slide

30. • ʮϋϯυϝΠυ࡞඼Λѻ͏ECαΠτʯͷ՝୊ΛؚΉλεΫͷ಺ɺ

    ࡞඼ͷΧςΰϦ෼ྨͷλεΫʹ͓͍ͯ෼ྨੑೳΛൺֱͨ͠
    
    
    • ݁Ռͱͯ͠ɺBERT+
    fi
    ne-tuning͕PrecisionҎ֎ͷධՁࢦඪͰྑ͍݁Ռ
    

    → ࡞඼ͷଟ༷ੑΛҰఆఔ౓͸ଊ͑Δ͜ͱ͕Ͱ͖ͨͷͰ͸ͳ͍͔
    

    30
    ·ͱΊ
    

    View Slide

31. • BERT௥ՃֶशʹΑΔίʔύεʹಛԽͨ͠஌ࣝͷڧԽ
    
    
    • খΧςΰϦͷ෼ྨʹରͯ͠͸ɺ2ஈ֊ʢେΧςΰϦˠখΧςΰϦʣͰ෼ྨ
    
    
    • ධՁํ๏ͷݕ౼
    
    
    • ਖ਼ղͷΧςΰϦ͕ਖ਼͘͠ͳ͍ՄೳੑΛ౿·͑ͨධՁͷݕ౼
    
    
    • ʮΠϠϦϯάorϐΞεʯͷΑ͏ʹɺҰҙʹ෼ྨ͢Δ͔Ͳ͏͔ͷݕ౼
    
    
    •
    fi
    ne-tuningʹ༻͍ΔσʔλྔΛ૿΍͢
    31
    ࠓޙͷ՝୊
    

    View Slide

32. View Slide