Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ハンドメイド作品を扱うECサイトに特化したBERTを用いた言語モデル構築に向けた取り組み/ipsj-NL250-05

tossy
September 28, 2021

 ハンドメイド作品を扱うECサイトに特化したBERTを用いた言語モデル構築に向けた取り組み/ipsj-NL250-05

2021.9.28 情報処理学会 第250回自然言語処理研究会(NL研)
https://nl-ipsj.or.jp/2021/09/10/nl250-program/

tossy

September 28, 2021
Tweet

More Decks by tossy

Other Decks in Research

Transcript

  1. ञҪහ඙(GMOϖύϘגࣜձࣾ ϖύϘݚڀॴ/۝भେֶେֶӃ),

    ࡾ୐༔հ(GMOϖύϘגࣜձࣾ ϖύϘݚڀॴ), ܀ྛ݈ଠ࿠(GMOϖύϘגࣜձࣾ ϖύϘݚڀॴ)
    ൃදऀɿञҪහ඙ / Pepabo R&D Institute, GMO Pepabo, Inc. / Kyushu University


    2021.09.28 ৘ใॲཧֶձ ୈ250ճࣗવݴޠॲཧݚڀձ
    ϋϯυϝΠυ࡞඼Λѻ͏ECαΠτʹಛԽͨ͠
    BERTΛ༻͍ͨݴޠϞσϧߏஙʹ޲͚ͨऔΓ૊Έ

    View full-size slide

  2. 1. ݚڀͷഎܠ


    2. ՝୊ͱղܾࡦ


    3. ࣮ݧ


    4. ࣮ݧ݁Ռͱߟ࡯


    5. ·ͱΊͱࠓޙͷ՝୊
    2
    ໨࣍

    View full-size slide

  3. 1.
    ݚڀͷഎܠ

    View full-size slide

  4. • ฐࣾͰ͸ɺϋϯυϝΠυ࡞඼Λର৅ͱͨ͠CtoCͷECαΠτ(ʮϋϯυϝΠυ࡞
    ඼Λѻ͏ECαΠτʯͱݺͿ)Ͱ͋ΔminneΛӡӦ


    4
    ݚڀͷഎܠ(1/3)
    ˞NJOOF࡞Ոొ࿥ϚχϡΞϧΑΓ
    ࡞඼Λൢച ࡞඼Λߪೖ

    View full-size slide

  5. • ֤࡞඼ʹ͸ɺλΠτϧ΍આ໌จͳͲ
    ͷςΩετ৘ใ͕෇༩͞Ε͍ͯΔ


    • ֤छλεΫʹར༻͍ͨ͠


    • ࡞඼ͷΧςΰϦ෼ྨ


    • λΠτϧɾઆ໌จͷࣗಈੜ੒


    • ࣭໰Ԡ౴


    ͳͲ
    5
    ݚڀͷഎܠ(2/3)
    େখΧςΰϦ
    ࡞඼λΠτϧ
    ࡞඼આ໌จ
    ˙࡞඼ͷྫ

    View full-size slide

  6. • ͜ΕΒͷλεΫΛਓखͰղ͘ͷ͸ࠔ೉ͳͨΊػցతͳΞϓϩʔν͕ඞཁͱͳΔ


    • ҰํɺʮϋϯυϝΠυ࡞඼Λѻ͏ECαΠτʯ͸ҎԼͷΑ͏ͳಛ௃Λ࣋ͭ


    • ࡞඼͕ଟ༷Ͱ͋ΔͨΊɺ֤࡞඼Λਖ਼͘͠ಛ௃͚ͮΔ͜ͱ͕ࠔ೉


    • αʔϏεͷมԽʹԠͨ͡ߏ଄తͳมԽʹ௥ै͢Δ͜ͱ͸ࠔ೉


    → ্هͷಛ௃Λଊ͑ͯɺ໰୊ΛղܾͰ͖Δख๏͕ඞཁ
    6
    ݚڀͷഎܠ(3/3)

    View full-size slide

  7. • ࣗવݴޠॲཧͰ͸ɺBERT[1]Λ࢖͏͜ͱ͕ओྲྀʹͳ͖͍ͬͯͯΔ


    • BERTͰ͸ɺࣄલֶशࡁΈϞσϧΛ࢖͏͜ͱͰɺ൚༻తͳ஌ࣝΛ࣋ͭݴޠϞσ
    ϧΛར༻͢Δ͜ͱ͕Ͱ͖Δ


    • ࣄલֶशࡁΈϞσϧΛɺλεΫʹԠͨ͡গͳ͍ڭࢣ༗ΓσʔλͰ
    fi
    ne-tuning͢
    Δ͜ͱͰɺߴ͍ੑೳΛಘΔ͜ͱ͕Մೳ(ຊݚڀͰ͸ɺBERT+
    fi
    ne-tuningͱݺͿ)


    • ௥ՃֶशʢBERTͷࣄલֶशࡁΈϞσϧ͔Βେن໛ͳίʔύεͰ࠶ࣄલֶशʣ
    Λߦ͏͜ͱͰɺίʔύεʹಛԽͨ͠஌ࣝ֫ಘ͕Մೳ [4],[6]
    7
    BERT
    [1] %FWMJO + $IBOH .8 -FF ,BOE5PVUBOPWB ,#&351SFUSBJOJOHPG%FFQ#JEJSFDUJPOBM5SBOTGPSNFSTGPS-BOHVBHF6OEFSTUBOEJOH

    <>-FF + :PPO 8 ,JN 4 ,JN % ,JN 4 4P $)BOE,BOH +#JP#&35BQSFUSBJOFECJPNFEJDBMMBOHVBHFSFQSFTFOUBUJPONPEFMGPSCJPNFEJDBMUFYUNJOJOH #JPJOGPSNBUJDT 7PM /P QQr

    <>/55
    デ
    ʔλۚ༥ۀք޲͚ࣗવݴޠॲཧٕज़ͷݕূ։࢝dۚ༥൛#&35Ϟ
    デ
    ϧͷ։ൃdɼIUUQTXXXOUUEBUBDPNKQKBOFXTSFMFBTF

    View full-size slide

  8. • ʮϋϯυϝΠυ࡞඼Λѻ͏ECαΠτʯͰͷࣗવݴޠॲཧͷλεΫΛղ͖͍ͨ


    → BERT+
    fi
    ne-tuningΛ࢖͏͜ͱͰɺ֤՝୊ΛղܾͰ͖ΔͷͰ͸ͳ͍͔


    • ࣗવݴޠॲཧͷλεΫͷ಺ɺࠓճ͸ɺ࡞඼ͷΧςΰϦ෼ྨʹऔΓ૊Ή


    • ຊݚڀͰ͸ɺ࡞඼ͷΧςΰϦ෼ྨʹ͓͍ͯϕʔεϥΠϯख๏ͱBERT+
    fi
    ne-
    tuningͷϞσϧΛൺֱɾධՁ͢Δɻ݁ՌΛ౿·͑ͯɺࠓޙͷํ޲ੑΛܾΊ͍ͨ
    8
    ຊݚڀͷશମ૾
    ˞ϕʔεϥΠϯख๏͸จॻϕΫτϧԽख๏ɿ5'*%'ɺ෼ྨثɿ47.ɺϩδεςΟοΫճؼͱͨ͠

    View full-size slide

  9. 2.
    ՝୊ͱղܾࡦ

    View full-size slide

  10. • ࡞඼Λग़඼͢Δࡍʹ͸ɺ࡞Ո͕࡞඼ͷେ/খΧςΰϦΛొ࿥͍ͯ͠Δ






    → ొ࿥͢ΔࡍʹɺࣗಈͰେ/খΧςΰϦΛਪଌ͠ɺ࡞Ո΁ఏ͍ࣔͨ͠

    ɹɹࣄલʹΧςΰϦΛఏࣔ͢Δ͜ͱͰɺΧςΰϦઃఆޡΓΛ๷ࢭ͍ͨ͠


    • ͔͠͠ɺ࡞඼ͷΧςΰϦ෼ྨΛӡӦଆ͕ਓखͰߦ͏͜ͱ͸ݱ࣮తͰͳ͍

    → ՝୊ᶃʹର͢Δղܾࡦɿػցతʹ෼ྨ͢Δ͜ͱ͕ඞཁ
    10
    ՝୊ᶃɿਓखͰλεΫΛղ͘ͷ͸ࠔ೉
    ˙େΧςΰϦͷҰཡ
    ΞΫηαϦʔ
    ࢦྠɾϦϯά ϐΞε
    ɾɾɾ
    ɾɾɾ
    ˙େখΧςΰϦͷྫ

    View full-size slide

  11. • ʮϋϯυϝΠυ࡞඼Λѻ͏ECαΠτʯͰ͸࡞Ո͕࡞ͬͨ࡞඼Λߪೖऀ͕ߪೖ

    → ࡞Ո͕ҟͳΔ৔߹ɺಉ͡࡞඼͸ଘࡏ͠ͳ͍

    ಉ͡࡞඼͕ଘࡏ͠ͳ͍͜ͱ͕େ͖ͳಛ௃Ͱ͋Δ


    • औΓѻ͏࡞඼ͷछྨ͕ଟ༷ͱͳΓɺ࡞඼Λద੾ͳΧςΰϦ΁෼ྨ͢ΔͨΊʹ͸
    ෯޿͍࡞඼஌͕ࣝඞཁͱͳΔ

    → ѻ͏࡞඼͕ଟ༷Ͱ͋ͬͯ΋ɺ࡞඼ͷಛ௃Λଊ͑ΒΕΔख๏͕ඞཁ
    11
    ՝୊ᶄɿʮϋϯυϝΠυ࡞඼Λѻ͏ECαΠτʯͷ࡞඼͕ଟ༷

    View full-size slide

  12. • BERTͰ͸ɺࣄલֶशࡁΈϞσϧΛར༻͢Δ͜ͱ͕Ͱ͖ɺ൚༻తͳ஌ࣝΛ࣋ͭ
    ݴޠϞσϧͷύϥϝʔλΛར༻Ͱ͖Δ


    • λεΫʹԠͨ͡গͳ͍ڭࢣ༗ΓσʔλͰ
    fi
    ne-tuning͢Δ͜ͱͰɺߴ͍ੑೳΛಘ
    Δ͜ͱ͕Մೳ


    • ͞ΒʹɺBERT௥ՃֶशʹΑΓɺଟ༷ͳ࡞඼ͷ஌ࣝ֫ಘ΋ظ଴Ͱ͖Δ


    → BERT(ࣄલֶशࡁΈ or ௥Ճֶश)+
    fi
    ne-tuningʹΑΓɺ

    ɹ࡞඼ͷଟ༷ੑΛଊ͑ΒΕΔͱߟ͑Δ
    12
    ՝୊ᶄʹର͢Δղܾࡦ

    View full-size slide

  13. • ߏ଄తͳมԽͷྫɿΧςΰϦͷ৽ن࡞੒΍࡟আɺ౷ഇ߹ͳͲ


    • ࡞඼ͷ෼ྨΛߦ͏৔߹ɺ࡞඼จॻʹରͯ͠ɺจॻϕΫτϧԽख๏ɾલॲཧɾద
    ੾ͳ෼ྨثͷબఆ͕ඞཁʹͳͬͯ͘Δ



    → ߏ଄తͳมԽ͕ى͖ͨࡍɺ౎౓ߟྀ͠ͳ͚Ε͹ͳΒͣɺ௥ै͢Δͷ͸ࠔ೉
    13
    ՝୊ᶅɿʮϋϯυϝΠυ࡞඼Λѻ͏ECαΠτʯͷߏ଄తͳมԽ΁ͷ௥ै͕ࠔ೉
    จॻϕΫτϧԽख๏ લॲཧ ෼ྨثʹΑΔֶश
    ࠷దͳจॻϕΫτϧख๏͸Կʁ ࠷దͳલॲཧ͸Կʁ ࠷దͳ෼ྨث͸Կʁ
    ه߸Λআ͘ɺ୯ޠස౓͕ߴස
    ౓ɺ௿ස౓͸আ͘ͳͲ
    47. ϩδεςΟοΫճؼ

    ϥϯμϜϑΥϨετ ܾఆ໦ͳͲ
    #P8 5'*%' %PD7FD

    48&. 4$%7ɹͳͲ

    View full-size slide

  14. • BERTͷ৔߹ɺ൚༻తͳϞσϧ͔Β
    fi
    ne-tuningͷΈΛ࣮ࢪ͢Δ͜ͱͰ௥ैՄ

    ྫʣΧςΰϦͷߏ଄͕มΘͬͨͱͯ͠΋ɺมΘͬͨΧςΰϦϥϕϧͷΈΛ
    fi
    ne-tuning͢Ε͹ରԠͰ͖Δ
    14
    ՝୊ᶅʹର͢Δղܾࡦ
    #&35ʢࣄલֶशࡁΈϞσϧʣPS#&35௥Ճֶश 'JOFUVOJOH
    จॻϕΫτϧԽख๏ લॲཧ ෼ྨثʹΑΔֶश
    ه߸Λআ͘ɺ୯ޠස౓͕ߴස
    ౓ɺ௿ස౓͸আ͘ͳͲ
    47. ϩδεςΟοΫճؼ

    ϥϯμϜϑΥϨετ ܾఆ໦ͳͲ
    #P8 5'*%' %PD7FD

    48&. 4$%7ɹͳͲ
    ϙΠϯτᶃɿࣄલʹ൚༻తͳϞσϧΛ࡞੒Ͱ͖Δ
    ϙΠϯτᶄɿߏ଄ͷมԽ͕ى͖ͨͱͯ͠΋ɺ
    fi
    OFUVOJOHͷΈͰ௥ैՄೳ

    View full-size slide

  15. • ຊݚڀͰ͸ɺ࡞඼෼ྨʹ͓͍ͯϕʔεϥΠϯख๏ͱBERT+
    fi
    ne-tuningͷϞσϧ
    ΛൺֱɾධՁ͢Δɻ


    • ՝୊ͱղܾࡦٴͼޮՌͷԾઆΛҎԼʹࣔ͢


    15
    ຊݚڀͷ՝୊ͱղܾࡦͷରԠ
    ϕʔεϥΠϯख๏˞ #&35
    fi
    OFUVOJOH
    #&35௥Ճֶश
    fi
    OF
    UVOJOH
    ՝୊ᶃɿਓखͰ͸ࠔ೉ ˓ ˓ ˓
    ՝୊ᶄɿ࡞඼͕ଟ༷ ˚ ˓ ˕
    ՝୊ᶅɿมԽʹ௥ै º ˓ ˓
    ຌྫɹºɿѱ͍ɹ˓ɿྑ͍ɹ˕ɿ͔ͳΓྑ͍
    ˞ϕʔεϥΠϯख๏͸จॻϕΫτϧԽख๏ɿ5'*%'ɺ෼ྨثɿ47.ɺϩδεςΟοΫճؼ
    ຊݚڀͰऔΓ૊Ήൣғ

    View full-size slide

  16. • ຊݚڀͷλεΫ͸ɺ֤࡞඼ʹ෇༩͞Ε͍ͯΔେ/খΧςΰϦΛ෼ྨ͢Δ͜ͱͱ
    ͨ͠ɻ࡞඼จॻ͸λΠτϧɺઆ໌จΛ࢖༻ͨ͠ɻ


    • จॻ෼ྨੑೳΛҎԼ3छྨͷख๏Ͱൺֱ


    • ϕʔεϥΠϯख๏ᶃɿ࡞඼จॻΛTF-IDFʹΑΔϕΫτϧԽɺ෼ྨثɿSVM


    • ϕʔεϥΠϯख๏ᶄɿ࡞඼จॻΛTF-IDFʹΑΔϕΫτϧԽɺ෼ྨثɿϩδε
    ςΟοΫճؼ


    • BERT+
    fi
    ne-tuningख๏ɿࣄલֶशࡁΈBERTΛ
    fi
    ne-tuning͢Δ
    17
    ࣮ݧ֓ཁ(1/2)

    View full-size slide

  17. • σʔληοτ͸֤ΧςΰϦʹ͓͍ͯਫ਼౓ͷ֬อ͕े෼ɺ͔ͭɺܭࢉϦιʔε಺ͰֶशՄೳ
    ͳ݅਺ΛԾͱͯ͠10ສ݅ͱઃఆ

    → minneͷ࡞඼܈͔Β104,161݅Λநग़(2021೥6݄29೔࣌఺)


    • খΧςΰϦͷΫϥε਺͸239, େΧςΰϦͷΫϥε਺͸19


    • લॲཧͱͯ͠ɺׅހه߸ɺશۭ֯നɺURLΛ࡟আ


    • ֶशσʔλͱςετσʔλͷൺ཰͸9:1ͱͨ͠


    • ධՁࢦඪ͸Accuracy, Precision, Recall, F1-scoreΛ༻͍ͨ


    • ຊݚڀͰ͸ɺ෼ྨੑೳΛ૯߹తʹධՁ͢ΔͨΊɺF1-scoreΛॏࢹ͢Δ
    18
    ࣮ݧ֓ཁ(2/2)

    View full-size slide

  18. • BERTͷࣄલֶशࡁΈϞσϧ͸ɺ౦๺େֶ͕ެ։͍ͯ͠Δ೔ຊޠֶशϞσϧ
    (Tohoku-BERTͱݺͿ)Λ༻͍ͨ[11]


    • Tokenizer͸Tohoku-BERT(MeCab<10>+NEologd<9>)Λ༻͍ɺ
    fi
    ne-tuningʹ͸
    BertForSequenceClassi
    fi
    cation<3>Λ࢖༻ͨ͠


    • ֶशύϥϝʔλ͸ɺଛࣦؔ਺͸ަࠩΤϯτϩϐʔޡࠩɺ࠷దԽؔ਺͸
    AdamW<5>ɺֶश཰͸2e-5
    19
    BERT+
    fi
    ne-tuning(1/2)
    [3] Huggingface, Transformers library, https://huggingface.co/transformers
    [5] Loshchilov, I. and Hutter, F.: Decoupled weight decay regularization, arXiv preprint arXiv:1711.05101 (2017).
    [9] Toshinori, S.: Neologism dictionary based on the language resources on the Web for Mecab (2015).
    [10] ޻౻୓ : MeCab, https://taku910.github.io/

    [11] ౦๺େֶެ։ͷ೔ຊޠࣄલֶशࡁΈBERT: https://github.com/cl-tohoku/bert-japanese

    View full-size slide

  19. • จॻͷ࠷େ௕͕512Λ௒͑ͨ৔߹͸ɺઆ໌จͷઌ಄ͱ຤ඌΛ࢒͢ํࣜΛద༻[8]


    • fine-tuningͷֶशঢ়گΛ౿·͑ɺࠓճ͸epoch10·Ͱͱͨ͠ɻ෼ྨʹ͸F1-
    score͕Ұ൪ߴ͔ͬͨepochͷϞσϧΛ༻͍ͨ
    20
    BERT+
    fi
    ne-tuning(2/2)
    <8> Sun, C., Qiu, X., Xu, Y. and Huang, X.: How to fine-tune bert for text classification?, China National Conference on Chinese Computational Linguistics, Springer, pp. 194–206 (2019)
    ਤ1ɹখΧςΰϦ෼ྨλεΫʹ͓͚ΔBERT+fine-tuningͷֶशঢ়گ(ೖྗ͸λΠτϧ) ਤɹখΧςΰϦ෼ྨλεΫʹ͓͚ΔBERT+fine-tuningͷֶशঢ়گ(ೖྗ͸λΠτϧઆ໌จ)

    View full-size slide

  20. • จॻϕΫτϧ࡞੒ʹ͓͍ͯɺϕʔεϥΠϯख๏ͱͯ͠ɺTF-IDFΛ༻͍ͨʢ඼ࢺ
    ͸໊ࢺͷΈʣ


    • ֤࡞඼จॻΛscikit-learnͷt
    fi
    dfvectorizerʹΑΓϕΫτϧԽ

    ೖྗ͕ʮλΠτϧ+આ໌จʯͷޠኮ਺ΛʮλΠτϧʯͱ߹Θͤͨ
    • 3෼ׂަࠩݕূͰ࠷దͳϋΠύʔύϥϝʔλΛٻΊͨޙɺςετσʔλΛධՁ


    21
    ϕʔεϥΠϯख๏ɿTF-IDF

    View full-size slide

  21. 4.
    ࣮ݧ݁Ռͱߟ࡯

    View full-size slide

  22. • ᶃ PrecisionҎ֎ɿ

    BERT+
    fi
    ne-tuning͕ྑ͍݁Ռ


    • ᶄ Precisionɿ

    TF-IDF, SVM͕ྑ͍݁Ռ
    • ೖྗจॻ͸ʮλΠτϧʯΑΓʮλ
    Πτϧ+આ໌จʯ͕ྑ͍݁Ռ

    → આ໌จ΋෼ྨੑೳ޲্ʹد༩
    23
    େΧςΰϦͷ෼ྨ݁Ռͱߟ࡯




    ˞֤ࢦඪʹ͓͍ͯҰ൪ੑೳ͕ߴ͍஋Λଠࣈͱͨ͠

    View full-size slide

  23. • ᶃ PrecisionҎ֎ɿ

    BERT+
    fi
    ne-tuning͕ྑ͍݁Ռ


    • ᶄ Precisionɿ

    TF-IDF, SVM͕ྑ͍݁Ռ
    • େΧςΰϦͷ෼ྨ݁ՌΑΓ΋શମ
    తʹੑೳ͕௿͍

    → Ϋϥε਺͕ଟ͍ͨΊɺશମ͔Β
    খΧςΰϦΛ෼ྨ͢Δ͜ͱ͸ࠔ೉
    Ͱ͸ͳ͍͔ 24
    খΧςΰϦͷ෼ྨ݁Ռͱߟ࡯




    ˞֤ࢦඪʹ͓͍ͯҰ൪ੑೳ͕ߴ͍஋Λଠࣈͱͨ͠

    View full-size slide

  24. • BERT+
    fi
    ne-tuningϞσϧͰͷʮλΠ
    τϧ+આ໌จʯͷ෼ྨޡΓ݁Ռ


    • ಉ͡େΧςΰϦ಺ͷখΧςΰϦΛਖ਼
    ͘͠෼ྨͰ͖ͳ͔ͬͨ


    • ࣮ࡍͷ࡞඼ςΩετΛ໨ࢹͰ֬ೝ


    25
    খΧςΰϦͷ෼ྨޡΓͷ݁Ռ
    ຌྫ
    খΧςΰϦ
    ʢେΧςΰϦʣ

    View full-size slide

  25. • ਖ਼ղͷΧςΰϦࣗମ͕ਖ਼͘͠ͳ͍৔߹΋͋Γ͏Δ


    • ྫ͑͹ɺϔΞΞΫηαϦʔͷΧςΰϦ͕ઌʹ࡞੒͞ΕɺͦͷޙɺϔΞΰϜͷΧςΰϦ
    ͕࡞੒͞Εͨɺͱ͍͏ܦҢ͕͋ΔʢଞͷΧςΰϦͰ΋ಉ༷ͷมԽ͕ߟ͑ΒΕΔʣ


    • ݱঢ়ͷσʔλʹجͮ͘ΧςΰϦߏ଄ͷมԽΛݕग़Ͱ͖ΔՄೳੑ͕͋Δ
    26
    খΧςΰϦͷ෼ྨʹ͓͚Δ෼ྨޡΓͷߟ࡯ᶃ
    • ໨ࢹͰ֬ೝ

    ϔΞΞΫηαϦʔɿ7݅ɺϔΞΰϜɿ64݅

    ϔΞΞΫηαϦʔɿ0݅ɺόϨολɾϔΞΫϦοϓ
    ɿ47݅

    όοάɿ12݅ɺτʔτόοάɿ20݅


    View full-size slide

  26. 27
    খΧςΰϦͷ෼ྨʹ͓͚Δ෼ྨޡΓͷߟ࡯ᶄ
    • ʮΠϠϦϯάɺϐΞεʯͲͪΒʹ΋౰ͯ͸·Δ࡞඼͕ଟ͍

    → Ұҙʹ෼ྨ͢Δඞཁ͕͋Δ͔Ͳ͏͔ɺݕ౼͕ඞཁ


    • ҰํɺϐΞεͷΈɾΠϠϦϯάͷΈɺͷ࡞඼͸ਖ਼͘͠෼ྨ͍ͨ͠
    • ໨ࢹͰ֬ೝ

    ΠϠϦϯάɿ2݅ɺϐΞεɿ13݅ɺ྆ํɿ55݅

    ϐΞεɿ2݅ɺΠϠϦϯάɿ10݅ɺ྆ํɿ22݅ɺ

    ɹͲͪΒͱ΋ݴ͑ͳ͍ɿ5݅

    View full-size slide

  27. 28
    খΧςΰϦͷ෼ྨੑೳ޲্ʹ޲͚ͨࠓޙͷऔΓ૊Έ
    • େΧςΰϦΛ෼ྨޙʹɺখΧςΰϦΛ෼ྨ͢ΔϞσϧΛ࡞Δ


    • ϝϦοτɹɿখΧςΰϦͷ෼ྨੑೳͷ޲্͕ݟࠐΊΔ


    • σϝϦοτɿେΧςΰϦ୯ҐͰͷϞσϧߏங͕ඞཁ


    ɾɾ
    STEP1:େΧςΰϦΛ෼ྨ
    ˙ࠓޙͷऔΓ૊Έ
    ˙ݱঢ়
    STEP2:֤େΧςΰϦ಺͔ΒখΧςΰϦΛ෼ྨ
    ௚઀ɺখΧςΰϦΛ෼ྨ͍ͯ͠Δ
    ΞΫηαϦʔ ϑΝογϣϯ ϚεΫ ͓΋ͪΌ
    ΞΫηαϦʔ
    ࢦྠɾϦϯά ϐΞε
    ɾɾɾ
    ɾɾɾ

    খ খ খ খ

    View full-size slide

  28. 5.
    ·ͱΊͱࠓޙͷ՝୊

    View full-size slide

  29. • ʮϋϯυϝΠυ࡞඼Λѻ͏ECαΠτʯͷ՝୊ΛؚΉλεΫͷ಺ɺ

    ࡞඼ͷΧςΰϦ෼ྨͷλεΫʹ͓͍ͯ෼ྨੑೳΛൺֱͨ͠


    • ݁Ռͱͯ͠ɺBERT+
    fi
    ne-tuning͕PrecisionҎ֎ͷධՁࢦඪͰྑ͍݁Ռ

    → ࡞඼ͷଟ༷ੑΛҰఆఔ౓͸ଊ͑Δ͜ͱ͕Ͱ͖ͨͷͰ͸ͳ͍͔

    30
    ·ͱΊ

    View full-size slide

  30. • BERT௥ՃֶशʹΑΔίʔύεʹಛԽͨ͠஌ࣝͷڧԽ


    • খΧςΰϦͷ෼ྨʹରͯ͠͸ɺ2ஈ֊ʢେΧςΰϦˠখΧςΰϦʣͰ෼ྨ


    • ධՁํ๏ͷݕ౼


    • ਖ਼ղͷΧςΰϦ͕ਖ਼͘͠ͳ͍ՄೳੑΛ౿·͑ͨධՁͷݕ౼


    • ʮΠϠϦϯάorϐΞεʯͷΑ͏ʹɺҰҙʹ෼ྨ͢Δ͔Ͳ͏͔ͷݕ౼



    fi
    ne-tuningʹ༻͍ΔσʔλྔΛ૿΍͢
    31
    ࠓޙͷ՝୊

    View full-size slide