$30 off During Our Annual Pro Sale. View Details »

Feedback Prize - English Language Learning 
における擬似ラベルの品質向上の取り組み

Shuhei Goda
November 08, 2023

Feedback Prize - English Language Learning 
における擬似ラベルの品質向上の取り組み

第4回 Data-Centric AI勉強会 -コンペLT大会-
https://dcai-jp.connpass.com/event/298953/

Shuhei Goda

November 08, 2023
Tweet

More Decks by Shuhei Goda

Other Decks in Technology

Transcript

  1. © 2023 Wantedly, Inc.
    Feedback Prize - English Language Learning 

    ʹ͓͚Δٖࣅϥϕϧͷ඼࣭޲্ͷऔΓ૊Έ
    ୈ4ճ Data-Centric AIษڧձ -ίϯϖLTେձ-
    Nov. 8 2023 - Shuhei Goda

    View Slide

  2. © 2023 Wantedly, Inc.
    ໊લɿ
    ߹ా पฏ(Shuhei Goda)
    ॴଐͱ໾ׂɿ
    ΢ΥϯςουϦʔגࣜձࣾ
    σʔλαΠΤϯςΟετ
    ࣗݾ঺հ
    @hakubishin3
    @jy_msc
    @shuheigoda

    View Slide

  3. © 2023 Wantedly, Inc.
    2023೥8݄ʙ11݄Ͱ։࠵͞Ε͍ͯͨ NLP ίϯϖ
    Feedback Prize - English Language Learning
    https://www.kaggle.com/competitions/feedback-prize-english-language-learning

    View Slide

  4. © 2023 Wantedly, Inc.
    ӳޠখ࿦จͷ6छྨͷ඼࣭༧ଌΛධՁ
    ● ֤ई౓ͷൣғ͸1.0~5.0(0.5ࠁΈ)


    ● ֶशσʔλ਺ 3,911 ݅
    Competition Data & Evaluation
    I believe that home-based
    learning could be
    advantageous for students as
    it eliminates the need for
    them to dress up and get
    ready...
    essay
    measure predicted value ground truth
    cohesion 2.53 3.5
    syntax 1.12 1.0
    vocabulary 3.25 5.0
    phraseology 2.12 2.5
    grammar 4.90 1.0
    conventions 2.12 3.0
    Scoring: MCRMSE


    (mean columnwise RMSE)

    View Slide

  5. © 2023 Wantedly, Inc.
    ӳޠখ࿦จͷࣗಈධՁʹয఺Λ౰ͯͨγϦʔζ
    1. Feedback Prize - Evaluating Student Writing (FB1 ͱݺশ)
    2. Feedback Prize - Predicting Effective Arguments (FB2 ͱݺশ)
    3. Feedback Prize - English Language Learning (FB3 ͱݺশ)
    FB3Ͱ͸

    աڈͷίϯϖ(FB1, FB2)ͷσʔλ΋ར༻Մೳ
    Feedback Prize Competitions Series
    FB1 essays
    FB3 essays
    15,594 ݅
    3,911 ݅
    ॏෳ 452 ݅

    View Slide

  6. © 2023 Wantedly, Inc.
    ίϯϖΛऔΓ૊Ή্Ͱͷઓུ
    ● ͪΐͬͱ৮ͬͯΈͯײͨ͜͡ͱ


    ○ σʔλ΍໰୊ઃఆ͸ඇৗʹγϯϓϧɺσʔλྔ΋গͳ͍


    ○ Magic(͍͢͝ͻΒΊ͖)Ͱ͕ࠩͭ͘Α͏ͳ΋ͷͰ͸ͳͦ͞͏


    ● ํ਑ΛཱͯΔ


    ○ ౰ͨΓલͷ͜ͱΛ΍ΕΔ͚ͩ΍Δ


    ○ ֶशʹར༻͢Δσʔλͷ࣭ͱྔͰࠩΛ͚ͭΔ


    ■ FB3σʔλ3,911݅ͱॏෳΛআ͍ͨFB1σʔλ15,142݅ΛͲ͏࢖͏͔
    My Strategy

    View Slide

  7. © 2023 Wantedly, Inc.
    ࣭ͷྑ͍ Pseudo Label(FB1σʔλ) ΛՃֶ͑ͯशͤ͞Δ
    ● Pseudo Label ͷ࣭Λ্͛ΔͨΊͷऔΓ૊Έ


    a. Adversarial Validation ʹΑΔϊΠδʔͳσʔλͷআ֎


    b. FB3 σʔληοτͷ All Data Training


    c. Pseudo Labeling ͷύλʔϯͷόϦΤʔγϣϯ


    d. Pseudo Labeling ͷΠλϨʔγϣϯճ਺


    e. ࣄલֶशࡁΈϞσϧͷόϦΤʔγϣϯ
    Key Point

    View Slide

  8. © 2023 Wantedly, Inc.
    Adversarial Validation ʹΑΔϊΠδʔͳσʔλͷআ֎
    ● FB1σʔληοτʹؚ·ΕΔશͯͷσʔλ͕FB3ͱಉ༷ͷ܏޲ΛऔΔͱ͸ݶΒͳ͍


    ○ ࢦఆͨ͠ essay ͕ FB3 ʹؚ·ΕΔ͔Ͳ͏͔Λ༧ଌ͢ΔλεΫΛ࣮ࢪ


    ○ ͦͷ༧ଌ஋͕͖͍͠஋Ҏ্ͱͳΔFB1σʔλ͚ͩΛֶशʹ࢖͏
    1. Adversarial Validation

    Pseudo Label ෇͖
    ͷFB1σʔλ
    FB3σʔλ
    ͖͍͠஋
    FB3Ά͍
    FB3Ά͘ͳ͍
    FB3Ά͍

    Pseudo Label ෇͖
    ͷFB1σʔλ
    Ϟσϧ

    View Slide

  9. © 2023 Wantedly, Inc.
    FB3 σʔληοτͷ All Data Training
    ● গͳ͍FB3σʔλΛग़དྷΔ͚ͩଟֶ͘शͰ͖ΔΑ͏ʹͯ͠ɺϞσϧͷ༧ଌਫ਼౓Λ্͛Δ


    ○ CV͕෼͔Βͳ͘ͳΔ໰୊ʹ͍ͭͯ͸ҎԼͷΑ͏ʹରԠ


    ■ 4 Fold Model ͷֶशͰ CV ଌఆ


    ■ ಉ͡ઃఆͰશσʔλֶश x 4ճ(seed͸όϥόϥ) Λ࣮ࢪ


    ○ Pseudo Label ͸ͦΕͧΕͰ࡞੒ͯ͠ϦʔΫ͠ͳ͍Α͏ʹ͢Δ


    ■ 4 Fold Model ൛ͷ pseudo label


    ■ શσʔλֶश൛ͷ pseudo label


    2. All Data Training

    View Slide

  10. © 2023 Wantedly, Inc.
    Pseudo Labeling ͷύλʔϯͷόϦΤʔγϣϯ
    ● 2ύλʔϯ࠾༻ͨ͠


    a. FB1σʔλͰࣄલֶशΛߦ͍ɺͦͷޙɺFB3 σʔλͷΈͰඍௐ੔Λߦ͏


    b. FB3σʔλʹFB1σʔλΛՃ͑ͯ܇࿅͢Δ
    3. Pseudo Labeling Patterns

    View Slide

  11. © 2023 Wantedly, Inc.
    Pseudo Labeling ͷΠλϨʔγϣϯճ਺
    ● Pseudo Labeling ͸܁Γฦ͢΄Ͳྑ͍ʢ͜ͱ͕͋Δʣ


    ○ ຊίϯϖͰ͸̏ճ܁Γฦͨ͠


    ○ NBME - Score Clinical Patient Notes Ͱ͸̎ճ


    ○ Parkinson's Freezing of Gait Prediction Ͱ͸̎ճ


    ● ܁Γฦ͢ͱ͸ʁ۩ମతʹ


    ○ 1st model ← FB3σʔλ͚ͩͰֶश


    ○ 2nd model ← FB3σʔλ + FB1σʔλ(1st modelʹΑΔpseudo labels)


    ○ 3rd model ← FB3σʔλ + FB1σʔλ(2nd modelʹΑΔpseudo labels)
    4. Number of Iterations

    View Slide

  12. © 2023 Wantedly, Inc.
    ࣄલֶशࡁΈϞσϧͷόϦΤʔγϣϯ
    ● ࣄલֶशࡁΈϞσϧͷଟ༷ੑΛ૿΍ͯ͠༧ଌਫ਼౓Λ্͛Δ


    ○ ࠓճͷίϯϖͰ͸17छྨͷϞσϧΛ࠾༻
    5. Pre-trained Models

    View Slide

  13. © 2023 Wantedly, Inc.
    Solution Overview

    View Slide

  14. © 2023 Wantedly, Inc.
    ຊίϯϖͰ͸ɺֶशσʔλͷ࣭ͱྔΛ্͛ΔͨΊͷ޻෉ʹऔΓ૊Μͩ
    ● ֶशσʔλͷྔΛ૿΍ͨ͢ΊͷऔΓ૊Έ


    ○ FB1 σʔληοτΛ࢖ͬͨ Pseudo Labeling


    ○ FB3 σʔληοτͷ All Data Training


    ● ֶशσʔλͷ࣭Λ্͛ΔͨΊͷऔΓ૊Έ


    ○ Adversarial Validation ʹΑΔ FB3 Ά͘ͳ͍ FB1 σʔλͷআ֎


    ○ Pseudo Labeling ͷΠλϨʔγϣϯճ਺


    ○ Pseudo Labeling ͷύλʔϯͷόϦΤʔγϣϯ


    ○ ࣄલֶशࡁΈϞσϧͷόϦΤʔγϣϯ
    Summary

    View Slide