Feedback Prize - English Language Learning における擬似ラベルの品質向上の取り組み

第4回 Data-Centric AI勉強会 -コンペLT大会-
https://dcai-jp.connpass.com/event/298953/

Transcript

1. © 2023 Wantedly, Inc.
   Feedback Prize - English Language Learning 

   ʹ͓͚Δٖࣅϥϕϧͷ඼࣭޲্ͷऔΓ૊Έ
   ୈ4ճ Data-Centric AIษڧձ -ίϯϖLTେձ-
   Nov. 8 2023 - Shuhei Goda
   

   View Slide

2. © 2023 Wantedly, Inc.
   ໊લɿ
   ߹ా पฏ(Shuhei Goda)
   ॴଐͱ໾ׂɿ
   ΢ΥϯςουϦʔגࣜձࣾ
   σʔλαΠΤϯςΟετ
   ࣗݾ঺հ
   @hakubishin3
   @jy_msc
   @shuheigoda
   

   View Slide

3. © 2023 Wantedly, Inc.
   2023೥8݄ʙ11݄Ͱ։࠵͞Ε͍ͯͨ NLP ίϯϖ
   Feedback Prize - English Language Learning
   https://www.kaggle.com/competitions/feedback-prize-english-language-learning
   

   View Slide

4. © 2023 Wantedly, Inc.
   ӳޠখ࿦จͷ6छྨͷ඼࣭༧ଌΛධՁ
   ● ֤ई౓ͷൣғ͸1.0~5.0(0.5ࠁΈ)
   
   
   ● ֶशσʔλ਺ 3,911 ݅
   Competition Data & Evaluation
   I believe that home-based
   learning could be
   advantageous for students as
   it eliminates the need for
   them to dress up and get
   ready...
   essay
   measure predicted value ground truth
   cohesion 2.53 3.5
   syntax 1.12 1.0
   vocabulary 3.25 5.0
   phraseology 2.12 2.5
   grammar 4.90 1.0
   conventions 2.12 3.0
   Scoring: MCRMSE
   
   
   (mean columnwise RMSE)
   

   View Slide

5. © 2023 Wantedly, Inc.
   ӳޠখ࿦จͷࣗಈධՁʹয఺Λ౰ͯͨγϦʔζ
   1. Feedback Prize - Evaluating Student Writing (FB1 ͱݺশ)
   2. Feedback Prize - Predicting Effective Arguments (FB2 ͱݺশ)
   3. Feedback Prize - English Language Learning (FB3 ͱݺশ)
   FB3Ͱ͸

   աڈͷίϯϖ(FB1, FB2)ͷσʔλ΋ར༻Մೳ
   Feedback Prize Competitions Series
   FB1 essays
   FB3 essays
   15,594 ݅
   3,911 ݅
   ॏෳ 452 ݅
   

   View Slide

6. © 2023 Wantedly, Inc.
   ίϯϖΛऔΓ૊Ή্Ͱͷઓུ
   ● ͪΐͬͱ৮ͬͯΈͯײͨ͜͡ͱ
   
   
   ○ σʔλ΍໰୊ઃఆ͸ඇৗʹγϯϓϧɺσʔλྔ΋গͳ͍
   
   
   ○ Magic(͍͢͝ͻΒΊ͖)Ͱ͕ࠩͭ͘Α͏ͳ΋ͷͰ͸ͳͦ͞͏
   
   
   ● ํ਑ΛཱͯΔ
   
   
   ○ ౰ͨΓલͷ͜ͱΛ΍ΕΔ͚ͩ΍Δ
   
   
   ○ ֶशʹར༻͢Δσʔλͷ࣭ͱྔͰࠩΛ͚ͭΔ
   
   
   ■ FB3σʔλ3,911݅ͱॏෳΛআ͍ͨFB1σʔλ15,142݅ΛͲ͏࢖͏͔
   My Strategy
   

   View Slide

7. © 2023 Wantedly, Inc.
   ࣭ͷྑ͍ Pseudo Label(FB1σʔλ) ΛՃֶ͑ͯशͤ͞Δ
   ● Pseudo Label ͷ࣭Λ্͛ΔͨΊͷऔΓ૊Έ
   
   
   a. Adversarial Validation ʹΑΔϊΠδʔͳσʔλͷআ֎
   
   
   b. FB3 σʔληοτͷ All Data Training
   
   
   c. Pseudo Labeling ͷύλʔϯͷόϦΤʔγϣϯ
   
   
   d. Pseudo Labeling ͷΠλϨʔγϣϯճ਺
   
   
   e. ࣄલֶशࡁΈϞσϧͷόϦΤʔγϣϯ
   Key Point
   

   View Slide

8. © 2023 Wantedly, Inc.
   Adversarial Validation ʹΑΔϊΠδʔͳσʔλͷআ֎
   ● FB1σʔληοτʹؚ·ΕΔશͯͷσʔλ͕FB3ͱಉ༷ͷ܏޲ΛऔΔͱ͸ݶΒͳ͍
   
   
   ○ ࢦఆͨ͠ essay ͕ FB3 ʹؚ·ΕΔ͔Ͳ͏͔Λ༧ଌ͢ΔλεΫΛ࣮ࢪ
   
   
   ○ ͦͷ༧ଌ஋͕͖͍͠஋Ҏ্ͱͳΔFB1σʔλ͚ͩΛֶशʹ࢖͏
   1. Adversarial Validation
   

   Pseudo Label ෇͖
   ͷFB1σʔλ
   FB3σʔλ
   ͖͍͠஋
   FB3Ά͍
   FB3Ά͘ͳ͍
   FB3Ά͍
   

   Pseudo Label ෇͖
   ͷFB1σʔλ
   Ϟσϧ
   

   View Slide

9. © 2023 Wantedly, Inc.
   FB3 σʔληοτͷ All Data Training
   ● গͳ͍FB3σʔλΛग़དྷΔ͚ͩଟֶ͘शͰ͖ΔΑ͏ʹͯ͠ɺϞσϧͷ༧ଌਫ਼౓Λ্͛Δ
   
   
   ○ CV͕෼͔Βͳ͘ͳΔ໰୊ʹ͍ͭͯ͸ҎԼͷΑ͏ʹରԠ
   
   
   ■ 4 Fold Model ͷֶशͰ CV ଌఆ
   
   
   ■ ಉ͡ઃఆͰશσʔλֶश x 4ճ(seed͸όϥόϥ) Λ࣮ࢪ
   
   
   ○ Pseudo Label ͸ͦΕͧΕͰ࡞੒ͯ͠ϦʔΫ͠ͳ͍Α͏ʹ͢Δ
   
   
   ■ 4 Fold Model ൛ͷ pseudo label
   
   
   ■ શσʔλֶश൛ͷ pseudo label
   
   
   2. All Data Training
   

   View Slide

10. © 2023 Wantedly, Inc.
    Pseudo Labeling ͷύλʔϯͷόϦΤʔγϣϯ
    ● 2ύλʔϯ࠾༻ͨ͠
    
    
    a. FB1σʔλͰࣄલֶशΛߦ͍ɺͦͷޙɺFB3 σʔλͷΈͰඍௐ੔Λߦ͏
    
    
    b. FB3σʔλʹFB1σʔλΛՃ͑ͯ܇࿅͢Δ
    3. Pseudo Labeling Patterns
    

    View Slide

11. © 2023 Wantedly, Inc.
    Pseudo Labeling ͷΠλϨʔγϣϯճ਺
    ● Pseudo Labeling ͸܁Γฦ͢΄Ͳྑ͍ʢ͜ͱ͕͋Δʣ
    
    
    ○ ຊίϯϖͰ͸̏ճ܁Γฦͨ͠
    
    
    ○ NBME - Score Clinical Patient Notes Ͱ͸̎ճ
    
    
    ○ Parkinson's Freezing of Gait Prediction Ͱ͸̎ճ
    
    
    ● ܁Γฦ͢ͱ͸ʁ۩ମతʹ
    
    
    ○ 1st model ← FB3σʔλ͚ͩͰֶश
    
    
    ○ 2nd model ← FB3σʔλ + FB1σʔλ(1st modelʹΑΔpseudo labels)
    
    
    ○ 3rd model ← FB3σʔλ + FB1σʔλ(2nd modelʹΑΔpseudo labels)
    4. Number of Iterations
    

    View Slide

12. © 2023 Wantedly, Inc.
    ࣄલֶशࡁΈϞσϧͷόϦΤʔγϣϯ
    ● ࣄલֶशࡁΈϞσϧͷଟ༷ੑΛ૿΍ͯ͠༧ଌਫ਼౓Λ্͛Δ
    
    
    ○ ࠓճͷίϯϖͰ͸17छྨͷϞσϧΛ࠾༻
    5. Pre-trained Models
    

    View Slide

13. © 2023 Wantedly, Inc.
    Solution Overview
    

    View Slide

14. © 2023 Wantedly, Inc.
    ຊίϯϖͰ͸ɺֶशσʔλͷ࣭ͱྔΛ্͛ΔͨΊͷ޻෉ʹऔΓ૊Μͩ
    ● ֶशσʔλͷྔΛ૿΍ͨ͢ΊͷऔΓ૊Έ
    
    
    ○ FB1 σʔληοτΛ࢖ͬͨ Pseudo Labeling
    
    
    ○ FB3 σʔληοτͷ All Data Training
    
    
    ● ֶशσʔλͷ࣭Λ্͛ΔͨΊͷऔΓ૊Έ
    
    
    ○ Adversarial Validation ʹΑΔ FB3 Ά͘ͳ͍ FB1 σʔλͷআ֎
    
    
    ○ Pseudo Labeling ͷΠλϨʔγϣϯճ਺
    
    
    ○ Pseudo Labeling ͷύλʔϯͷόϦΤʔγϣϯ
    
    
    ○ ࣄલֶशࡁΈϞσϧͷόϦΤʔγϣϯ
    Summary
    

    View Slide