Feedback Prize - English Language Learning における擬似ラベルの品質向上の取り組み

第4回 Data-Centric AI勉強会 -コンペLT大会-
https://dcai-jp.connpass.com/event/298953/

1. © 2023 Wantedly, Inc. Feedback Prize - English Language Learning 
 ʹ͓͚Δٖࣅϥϕϧͷ඼࣭޲্ͷऔΓ૊Έ ୈ4ճ Data-Centric AIษڧձ -ίϯϖLTେձ- Nov. 8 2023 - Shuhei Goda
2. © 2023 Wantedly, Inc. ໊લɿ ߹ా पฏ(Shuhei Goda) ॴଐͱ໾ׂɿ ΢ΥϯςουϦʔגࣜձࣾ σʔλαΠΤϯςΟετ ࣗݾ঺հ @hakubishin3 @jy_msc @shuheigoda
3. © 2023 Wantedly, Inc. 2023೥8݄ʙ11݄Ͱ։࠵͞Ε͍ͯͨ NLP ίϯϖ Feedback Prize - English Language Learning https://www.kaggle.com/competitions/feedback-prize-english-language-learning
4. © 2023 Wantedly, Inc. ӳޠখ࿦จͷ6छྨͷ඼࣭༧ଌΛධՁ • ֤ई౓ͷൣғ͸1.0~5.0(0.5ࠁΈ) • ֶशσʔλ਺ 3,911 ݅ Competition Data & Evaluation I believe that home-based learning could be advantageous for students as it eliminates the need for them to dress up and get ready... essay measure predicted value ground truth cohesion 2.53 3.5 syntax 1.12 1.0 vocabulary 3.25 5.0 phraseology 2.12 2.5 grammar 4.90 1.0 conventions 2.12 3.0 Scoring: MCRMSE (mean columnwise RMSE)
5. © 2023 Wantedly, Inc. ӳޠখ࿦จͷࣗಈධՁʹয఺Λ౰ͯͨγϦʔζ 1. Feedback Prize - Evaluating Student Writing (FB1 ͱݺশ) 2. Feedback Prize - Predicting Effective Arguments (FB2 ͱݺশ) 3. Feedback Prize - English Language Learning (FB3 ͱݺশ) FB3Ͱ͸
 աڈͷίϯϖ(FB1, FB2)ͷσʔλ΋ར༻Մೳ Feedback Prize Competitions Series FB1 essays FB3 essays 15,594 ݅ 3,911 ݅ ॏෳ 452 ݅
6. © 2023 Wantedly, Inc. ίϯϖΛऔΓ૊Ή্Ͱͷઓུ • ͪΐͬͱ৮ͬͯΈͯײͨ͜͡ͱ ◦ σʔλ΍໰୊ઃఆ͸ඇৗʹγϯϓϧɺσʔλྔ΋গͳ͍ ◦ Magic(͍͢͝ͻΒΊ͖)Ͱ͕ࠩͭ͘Α͏ͳ΋ͷͰ͸ͳͦ͞͏ • ํ਑ΛཱͯΔ ◦ ౰ͨΓલͷ͜ͱΛ΍ΕΔ͚ͩ΍Δ ◦ ֶशʹར༻͢Δσʔλͷ࣭ͱྔͰࠩΛ͚ͭΔ ▪ FB3σʔλ3,911݅ͱॏෳΛআ͍ͨFB1σʔλ15,142݅ΛͲ͏࢖͏͔ My Strategy
7. © 2023 Wantedly, Inc. ࣭ͷྑ͍ Pseudo Label(FB1σʔλ) ΛՃֶ͑ͯशͤ͞Δ • Pseudo Label ͷ࣭Λ্͛ΔͨΊͷऔΓ૊Έ a. Adversarial Validation ʹΑΔϊΠδʔͳσʔλͷআ֎ b. FB3 σʔληοτͷ All Data Training c. Pseudo Labeling ͷύλʔϯͷόϦΤʔγϣϯ d. Pseudo Labeling ͷΠλϨʔγϣϯճ਺ e. ࣄલֶशࡁΈϞσϧͷόϦΤʔγϣϯ Key Point
8. © 2023 Wantedly, Inc. Adversarial Validation ʹΑΔϊΠδʔͳσʔλͷআ֎ • FB1σʔληοτʹؚ·ΕΔશͯͷσʔλ͕FB3ͱಉ༷ͷ܏޲ΛऔΔͱ͸ݶΒͳ͍ ◦ ࢦఆͨ͠ essay ͕ FB3 ʹؚ·ΕΔ͔Ͳ͏͔Λ༧ଌ͢ΔλεΫΛ࣮ࢪ ◦ ͦͷ༧ଌ஋͕͖͍͠஋Ҏ্ͱͳΔFB1σʔλ͚ͩΛֶशʹ࢖͏ 1. Adversarial Validation 
 Pseudo Label ෇͖ ͷFB1σʔλ FB3σʔλ ͖͍͠஋ FB3Ά͍ FB3Ά͘ͳ͍ FB3Ά͍ 
 Pseudo Label ෇͖ ͷFB1σʔλ Ϟσϧ
9. © 2023 Wantedly, Inc. FB3 σʔληοτͷ All Data Training • গͳ͍FB3σʔλΛग़དྷΔ͚ͩଟֶ͘शͰ͖ΔΑ͏ʹͯ͠ɺϞσϧͷ༧ଌਫ਼౓Λ্͛Δ ◦ CV͕෼͔Βͳ͘ͳΔ໰୊ʹ͍ͭͯ͸ҎԼͷΑ͏ʹରԠ ▪ 4 Fold Model ͷֶशͰ CV ଌఆ ▪ ಉ͡ઃఆͰશσʔλֶश x 4ճ(seed͸όϥόϥ) Λ࣮ࢪ ◦ Pseudo Label ͸ͦΕͧΕͰ࡞੒ͯ͠ϦʔΫ͠ͳ͍Α͏ʹ͢Δ ▪ 4 Fold Model ൛ͷ pseudo label ▪ શσʔλֶश൛ͷ pseudo label 2. All Data Training
10. © 2023 Wantedly, Inc. Pseudo Labeling ͷύλʔϯͷόϦΤʔγϣϯ • 2ύλʔϯ࠾༻ͨ͠ a. FB1σʔλͰࣄલֶशΛߦ͍ɺͦͷޙɺFB3 σʔλͷΈͰඍௐ੔Λߦ͏ b. FB3σʔλʹFB1σʔλΛՃ͑ͯ܇࿅͢Δ 3. Pseudo Labeling Patterns
11. © 2023 Wantedly, Inc. Pseudo Labeling ͷΠλϨʔγϣϯճ਺ • Pseudo Labeling ͸܁Γฦ͢΄Ͳྑ͍ʢ͜ͱ͕͋Δʣ ◦ ຊίϯϖͰ͸̏ճ܁Γฦͨ͠ ◦ NBME - Score Clinical Patient Notes Ͱ͸̎ճ ◦ Parkinson's Freezing of Gait Prediction Ͱ͸̎ճ • ܁Γฦ͢ͱ͸ʁ۩ମతʹ ◦ 1st model ← FB3σʔλ͚ͩͰֶश ◦ 2nd model ← FB3σʔλ + FB1σʔλ(1st modelʹΑΔpseudo labels) ◦ 3rd model ← FB3σʔλ + FB1σʔλ(2nd modelʹΑΔpseudo labels) 4. Number of Iterations
12. © 2023 Wantedly, Inc. ࣄલֶशࡁΈϞσϧͷόϦΤʔγϣϯ • ࣄલֶशࡁΈϞσϧͷଟ༷ੑΛ૿΍ͯ͠༧ଌਫ਼౓Λ্͛Δ ◦ ࠓճͷίϯϖͰ͸17छྨͷϞσϧΛ࠾༻ 5. Pre-trained Models
13. © 2023 Wantedly, Inc. Solution Overview
14. © 2023 Wantedly, Inc. ຊίϯϖͰ͸ɺֶशσʔλͷ࣭ͱྔΛ্͛ΔͨΊͷ޻෉ʹऔΓ૊Μͩ • ֶशσʔλͷྔΛ૿΍ͨ͢ΊͷऔΓ૊Έ ◦ FB1 σʔληοτΛ࢖ͬͨ Pseudo Labeling ◦ FB3 σʔληοτͷ All Data Training • ֶशσʔλͷ࣭Λ্͛ΔͨΊͷऔΓ૊Έ ◦ Adversarial Validation ʹΑΔ FB3 Ά͘ͳ͍ FB1 σʔλͷআ֎ ◦ Pseudo Labeling ͷΠλϨʔγϣϯճ਺ ◦ Pseudo Labeling ͷύλʔϯͷόϦΤʔγϣϯ ◦ ࣄલֶशࡁΈϞσϧͷόϦΤʔγϣϯ Summary