CommonLitコンペで学んだこと

1. ©2023 Wantedly, Inc. CommonLitίϯϖͰֶΜͩ͜ͱ Custom HeaderʹΑΔTransformerੑೳվળ ΈΜͳͷPythonษڧձ #98 Oct. 12 2023 - @nogawanogawa
2. ©2023 Wantedly, Inc. CommonLit ίϯϖ ? CommonLit - Evaluate Student Summaries
3. ©2023 Wantedly, Inc. CommonLit - Evaluate Student Summaries KaggleʢσʔλαΠΤϯείϯϖςΟγϣϯϓϥοτ ϑΥʔϜʣͰ։࠵͞Ε͍ͯͨେձ • ظؒ: 7/13 ~ 10/12 (3͔݄ɺࠓேऴྃʂ) • ࢀՃऀ: 2106νʔϜ • ࣗ෼ͷ࠷ऴॱҐ͸25ҐͰͨ͠
4. ©2023 Wantedly, Inc. CommonLit - Evaluate Student Summaries • খֶߍ3೥ੜʙߴߍ3೥ੜͷੜెʹΑΔจষͷཁ໿͕ςʔϚ • ੜె͕࡞੒ͨ͠ཁ໿จͷ࣭Λهड़಺༰(contentʣɺޠኮɾจ๏ʢwordingʣͷ2ͭͷ؍఺Ͱ࠾఺ ◦ ࣮ࡍʹਓ͕ؒ࠾఺ͨ݁͠Ռʹ࠷΋͍ۙػցֶशϞσϧΛ࡞ͬͨਓ͕উར จষΛཁ໿ ੜె ઌੜ XPSEJOH
   ఺ DPOUFOU
   ఺ XPSEJOH
   ఺ DPOUFOU
   ఺ ػցֶशϞσϧ ͳΔ΂͘ਓؒͷ ࠾఺݁Ռʹ͍ۙ ϞσϧΛ࡞Γ͍ͨʂ ࢀՃऀ͸͜ΕΛ࡞͍ͬͯͨ ࠾఺ ༧ଌ
5. ©2023 Wantedly, Inc. TransformerͱCustom Header
6. ©2023 Wantedly, Inc. TransformerΛ࢖ͬͨجຊతͳղ๏ • ࣗવจΛ࢖͏ίϯϖͰ͸Α͘Transformer͕࢖༻͞ΕΔ • χϡʔϥϧωοτϫʔΫͷݱࡏओྲྀͷΞʔΩςΫνϟ • ߏ଄͸ӈਤͷΑ͏ʹͳ͍ͬͯΔ ◦ ChatGPT΋ཪଆ΋͜Εͷ஥ؒ • ࢖͏͜ͱࣗମ͸ҙ֎ͱ؆୯ ◦ HuggingFaceͷTransformersϥΠϒϥϦΛ࢖͑͹਺ ेߦఔ౓ॻ͚͹खܰʹ࢖͑Δ ◦ ੈքதͷਓֶ͕शࡁΈϞσϧͷσʔλΛެ։ͯ͘͠Εͯ ͍ͯɺDLͯ͠࢖͏͚ͩ
7. ©2023 Wantedly, Inc. Transformers ࢖༻͍ͨ͠ϞσϧͷछྨΛબ୒ ֶशࡁΈϞσϧΛμ΢ϯϩʔυ ೉͘͠ͳ͍ʂ ۩ମతʹղ͖͍ͨ໰୊ͷͨΊʹඍௐ੔(fine tune) ͢ΔίʔυΛॻ͍ͨͱͯ͠΋ϓϥε਺ेߦఔ౓ • ࢖༻͢Δ͚ͩͳΒ਺ߦͰॻ͚Δ
8. ©2023 Wantedly, Inc. ਫ਼౓޲্ͷํ਑ • લॲཧ • ଟ͘ͷσʔλ͸͋·Γ͖Ε͍Ͱ͸ͳ͍ʢϊΠζͱͳΔ΋ͷ͕ଟ͍ʣࣄ͕ଟ͘ɺͦΕΒΛআڈ͢Δ • ࣄલֶशࡁΈϞσϧͷมߋ • ࢖͑ΔֶशࡁΈϞσϧʹ΋ͨ͘͞Μछྨ͕͋ΔʢBertɺRobertaɺDebertaɺetc… ʣ • ֶशσʔλͷ૿ڧ • ΑΓଟ͘ɾଟ༷ͳσʔλͰֶशͨ͠Ϟσϧ͕ڧ͘ͳΓ͕ͪ • Ξϯαϯϒϧ • ෳ਺ͷػցֶशϞσϧΛ࢖ͬͨଟ਺ܾ • Transformerͷߏ଄ࣗମʹςίೖΕ • Custom Header ʢPoolerʣ • etc… ࠓ೔͸͜͜Λ࿩͍ͨ͠ʂ Transformers͸؆୯ʹ࢖͑Δ & ߴ͍ਫ਼౓ͷϞσϧΛ࢖༻Ͱ͖Δɻͨͩ͠ɺKaggleͳͲͷίϯϖͰ͸͜Ε Λ࢖͏ͷ͕ελʔτϥΠϯͰɺίϯϖͰॱҐΛ্͛Δʹ͸͔͜͜Βߋʹ޻෉͕ඞཁ
9. ©2023 Wantedly, Inc. Custom Header (Pooler)ʹΑΔਫ਼౓޲্ࡦ • σϑΥϧτͰTransformerΛ࢖༻͢Δͱɺೖྗͷ࠷ॳͷτʔΫϯ (CLSτʔΫϯ)ͷ࠷ऴ૚ग़ྗΛ΋ͱʹ༧ଌ ࠷ॳͷτʔΫϯग़ྗ͚ͩΛ࢖༻ɺଞͷτʔΫϯͷग़ྗ͸ࣺ͍ͯͯΔ →௚ײతʹ͸ඇৗʹ΋͍ͬͨͳ͘ײ͡Δ ʢจষશମͷग़ྗΛ࢖ͬͨ΄͏͕දݱྗ͸ߴ͘ͳΓͦ͏ʹࢥ͑Δʣ
10. ©2023 Wantedly, Inc. Custom Header (Pooler)ʹΑΔਫ਼౓޲্ࡦ • Custom Header (Pooler) : จશମͷग़ྗΛ࢖ͬͯpoolingΛߦ͍ɺ࠷ऴ૚ʹೖྗ • poolingͷྫ: ฏۉɺ࠷େɺ࠷ऴ4૚ͷग़ྗΛͭͳ͛Δɺetc… ࢀߟ: https://www.kaggle.com/code/rhtsingh/utilizing-transformer-representations-efficiently ೉ͦ͠͏ʹݟ͑ͯ ҙ֎ͱ਺ߦͰॻ͚Δ ྫ: Average pooling
11. ©2023 Wantedly, Inc. ΦϦδφϧCustom HeaderʹΑΔਫ਼౓޲্ࡦ • ࠓճࣗ෼͕΍ͬͯҰ൪͏·͘ߦͬͨํ๏ • ཁ໿ର৅ͷจͷՕॴ͚ͩΛaverage poolingͯ͠࢖༻ • ղऍ • ࠓճ͸ੜెͷཁ໿จͷ࣭ΛධՁ͢Δͷ͕໨త • ධՁର৅ͷ෦෼͚ͩ࢖༻͢Δ͜ͱͰ࠷ऴ૚ ΁ͷϊΠζΛݮΒ͢
12. ©2023 Wantedly, Inc. ·ͱΊɿCommonLitίϯϖͰֶΜͩ͜ͱ •TransformerϞσϧ͸ͦ͜·Ͱۤ࿑ͤͣ࢖͑ͯߴੑೳɻͨͩ͠Ԟ΋ਂ͍ɻ • TransformersΛ࢖͏ͱઌਓ͕࡞ͬͨߴੑೳͷֶशࡁΈϞσϧΛ࢖༻Ͱ͖Δ ◦ ࢖͏͚ͩͳΒͦ͜·Ͱ೉͘͠ͳ͍ • ͨͩ͠࠷ۙͷKaggleͳͲͷNLPίϯϖͰ͸ɺTransformerΛ࢖͏ͷ͸͋͘·ͰελʔτϥΠϯ ◦ σʔλαΠΤϯςΟετతʹ͸΋͏Ұา౿ΈࠐΜͰ͍͖͍ͨ • TransformerͷੑೳΛߋʹ޲্ͤ͞ΔΞϓϩʔνͷ1ͭ: Custom Header (Pooler) ◦ Transformer͕λεΫΛղ͖΍͍͢Α͏ʹ࠷ޙͷPoolingΛௐ੔͢Δͱ্ख͍͘͘͜ͱ͕͋Δ