CommonLitコンペで学んだこと

Transcript

1. ©2023 Wantedly, Inc.
   CommonLitίϯϖͰֶΜͩ͜ͱ
   Custom HeaderʹΑΔTransformerੑೳվળ
   ΈΜͳͷPythonษڧձ #98
   
   
   Oct. 12 2023 - @nogawanogawa
   

   View Slide

2. ©2023 Wantedly, Inc.
   CommonLit ίϯϖ ?
   CommonLit - Evaluate Student Summaries
   

   View Slide

3. ©2023 Wantedly, Inc.
   CommonLit - Evaluate Student Summaries
   KaggleʢσʔλαΠΤϯείϯϖςΟγϣϯϓϥοτ
   ϑΥʔϜʣͰ։࠵͞Ε͍ͯͨେձ
   
   
   • ظؒ: 7/13 ~ 10/12 (3͔݄ɺࠓேऴྃʂ)
   
   
   • ࢀՃऀ: 2106νʔϜ
   • ࣗ෼ͷ࠷ऴॱҐ͸25ҐͰͨ͠
   

   View Slide

4. ©2023 Wantedly, Inc.
   CommonLit - Evaluate Student Summaries
   • খֶߍ3೥ੜʙߴߍ3೥ੜͷੜెʹΑΔจষͷཁ໿͕ςʔϚ
   
   
   • ੜె͕࡞੒ͨ͠ཁ໿จͷ࣭Λهड़಺༰(contentʣɺޠኮɾจ๏ʢwordingʣͷ2ͭͷ؍఺Ͱ࠾఺
   
   
   ◦ ࣮ࡍʹਓ͕ؒ࠾఺ͨ݁͠Ռʹ࠷΋͍ۙػցֶशϞσϧΛ࡞ͬͨਓ͕উར
   จষΛཁ໿
   ੜె
   ઌੜ
   XPSEJOH఺
   DPOUFOU఺
   XPSEJOH఺
   DPOUFOU఺
   ػցֶशϞσϧ
   ͳΔ΂͘ਓؒͷ
   
   
   ࠾఺݁Ռʹ͍ۙ
   
   
   ϞσϧΛ࡞Γ͍ͨʂ
   ࢀՃऀ͸͜ΕΛ࡞͍ͬͯͨ
   ࠾఺
   ༧ଌ
   

   View Slide

5. ©2023 Wantedly, Inc.
   TransformerͱCustom Header
   

   View Slide

6. ©2023 Wantedly, Inc.
   TransformerΛ࢖ͬͨجຊతͳղ๏
   • ࣗવจΛ࢖͏ίϯϖͰ͸Α͘Transformer͕࢖༻͞ΕΔ
   
   
   • χϡʔϥϧωοτϫʔΫͷݱࡏओྲྀͷΞʔΩςΫνϟ
   
   
   • ߏ଄͸ӈਤͷΑ͏ʹͳ͍ͬͯΔ
   
   
   ◦ ChatGPT΋ཪଆ΋͜Εͷ஥ؒ
   
   
   • ࢖͏͜ͱࣗମ͸ҙ֎ͱ؆୯
   
   
   ◦ HuggingFaceͷTransformersϥΠϒϥϦΛ࢖͑͹਺
   ेߦఔ౓ॻ͚͹खܰʹ࢖͑Δ
   
   
   ◦ ੈքதͷਓֶ͕शࡁΈϞσϧͷσʔλΛެ։ͯ͘͠Εͯ
   ͍ͯɺDLͯ͠࢖͏͚ͩ
   

   View Slide

7. ©2023 Wantedly, Inc.
   Transformers
   ࢖༻͍ͨ͠ϞσϧͷछྨΛબ୒
   ֶशࡁΈϞσϧΛμ΢ϯϩʔυ
   ೉͘͠ͳ͍ʂ
   ۩ମతʹղ͖͍ͨ໰୊ͷͨΊʹඍௐ੔(fine tune)
   ͢ΔίʔυΛॻ͍ͨͱͯ͠΋ϓϥε਺ेߦఔ౓
   • ࢖༻͢Δ͚ͩͳΒ਺ߦͰॻ͚Δ
   

   View Slide

8. ©2023 Wantedly, Inc.
   ਫ਼౓޲্ͷํ਑
   • લॲཧ
   
   
   • ଟ͘ͷσʔλ͸͋·Γ͖Ε͍Ͱ͸ͳ͍ʢϊΠζͱͳΔ΋ͷ͕ଟ͍ʣࣄ͕ଟ͘ɺͦΕΒΛআڈ͢Δ
   
   
   • ࣄલֶशࡁΈϞσϧͷมߋ
   
   
   • ࢖͑ΔֶशࡁΈϞσϧʹ΋ͨ͘͞Μछྨ͕͋ΔʢBertɺRobertaɺDebertaɺetc… ʣ
   
   
   • ֶशσʔλͷ૿ڧ
   
   
   • ΑΓଟ͘ɾଟ༷ͳσʔλͰֶशͨ͠Ϟσϧ͕ڧ͘ͳΓ͕ͪ
   
   
   • Ξϯαϯϒϧ
   
   
   • ෳ਺ͷػցֶशϞσϧΛ࢖ͬͨଟ਺ܾ
   
   
   • Transformerͷߏ଄ࣗମʹςίೖΕ
   
   
   • Custom Header ʢPoolerʣ
   
   
   • etc…
   ࠓ೔͸͜͜Λ࿩͍ͨ͠ʂ
   Transformers͸؆୯ʹ࢖͑Δ & ߴ͍ਫ਼౓ͷϞσϧΛ࢖༻Ͱ͖Δɻͨͩ͠ɺKaggleͳͲͷίϯϖͰ͸͜Ε
   Λ࢖͏ͷ͕ελʔτϥΠϯͰɺίϯϖͰॱҐΛ্͛Δʹ͸͔͜͜Βߋʹ޻෉͕ඞཁ
   

   View Slide

9. ©2023 Wantedly, Inc.
   Custom Header (Pooler)ʹΑΔਫ਼౓޲্ࡦ
   • σϑΥϧτͰTransformerΛ࢖༻͢Δͱɺೖྗͷ࠷ॳͷτʔΫϯ (CLSτʔΫϯ)ͷ࠷ऴ૚ग़ྗΛ΋ͱʹ༧ଌ
   ࠷ॳͷτʔΫϯग़ྗ͚ͩΛ࢖༻ɺଞͷτʔΫϯͷग़ྗ͸ࣺ͍ͯͯΔ
   →௚ײతʹ͸ඇৗʹ΋͍ͬͨͳ͘ײ͡Δ
   
   
   ʢจষશମͷग़ྗΛ࢖ͬͨ΄͏͕දݱྗ͸ߴ͘ͳΓͦ͏ʹࢥ͑Δʣ
   

   View Slide

10. ©2023 Wantedly, Inc.
    Custom Header (Pooler)ʹΑΔਫ਼౓޲্ࡦ
    • Custom Header (Pooler) : จશମͷग़ྗΛ࢖ͬͯpoolingΛߦ͍ɺ࠷ऴ૚ʹೖྗ
    
    
    • poolingͷྫ: ฏۉɺ࠷େɺ࠷ऴ4૚ͷग़ྗΛͭͳ͛Δɺetc…
    ࢀߟ: https://www.kaggle.com/code/rhtsingh/utilizing-transformer-representations-efficiently
    ೉ͦ͠͏ʹݟ͑ͯ
    
    
    ҙ֎ͱ਺ߦͰॻ͚Δ
    ྫ: Average pooling
    

    View Slide

11. ©2023 Wantedly, Inc.
    ΦϦδφϧCustom HeaderʹΑΔਫ਼౓޲্ࡦ
    • ࠓճࣗ෼͕΍ͬͯҰ൪͏·͘ߦͬͨํ๏
    
    
    • ཁ໿ର৅ͷจͷՕॴ͚ͩΛaverage
    poolingͯ͠࢖༻
    
    
    • ղऍ
    
    
    • ࠓճ͸ੜెͷཁ໿จͷ࣭ΛධՁ͢Δͷ͕໨త
    
    
    • ධՁର৅ͷ෦෼͚ͩ࢖༻͢Δ͜ͱͰ࠷ऴ૚
    ΁ͷϊΠζΛݮΒ͢
    

    View Slide

12. ©2023 Wantedly, Inc.
    ·ͱΊɿCommonLitίϯϖͰֶΜͩ͜ͱ
    •TransformerϞσϧ͸ͦ͜·Ͱۤ࿑ͤͣ࢖͑ͯߴੑೳɻͨͩ͠Ԟ΋ਂ͍ɻ
    
    
    • TransformersΛ࢖͏ͱઌਓ͕࡞ͬͨߴੑೳͷֶशࡁΈϞσϧΛ࢖༻Ͱ͖Δ
    
    
    ◦ ࢖͏͚ͩͳΒͦ͜·Ͱ೉͘͠ͳ͍
    
    
    • ͨͩ͠࠷ۙͷKaggleͳͲͷNLPίϯϖͰ͸ɺTransformerΛ࢖͏ͷ͸͋͘·ͰελʔτϥΠϯ
    
    
    ◦ σʔλαΠΤϯςΟετతʹ͸΋͏Ұา౿ΈࠐΜͰ͍͖͍ͨ
    
    
    • TransformerͷੑೳΛߋʹ޲্ͤ͞ΔΞϓϩʔνͷ1ͭ: Custom Header (Pooler)
    
    
    ◦ Transformer͕λεΫΛղ͖΍͍͢Α͏ʹ࠷ޙͷPoolingΛௐ੔͢Δͱ্ख͍͘͘͜ͱ͕͋Δ
    

    View Slide