Upgrade to Pro — share decks privately, control downloads, hide ads and more …

CommonLitコンペで学んだこと

nogawanogawa
October 12, 2023
1.9k

 CommonLitコンペで学んだこと

nogawanogawa

October 12, 2023
Tweet

Transcript

  1. ©2023 Wantedly, Inc. CommonLit - Evaluate Student Summaries KaggleʢσʔλαΠΤϯείϯϖςΟγϣϯϓϥοτ ϑΥʔϜʣͰ։࠵͞Ε͍ͯͨେձ

    • ظؒ: 7/13 ~ 10/12 (3͔݄ɺࠓேऴྃʂ) • ࢀՃऀ: 2106νʔϜ • ࣗ෼ͷ࠷ऴॱҐ͸25ҐͰͨ͠
  2. ©2023 Wantedly, Inc. CommonLit - Evaluate Student Summaries • খֶߍ3೥ੜʙߴߍ3೥ੜͷੜెʹΑΔจষͷཁ໿͕ςʔϚ

    • ੜె͕࡞੒ͨ͠ཁ໿จͷ࣭Λهड़಺༰(contentʣɺޠኮɾจ๏ʢwordingʣͷ2ͭͷ؍఺Ͱ࠾఺ ◦ ࣮ࡍʹਓ͕ؒ࠾఺ͨ݁͠Ռʹ࠷΋͍ۙػցֶशϞσϧΛ࡞ͬͨਓ͕উར จষΛཁ໿ ੜె ઌੜ XPSEJOH఺ DPOUFOU఺ XPSEJOH఺ DPOUFOU఺ ػցֶशϞσϧ ͳΔ΂͘ਓؒͷ ࠾఺݁Ռʹ͍ۙ ϞσϧΛ࡞Γ͍ͨʂ ࢀՃऀ͸͜ΕΛ࡞͍ͬͯͨ ࠾఺ ༧ଌ
  3. ©2023 Wantedly, Inc. TransformerΛ࢖ͬͨجຊతͳղ๏ • ࣗવจΛ࢖͏ίϯϖͰ͸Α͘Transformer͕࢖༻͞ΕΔ • χϡʔϥϧωοτϫʔΫͷݱࡏओྲྀͷΞʔΩςΫνϟ • ߏ଄͸ӈਤͷΑ͏ʹͳ͍ͬͯΔ

    ◦ ChatGPT΋ཪଆ΋͜Εͷ஥ؒ • ࢖͏͜ͱࣗମ͸ҙ֎ͱ؆୯ ◦ HuggingFaceͷTransformersϥΠϒϥϦΛ࢖͑͹਺ ेߦఔ౓ॻ͚͹खܰʹ࢖͑Δ ◦ ੈքதͷਓֶ͕शࡁΈϞσϧͷσʔλΛެ։ͯ͘͠Εͯ ͍ͯɺDLͯ͠࢖͏͚ͩ
  4. ©2023 Wantedly, Inc. ਫ਼౓޲্ͷํ਑ • લॲཧ • ଟ͘ͷσʔλ͸͋·Γ͖Ε͍Ͱ͸ͳ͍ʢϊΠζͱͳΔ΋ͷ͕ଟ͍ʣࣄ͕ଟ͘ɺͦΕΒΛআڈ͢Δ • ࣄલֶशࡁΈϞσϧͷมߋ

    • ࢖͑ΔֶशࡁΈϞσϧʹ΋ͨ͘͞Μछྨ͕͋ΔʢBertɺRobertaɺDebertaɺetc… ʣ • ֶशσʔλͷ૿ڧ • ΑΓଟ͘ɾଟ༷ͳσʔλͰֶशͨ͠Ϟσϧ͕ڧ͘ͳΓ͕ͪ • Ξϯαϯϒϧ • ෳ਺ͷػցֶशϞσϧΛ࢖ͬͨଟ਺ܾ • Transformerͷߏ଄ࣗମʹςίೖΕ • Custom Header ʢPoolerʣ • etc… ࠓ೔͸͜͜Λ࿩͍ͨ͠ʂ Transformers͸؆୯ʹ࢖͑Δ & ߴ͍ਫ਼౓ͷϞσϧΛ࢖༻Ͱ͖Δɻͨͩ͠ɺKaggleͳͲͷίϯϖͰ͸͜Ε Λ࢖͏ͷ͕ελʔτϥΠϯͰɺίϯϖͰॱҐΛ্͛Δʹ͸͔͜͜Βߋʹ޻෉͕ඞཁ
  5. ©2023 Wantedly, Inc. Custom Header (Pooler)ʹΑΔਫ਼౓޲্ࡦ • Custom Header (Pooler)

    : จશମͷग़ྗΛ࢖ͬͯpoolingΛߦ͍ɺ࠷ऴ૚ʹೖྗ • poolingͷྫ: ฏۉɺ࠷େɺ࠷ऴ4૚ͷग़ྗΛͭͳ͛Δɺetc… ࢀߟ: https://www.kaggle.com/code/rhtsingh/utilizing-transformer-representations-efficiently ೉ͦ͠͏ʹݟ͑ͯ ҙ֎ͱ਺ߦͰॻ͚Δ ྫ: Average pooling
  6. ©2023 Wantedly, Inc. ΦϦδφϧCustom HeaderʹΑΔਫ਼౓޲্ࡦ • ࠓճࣗ෼͕΍ͬͯҰ൪͏·͘ߦͬͨํ๏ • ཁ໿ର৅ͷจͷՕॴ͚ͩΛaverage poolingͯ͠࢖༻

    • ղऍ • ࠓճ͸ੜెͷཁ໿จͷ࣭ΛධՁ͢Δͷ͕໨త • ධՁର৅ͷ෦෼͚ͩ࢖༻͢Δ͜ͱͰ࠷ऴ૚ ΁ͷϊΠζΛݮΒ͢
  7. ©2023 Wantedly, Inc. ·ͱΊɿCommonLitίϯϖͰֶΜͩ͜ͱ •TransformerϞσϧ͸ͦ͜·Ͱۤ࿑ͤͣ࢖͑ͯߴੑೳɻͨͩ͠Ԟ΋ਂ͍ɻ • TransformersΛ࢖͏ͱઌਓ͕࡞ͬͨߴੑೳͷֶशࡁΈϞσϧΛ࢖༻Ͱ͖Δ ◦ ࢖͏͚ͩͳΒͦ͜·Ͱ೉͘͠ͳ͍ •

    ͨͩ͠࠷ۙͷKaggleͳͲͷNLPίϯϖͰ͸ɺTransformerΛ࢖͏ͷ͸͋͘·ͰελʔτϥΠϯ ◦ σʔλαΠΤϯςΟετతʹ͸΋͏Ұา౿ΈࠐΜͰ͍͖͍ͨ • TransformerͷੑೳΛߋʹ޲্ͤ͞ΔΞϓϩʔνͷ1ͭ: Custom Header (Pooler) ◦ Transformer͕λεΫΛղ͖΍͍͢Α͏ʹ࠷ޙͷPoolingΛௐ੔͢Δͱ্ख͍͘͘͜ͱ͕͋Δ