自己教師あり学習の新しいアプローチ / SimSiam: Exploring Simple Siamese Representation Learning

■イベント 
：画像処理勉強会
https://sansan.connpass.com/event/195202/

■登壇概要
タイトル：自己教師あり学習の新しいアプローチ
発表者： 
ソフトウェアエンジニア 笹野 泰正 さん

▼Sansan R&D Twitter
https://twitter.com/SansanRandD

Transcript

1. ࣗݾڭࢣ͋Γֶशͷ৽͍͠Ξϓϩʔν
   SimSiam: Exploring Simple Siamese Representation Learning
   Facebook AI Research (FAIR)
   Yasumasa Sasano (@SquirrelYellow)
   

   View Slide

2. ࣗݾ঺հ
   ໊લ: ࡫໺ ହਖ਼
   ژ౎ͷاۀͰιϑτ΢ΣΞΤϯδχΞ
   ࢓ࣄͰ͸ࣗવݴޠॲཧ & ਂ૚ֶश
   झຯͰڧԽֶश
   (ֶੜ࣌୅ɺएख࣌୅͸ը૾ೝࣝ)
   ࣄલֶश͕ϒʔϜ
   #&35ɺ(15
   ڧԽֶशͰ΋
   ྨࣅͷݚڀ͕ߦΘΕ͍ͯΔ
   

   View Slide

3. ࣄલֶश͸ԿΛબͿ΂͖ʁ
   ੜ੒తख๏ (GANɺVAEɺࣗݾճؼɺFlowͳͲ)
   ର৅Λੜ੒Ͱ͖ΔͳΒಛ௃Λ͔ͭΊ͍ͯΔ͸ͣɻࣗવݴޠॲཧͳΒGPT-2ɻ
   [՝୊] ࣄલֶशʹ͸ΦʔόʔεϖοΫͰܭࢉίετ͕ߴ͍ɻ
   ิॿతͳख࡞Γͷ༧ଌλεΫ
   δάιʔύζϧ΍ࣗಈண৭ͳͲը૾͔Β؆୯ʹ࡞ΕΔλεΫͰֶशɻ
   ࣗવݴޠॲཧͳΒBERTɻ
   [՝୊] ରরֶश(ޙड़)΄Ͳͷੑೳ͸ग़ͳ͍
   

   View Slide

4. ࣄલֶशͷτϨϯυ: ࣗݾڭࢣ͋Γֶश
   ରরֶश (MoCoɺSimCLRͳͲ)
   ର৅ΛՃ޻ͨ͠ը૾(ਖ਼ͷڭࢣσʔλ)ͱɺର৅ͱແؔ܎ͷը૾(ෛͷڭࢣσʔλ)Λ
   ݟ෼͚ΒΕΕ͹ಛ௃Λ͔ͭΊ͍ͯΔ͸ͣɻݱ࣌఺Ͱͷ࠷ߴੑೳɻ
   [՝୊] ৭Μͳෛྫͱൺֱ͢ΔͨΊόοναΠζ͕େ͖͍ɻ௵Εͨղͷ໰୊(ޙड़)ɻ
   ྨࣅ౓ϕʔεͷख๏ (BYOLɺSimSiam)
   ର৅ΛՃ޻ͨ͠ը૾(ਖ਼ͷڭࢣσʔλ)͚͔ͩΒਖ਼ղσʔλΛ࡞Δ
   → όοναΠζͷ໰୊ͱ௵Εͨղͷ໰୊͕ղফʂ࣮༻Խʹ͍ۙͮͨʂ
   ϒʔτετϥοϓʹج͍ͮͨख๏ (Deep Cluster)
   Ϋϥελத৺Λٖࣅతͳڭࢣσʔλͱͯ͠༻͍Δɻ
   [՝୊]ΫϥελϦϯάͷ͕͔͔࣌ؒΓ͗͢Δɻ௵Εͨղͷ໰୊(ޙड़)
   

   View Slide

5. ྨࣅ౓ϕʔεͷख๏
   SimSiam
   

   View Slide

6. SimSiamͷϙΠϯτ
   ᶄ
   ᶅ
   ᶄ λʔήοτํ޲΁ͷޯ഑ఀࢭ
   ᶅ ༧ଌϨΠϠʔΛ௥Ճ
   ᶃ
   ᶃ 2ͭͷωοτϫʔΫʹରͯ͠ҟ
   ͳΔσʔλ֦ுΛࢪͯ͠ྨࣅ౓Λ
   ͚ۙͮΔΑ͏ֶश (ෛྫෆཁ)
   https://arxiv.org/pdf/2011.10566.pdf
   

   View Slide

7. ᶃσʔλ֦ு+ྨࣅ౓ֶश(Siamese Network)
   σʔλ֦ு
   ྨࣅ౓
   z1
   z2
   σʔλ֦ு ࣹӨಛ௃ྔ
   t2
   ∼
   t1
   ∼
   σʔλ֦ுख๏ΛϥϯμϜʹαϯϓϦϯά
   ҟͳΔࢹ఺Ͱݟͨ΋ͷΛ
   ಉ͡΋ͷʹͳΔΑ͏࠷దԽ
   https://arxiv.org/pdf/2002.05709.pdf
   fθ
   fθ
   ※͜ͷ··ͩͱ ʹऩଋ
   ྨࣅ౓ֶशʹ͸ͳΔ͕ɺ
   දݱֶशʹ͸޲͍͍ͯͳ͍
   z1
   = z2
   

   View Slide

8. ᶄλʔήοτํ޲΁ͷޯ഑ఀࢭ
   ޯ഑ఀࢭ(stop-gradient): ޡࠩٯ఻೻ͷޯ഑ܭࢉΛࢭΊΔؔ਺ɻGAN΍
   Actor-Clitic(ڧԽֶश)ͳͲͷख๏Ͱ࢖ΘΕΔ
   σʔλ֦ு ࣹӨಛ௃ྔ
   ྨࣅ౓
   Έ͍ͨͳࣗ໌ͳղʹऩଋ(௵Εͨղͷ໰୊)ରࡦཁҼͷҰͭ
   ͕ͩɺޯ഑ఀࢭͷΈͷରԠͰ͸௵ΕͨղΛճආ͢Δ͜ͱ͸Ͱ͖ͳ͍
   z1
   = z2
   z1
   z2
   t2
   ∼
   t1
   ∼
   ޯ഑ఀࢭ
   ✕
   fθ
   fθ
   

   View Slide

9. ᶅ༧ଌϨΠϠʔΛ௥Ճ
   σʔλ֦ு ࣹӨಛ௃ྔ
   ޯ഑ఀࢭ
   t2
   ∼
   t1
   ∼
   ✕
   p1
   ༧ଌಛ௃ྔ
   ྨࣅ౓
   z1
   z2
   fθ
   fθ hθ
   ࣹӨޙͷಛ௃ʹରͯ͠༧ଌϨΠϠʔΛ௥Ճ
   ༧ଌϨΠϠʔ͸ൺֱతઙ͍MLPΛ༻͍Δ
   hθ
   :
   → ޯ഑ఀࢭ + ༧ଌϨΠϠʔ͕ຊ࣭తͰ͋Δ͜ͱ͕࣮ݧతʹࣔ͞Εͨ
   

   View Slide

10. ख๏ͷ·ͱΊ
    ᶄ
    ᶅ
    ᶃ
    ↖૬ޓʹ࠷దԽ
    ↖ྨࣅ౓
    ↖λʔήοτ͸ޯ഑ఀࢭ
    

    View Slide

11. ख๏ͷ෼ੳ
    

    View Slide

12. సҠֶशͰImageNetࣄલֶशΛ௒͑Δੑೳ
    ͞Βʹطଘख๏ΑΓ΋ੑೳ͕ྑ͍
    

    View Slide

13. ޯ഑ఀࢭͷॏཁੑ
    ޯ഑ఀࢭ͕ͳ͍ͱਫ਼౓͕67.7%͔Β0.1%ʹμ΢ϯ
    

    View Slide

14. ༧ଌϨΠϠʔͷॏཁੑ
    ༧ଌϨΠϠʔ͕ͳ͍৔߹΋ਫ਼౓͕67.7%͔Β0.1%ʹμ΢ϯ
    

    View Slide

15. Ծઆ: ͳͥ͏·͍͘͘ͷ͔
    

    View Slide

16. Ծઆ: ༧ଌϨΠϠʔͱ͸Կͳͷ͔ʁ(1)
    σʔλ֦ு ࣹӨಛ௃ྔ
    t1
    ∼
    ྨࣅ౓
    z1
    fθ
    x
    ηx
    ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹ ݩը૾ / ݩը૾ ͷಛ௃දݱ
    ͱͨ͠ͱ͖ʹɺɹɹɹɹɹɹΛ࠷దԽ͢ΔΑ͏ͳEMΞϧΰϦζϜΛߟ͑Δ
    Eεςοϓ ࢑ఆͷಛ௃දݱ Λܭࢉ + Mεςοϓ Λ࠷దԽ ⇢ දݱֶश
    x : ηx
    : x
    ηx
    θ
    ʪ४උʫදݱֶशͷఆࣜԽ
    

    View Slide

17. Ծઆ: ༧ଌϨΠϠʔͱ͸Կͳͷ͔ʁ(2)
    ʪओுʫ༧ଌϨΠϠʔͱ͸࢑ఆͷಛ௃දݱ Λ༧ଌ͢ΔϨΠϠʔͰ͋Δɻ
    ηx
    σʔλ֦ு ࣹӨಛ௃ྔ
    ޯ഑ఀࢭ
    t2
    ∼
    t1
    ∼
    ✕
    p1
    ༧ଌಛ௃ྔ
    ྨࣅ౓
    z1
    z2
    fθ
    fθ hθ
    ͕ ʹରͯ͠࠷దԽ͞ΕΔͱ
    h
    ͱͳΓɺ͜Ε͸ ͷ࢑ఆతͳಛ௃දݱ ͱͯ͠࢖͏͜ͱ͕Ͱ͖Δɻ
    x ηx
    = ηx
    ͱ ͷ
    ྨࣅ౓Λ࠷దԽ
    ηx
    z2
    EMΞϧΰϦζϜͱͯ͠ߟ͑ΔͳΒMεςοϓͰ ͸ݻఆ͢Δඞཁ͕͋ΔͷͰޯ഑ఀࢭ͕ඞཁ
    ηx
    ⇢ ͱ ͷྨࣅ౓Λ࠷దԽ͢Δ͜ͱʹΑͬͯ ͕࠷దԽ͞ΕΔɻ
    ⇢ ͱ ͕ަޓʹ࠷దԽ͞Εɺಛ௃දݱ͕ՄೳʹͳΔɻ
    ηx
    z2
    θ
    ηx
    θ
    

    View Slide

18. ଞͷख๏ͱͷൺֱ
    Facebook AI Research (FAIR)
    

    View Slide

19. BYOL
    

    View Slide

20. SwAV
    Sinkhorn-Knopp
    Sinkhorn-Knopp
    

    View Slide

21. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
    

    View Slide