自己教師あり学習の新しいアプローチ / SimSiam: Exploring Simple Siamese Representation Learning

Slide 1

Slide 1 text

ࣗݾڭࢣ͋Γֶशͷ৽͍͠Ξϓϩʔν SimSiam: Exploring Simple Siamese Representation Learning Facebook AI Research (FAIR) Yasumasa Sasano (@SquirrelYellow)

Slide 2

Slide 2 text

ࣗݾ঺հ ໊લ: ࡫໺ ହਖ਼ ژ౎ͷاۀͰιϑτ΢ΣΞΤϯδχΞ ࢓ࣄͰ͸ࣗવݴޠॲཧ & ਂ૚ֶश झຯͰڧԽֶश (ֶੜ࣌୅ɺएख࣌୅͸ը૾ೝࣝ) ࣄલֶश͕ϒʔϜ #&35ɺ(15 ڧԽֶशͰ΋ ྨࣅͷݚڀ͕ߦΘΕ͍ͯΔ

Slide 3

Slide 3 text

ࣄલֶश͸ԿΛબͿ΂͖ʁ ੜ੒తख๏ (GANɺVAEɺࣗݾճؼɺFlowͳͲ) ର৅Λੜ੒Ͱ͖ΔͳΒಛ௃Λ͔ͭΊ͍ͯΔ͸ͣɻࣗવݴޠॲཧͳΒGPT-2ɻ [՝୊] ࣄલֶशʹ͸ΦʔόʔεϖοΫͰܭࢉίετ͕ߴ͍ɻ ิॿతͳख࡞Γͷ༧ଌλεΫ δάιʔύζϧ΍ࣗಈண৭ͳͲը૾͔Β؆୯ʹ࡞ΕΔλεΫͰֶशɻ ࣗવݴޠॲཧͳΒBERTɻ [՝୊] ରরֶश(ޙड़)΄Ͳͷੑೳ͸ग़ͳ͍

Slide 4

Slide 4 text

ࣄલֶशͷτϨϯυ: ࣗݾڭࢣ͋Γֶश ରরֶश (MoCoɺSimCLRͳͲ) ର৅ΛՃ޻ͨ͠ը૾(ਖ਼ͷڭࢣσʔλ)ͱɺର৅ͱແؔ܎ͷը૾(ෛͷڭࢣσʔλ)Λ ݟ෼͚ΒΕΕ͹ಛ௃Λ͔ͭΊ͍ͯΔ͸ͣɻݱ࣌఺Ͱͷ࠷ߴੑೳɻ [՝୊] ৭Μͳෛྫͱൺֱ͢ΔͨΊόοναΠζ͕େ͖͍ɻ௵Εͨղͷ໰୊(ޙड़)ɻ ྨࣅ౓ϕʔεͷख๏ (BYOLɺSimSiam) ର৅ΛՃ޻ͨ͠ը૾(ਖ਼ͷڭࢣσʔλ)͚͔ͩΒਖ਼ղσʔλΛ࡞Δ → όοναΠζͷ໰୊ͱ௵Εͨղͷ໰୊͕ղফʂ࣮༻Խʹ͍ۙͮͨʂ ϒʔτετϥοϓʹج͍ͮͨख๏ (Deep Cluster) Ϋϥελத৺Λٖࣅతͳڭࢣσʔλͱͯ͠༻͍Δɻ [՝୊]ΫϥελϦϯάͷ͕͔͔࣌ؒΓ͗͢Δɻ௵Εͨղͷ໰୊(ޙड़)

Slide 5

Slide 5 text

ྨࣅ౓ϕʔεͷख๏ SimSiam

Slide 6

Slide 6 text

SimSiamͷϙΠϯτ ᶄ ᶅ ᶄ λʔήοτํ޲΁ͷޯ഑ఀࢭ ᶅ ༧ଌϨΠϠʔΛ௥Ճ ᶃ ᶃ 2ͭͷωοτϫʔΫʹରͯ͠ҟ ͳΔσʔλ֦ுΛࢪͯ͠ྨࣅ౓Λ ͚ۙͮΔΑ͏ֶश (ෛྫෆཁ) https://arxiv.org/pdf/2011.10566.pdf

Slide 7

Slide 7 text

ᶃσʔλ֦ு+ྨࣅ౓ֶश(Siamese Network) σʔλ֦ு ྨࣅ౓ z1 z2 σʔλ֦ு ࣹӨಛ௃ྔ t2 ∼ t1 ∼ σʔλ֦ுख๏ΛϥϯμϜʹαϯϓϦϯά ҟͳΔࢹ఺Ͱݟͨ΋ͷΛ ಉ͡΋ͷʹͳΔΑ͏࠷దԽ https://arxiv.org/pdf/2002.05709.pdf fθ fθ ※͜ͷ··ͩͱ ʹऩଋ ྨࣅ౓ֶशʹ͸ͳΔ͕ɺ දݱֶशʹ͸޲͍͍ͯͳ͍ z1 = z2

Slide 8

Slide 8 text

ᶄλʔήοτํ޲΁ͷޯ഑ఀࢭ ޯ഑ఀࢭ(stop-gradient): ޡࠩٯ఻೻ͷޯ഑ܭࢉΛࢭΊΔؔ਺ɻGAN΍ Actor-Clitic(ڧԽֶश)ͳͲͷख๏Ͱ࢖ΘΕΔ σʔλ֦ு ࣹӨಛ௃ྔ ྨࣅ౓ Έ͍ͨͳࣗ໌ͳղʹऩଋ(௵Εͨղͷ໰୊)ରࡦཁҼͷҰͭ ͕ͩɺޯ഑ఀࢭͷΈͷରԠͰ͸௵ΕͨղΛճආ͢Δ͜ͱ͸Ͱ͖ͳ͍ z1 = z2 z1 z2 t2 ∼ t1 ∼ ޯ഑ఀࢭ ✕ fθ fθ

Slide 9

Slide 9 text

ᶅ༧ଌϨΠϠʔΛ௥Ճ σʔλ֦ு ࣹӨಛ௃ྔ ޯ഑ఀࢭ t2 ∼ t1 ∼ ✕ p1 ༧ଌಛ௃ྔ ྨࣅ౓ z1 z2 fθ fθ hθ ࣹӨޙͷಛ௃ʹରͯ͠༧ଌϨΠϠʔΛ௥Ճ ༧ଌϨΠϠʔ͸ൺֱతઙ͍MLPΛ༻͍Δ hθ : → ޯ഑ఀࢭ + ༧ଌϨΠϠʔ͕ຊ࣭తͰ͋Δ͜ͱ͕࣮ݧతʹࣔ͞Εͨ

Slide 10

Slide 10 text

ख๏ͷ·ͱΊ ᶄ ᶅ ᶃ ↖૬ޓʹ࠷దԽ ↖ྨࣅ౓ ↖λʔήοτ͸ޯ഑ఀࢭ

Slide 11

Slide 11 text

ख๏ͷ෼ੳ

Slide 12

Slide 12 text

సҠֶशͰImageNetࣄલֶशΛ௒͑Δੑೳ ͞Βʹطଘख๏ΑΓ΋ੑೳ͕ྑ͍

Slide 13

Slide 13 text

ޯ഑ఀࢭͷॏཁੑ ޯ഑ఀࢭ͕ͳ͍ͱਫ਼౓͕67.7%͔Β0.1%ʹμ΢ϯ

Slide 14

Slide 14 text

༧ଌϨΠϠʔͷॏཁੑ ༧ଌϨΠϠʔ͕ͳ͍৔߹΋ਫ਼౓͕67.7%͔Β0.1%ʹμ΢ϯ

Slide 15

Slide 15 text

Ծઆ: ͳͥ͏·͍͘͘ͷ͔

Slide 16

Slide 16 text

Ծઆ: ༧ଌϨΠϠʔͱ͸Կͳͷ͔ʁ(1) σʔλ֦ு ࣹӨಛ௃ྔ t1 ∼ ྨࣅ౓ z1 fθ x ηx ɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹ ݩը૾ / ݩը૾ ͷಛ௃දݱ ͱͨ͠ͱ͖ʹɺɹɹɹɹɹɹΛ࠷దԽ͢ΔΑ͏ͳEMΞϧΰϦζϜΛߟ͑Δ Eεςοϓ ࢑ఆͷಛ௃දݱ Λܭࢉ + Mεςοϓ Λ࠷దԽ ⇢ දݱֶश x : ηx : x ηx θ ʪ४උʫදݱֶशͷఆࣜԽ

Slide 17

Slide 17 text

Ծઆ: ༧ଌϨΠϠʔͱ͸Կͳͷ͔ʁ(2) ʪओுʫ༧ଌϨΠϠʔͱ͸࢑ఆͷಛ௃දݱ Λ༧ଌ͢ΔϨΠϠʔͰ͋Δɻ ηx σʔλ֦ு ࣹӨಛ௃ྔ ޯ഑ఀࢭ t2 ∼ t1 ∼ ✕ p1 ༧ଌಛ௃ྔ ྨࣅ౓ z1 z2 fθ fθ hθ ͕ ʹରͯ͠࠷దԽ͞ΕΔͱ h ͱͳΓɺ͜Ε͸ ͷ࢑ఆతͳಛ௃දݱ ͱͯ͠࢖͏͜ͱ͕Ͱ͖Δɻ x ηx = ηx ͱ ͷ ྨࣅ౓Λ࠷దԽ ηx z2 EMΞϧΰϦζϜͱͯ͠ߟ͑ΔͳΒMεςοϓͰ ͸ݻఆ͢Δඞཁ͕͋ΔͷͰޯ഑ఀࢭ͕ඞཁ ηx ⇢ ͱ ͷྨࣅ౓Λ࠷దԽ͢Δ͜ͱʹΑͬͯ ͕࠷దԽ͞ΕΔɻ ⇢ ͱ ͕ަޓʹ࠷దԽ͞Εɺಛ௃දݱ͕ՄೳʹͳΔɻ ηx z2 θ ηx θ