CVPR2020読み会 Proxy Anchor Loss for Deep Metric Learning

by SatoKeiju

Slide 1

Slide 1 text

CVPR2020ಡΈձ (2020/7/18)  sandglass Proxy Anchor Loss for  Deep Metric Learning

Slide 2

Slide 2 text

·ͣ͸ࣗݾ঺հ !2 • ໊લ: ࠤ౻ ܒथ Sato Keiju • Twitter: @TodayInsane

Slide 3

Slide 3 text

·ͣ͸ࣗݾ঺հ !3 • ໊લ: ࠤ౻ ܒथ Sato Keiju • Twitter: @TodayInsane K-POPେ޷͖ͳͷͰ Metric LearningͰ  PyTorchͱTWICEΛ஌Δ  ABEJAΞυϕϯτΧϨϯμʔ ॻ͖·ͨ͠

Slide 4

Slide 4 text

·ͣ͸ࣗݾ঺հ !4 • ໊લ: ࠤ౻ ܒथ Sato Keiju • Twitter: @TodayInsane • ॴଐ: ౦ژେֶిؾܥ޻ֶઐ߈ͷ৘ใܥ म࢜2೥ • ֶ෦: CMOSΞφϩάిࢠճ࿏ in ਆށେֶిి • גࣜձࣾABEJAͰ௕ظΠϯλʔϯ 2019/4 ʙ • Vue.jsͱ͔Firebaseͱ͔Goͱ͔Kubernetesͱ͔AWSͱ͔Deep Learningͱ͔Λϝϯλʔ͞Μํʹڭ͑ͯ΋Β͍·͍ͬͯ͘·͢

Slide 5

Slide 5 text

·ͣ͸ࣗݾ঺հ !5 • ໊લ: ࠤ౻ ܒथ Sato Keiju • Twitter: @TodayInsane • ॴଐ: ౦ژେֶిؾܥ޻ֶઐ߈ͷ৘ใܥ म࢜2೥ • ֶ෦: CMOSΞφϩάిࢠճ࿏ in ਆށେֶిి • גࣜձࣾABEJAͰ௕ظΠϯλʔϯ 2019/4 ʙ • Vue.jsͱ͔Firebaseͱ͔Goͱ͔Kubernetesͱ͔AWSͱ͔Deep Learningͱ͔Λϝϯλʔ͞Μํʹڭ͑ͯ΋Β͍·͍ͬͯ͘·͢

Slide 6

Slide 6 text

• ໊લ: ࠤ౻ ܒथ Sato Keiju • Twitter: @TodayInsane • ॴଐ: ౦ژେֶిؾܥ޻ֶઐ߈ͷ৘ใܥ म࢜2೥ • ֶ෦: CMOSΞφϩάిࢠճ࿏ in ਆށେֶిి • גࣜձࣾABEJAͰ௕ظΠϯλʔϯ 2019/4 ʙ • Vue.jsͱ͔Firebaseͱ͔Goͱ͔Kubernetesͱ͔AWSͱ͔Deep Learningͱ͔Λϝϯλʔ͞Μํʹڭ͑ͯ΋Β͍·͍ͬͯ͘·͢ ·ͣ͸ࣗݾ঺հ !6 ΤϯδχΞ: @xecus͞Μ ͕։ൃͨ͠ɺإೝূٕज़Λ࢖ͬͨ ΊͪΌͪ͘Ό໘ന͍ϓϩμΫτͷ ஀ੜൿ࿩ ݱࡏΠϯλʔϯͰ͓ख఻͍ͨ͠࿩ Λଓฤͱͯࣥ͠චதͰ͢

Slide 7

Slide 7 text

• ໊લ: ࠤ౻ ܒथ Sato Keiju • Twitter: @TodayInsane • ॴଐ: ౦ژେֶిؾܥ޻ֶઐ߈ͷ৘ใܥ म࢜2೥ • ֶ෦: CMOSΞφϩάిࢠճ࿏ in ਆށେֶిి • גࣜձࣾABEJAͰ௕ظΠϯλʔϯ 2019/4 ʙ • Vue.jsͱ͔Firebaseͱ͔Goͱ͔Kubernetesͱ͔AWSͱ͔Deep Learningͱ͔Λϝϯλʔ͞Μํʹڭ͑ͯ΋Β͍·͍ͬͯ͘·͢ ·ͣ͸ࣗݾ঺հ !7 Ϧαʔν: @peisuke͞Μ ʹ༏͘͠എதΛԡ͍͖ͯͨͩ͠ ຊ೔ਓੜॳొஃͰ͢

Slide 8

Slide 8 text

ΑΖ͓͘͠ئ͍͠·͢ !8

Slide 9

Slide 9 text

·ͣ͸͜ͷൃදͷΰʔϧΛ֬ೝʂ !9

Slide 10

Slide 10 text

ࠓ೔ͷΰʔϧ !10 ͷ͓ؾ࣋ͪΛཧղ͢Δ ঺հ࿦จΑΓҾ༻

Slide 11

Slide 11 text

ࠓ೔ͷΰʔϧ !11 ͷ͓ؾ࣋ͪΛཧղ͢Δ ԿͷֆͱԿͷࣜʁʁʁ

Slide 12

Slide 12 text

͓͠ͳ͕͖ • Metric Learningͬͯʁ • CVʹ͓͚ΔMetric LearningͷྲྀΕ • ຊ೔ͷ࿦จ঺հ !12 ͷ3ຊͰɺMetric Learningͷੈքͷུ஍ਤΛ͓౉͠Ͱ͖Ε͹ͱࢥ͍ͬͯ·͢ ͍ΖΜͳํͱ͍ΖΜͳ͓࿩͍ͨ͠ͷͰTwitterͳͲ͓ؾܰʹʂ͓ئ͍͠·͢ʂ

Slide 13

Slide 13 text

͓͠ͳ͕͖ • Metric Learningͬͯʁ • CVʹ͓͚ΔMetric LearningͷྲྀΕ • ຊ೔ͷ࿦จ঺հ !13

Slide 14

Slide 14 text

Metric Learning: σʔλͷڑ཭ࢦඪΛ֫ಘ͢Δ !14 Black footed Albatross Sooty Albatross

Slide 15

Slide 15 text

Metric Learning: σʔλͷڑ཭ࢦඪΛ֫ಘ͢Δ !15 Black footed Albatross Sooty Albatross Black footed Albatross Sooty Albatross • ಉ͡Ϋϥεͷσʔλಉ࢜͸͍ۙ • ҟͳΔΫϥεؒ͸཭Ε͍ͯΔ

Slide 16

Slide 16 text

Metric Learning: σʔλͷڑ཭ࢦඪΛ֫ಘ͢Δ !16 Black footed Albatross Sooty Albatross Black footed Albatross Sooty Albatross • ಉ͡Ϋϥεͷσʔλಉ࢜͸͍ۙ • ҟͳΔΫϥεؒ͸཭Ε͍ͯΔ σʔλ΍ ͦͷಛ௃ྔ ࣅͯΔͱ ۙ͘ͳΔ ۭؒ

Slide 17

Slide 17 text

Metric Learning: σʔλͷڑ཭ࢦඪΛ֫ಘ͢Δ !17 Black footed Albatross Sooty Albatross Black footed Albatross Sooty Albatross • ಉ͡Ϋϥεͷσʔλಉ࢜͸͍ۙ • ҟͳΔΫϥεؒ͸཭Ε͍ͯΔ σʔλ΍ ͦͷಛ௃ྔ ࣅͯΔͱ ۙ͘ͳΔ ۭؒ ͜ΕΒΛ֫ಘ͢Δͷ͕Metric(ࢦඪ)Learning(ֶश)

Slide 18

Slide 18 text

͸͡·Γ͸ม׵ߦྻ(ϚϋϥϊϏεڑ཭)͔Β… !18

Slide 19

Slide 19 text

͸͡·Γ͸ม׵ߦྻ(ϚϋϥϊϏεڑ཭)͔Β… !19 ม਺ಉ࢜ͷڞ෼ࢄߦྻ • ֤࣍ݩͷ෼ࢄΛߟྀͨ͠Ұൠతͳڑ཭ • ൒ਖ਼ఆ஋ߦྻͰ͋Ε͹͞Βʹɹɹɹɹɹɹɹɹɹɹɹɹɹɹɹͱม׵  ͭ·ΓσʔλؒͷϢʔΫϦουڑ཭͕ద੾ʹͳΔΑ͏ͳม׵LΛֶश  →ྨࣅσʔλಉ࢜͸ۙ͘ɺඇྨࣅσʔλಉ࢜ͷڑ཭͸ԕ͘͢Δ  ɹ࠷దԽ໰୊

Slide 20

Slide 20 text

Deep LearningͷൃలͰMetric Learning΋Ξπ͍ !20 Channel Width Height ը૾: H × W × C࣍ݩσʔλ

Slide 21

Slide 21 text

Deep LearningͷൃలͰMetric Learning΋Ξπ͍ !21 Channel Width Height ը૾: H × W × C࣍ݩσʔλ Hand-craftedͳಛ௃ྔઃܭ x1 x2 : : : : : : : :  : xN N࣍ݩಛ௃ϕΫτϧ = N࣍ݩۭؒ಺ͷ఺

Slide 22

Slide 22 text

Deep LearningͷൃలͰMetric Learning΋Ξπ͍ !22 Channel Width Height ը૾: H × W × C࣍ݩσʔλ Hand-craftedͳಛ௃ྔઃܭ x1 x2 : : : : : : : :  : xN N࣍ݩಛ௃ϕΫτϧ = N࣍ݩۭؒ಺ͷ఺ Deep CNN

Slide 23

Slide 23 text

Deep LearningͷൃలͰMetric Learning΋Ξπ͍ !23 Channel Width Height ը૾: H × W × C࣍ݩσʔλ Hand-craftedͳಛ௃ྔઃܭ x1 x2 : : : : : : : :  : xN N࣍ݩಛ௃ϕΫτϧ = N࣍ݩۭؒ಺ͷ఺ Deep CNN ࣮ݱ͍ۭͨؒ͠ͷڑ཭తͳ ͓ؾ͕࣋ͪ࿅Γ͜·ΕͨLossؔ਺ ֶश

Slide 24

Slide 24 text

ը૾ೝࣝ: ֤Ϋϥε͕·ͱ·Δಛ௃ྔۭؒͰࣝผ !24 ಉ͡Ϋϥεϥϕϧͷ͍ͭͨը૾͸ۙ͘ ҧ͏Ϋϥε͸཭ΕͯຒΊࠐ·ΕΔ

Slide 25

Slide 25 text

ը૾ೝࣝ: ֤Ϋϥε͕·ͱ·Δಛ௃ྔۭؒͰࣝผ !25 https://twitter.com/sanacottoncandy/status/1131140431775801345 https://twitter.com/WinterFrame/status/1200973640197492736 https://twitter.com/JYPETWICE/status/1272053602790502400 ಉ͡Ϋϥεϥϕϧͷ͍ͭͨը૾͸ۙ͘ ҧ͏Ϋϥε͸཭ΕͯຒΊࠐ·ΕΔ

Slide 26

Slide 26 text

ը૾ೝࣝ: ֤Ϋϥε͕·ͱ·Δಛ௃ྔۭؒͰࣝผ !26 ಉ͡Ϋϥεϥϕϧͷ͍ͭͨը૾͸ۙ͘ ҧ͏Ϋϥε͸཭ΕͯຒΊࠐ·ΕΔ ʮಉ͡Ϋϥεͱ൑அ͢Δڑ཭ͷᮢ஋ʯΛ ͍͍ײ͡ʹઃఆ͢Δ ը૾෼ྨ αφͪΌΜ π΢ΟͪΌΜ

Slide 27

Slide 27 text

ը૾ೝࣝ: ֤Ϋϥε͕·ͱ·Δಛ௃ྔۭؒͰࣝผ !27 ಉ͡Ϋϥεϥϕϧͷ͍ͭͨը૾͸ۙ͘ ҧ͏Ϋϥε͸཭ΕͯຒΊࠐ·ΕΔ ʮಉ͡Ϋϥεͱ൑அ͢Δڑ཭ͷᮢ஋ʯΛ ͍͍ײ͡ʹઃఆ͢Δ ը૾෼ྨ (ओʹϢʔΫϦουڑ཭Ͱ)ΫΤϦը૾ͷ  ຒΊࠐΈ఺ʹ͍ۙॱʹྨࣅը૾ͱ͢Δ ը૾ݕࡧ ΫΤϦ Top-1(࠷ۙ๣) https://twitter.com/JYPETWICE_JAPAN/status/1156761954024734721

Slide 28

Slide 28 text

ը૾ೝࣝ: ֤Ϋϥε͕·ͱ·Δಛ௃ྔۭؒͰࣝผ !28 ಉ͡Ϋϥεϥϕϧͷ͍ͭͨը૾͸ۙ͘ ҧ͏Ϋϥε͸཭ΕͯຒΊࠐ·ΕΔ ʮಉ͡Ϋϥεͱ൑அ͢Δڑ཭ͷᮢ஋ʯΛ ͍͍ײ͡ʹઃఆ͢Δ ը૾෼ྨ (ओʹϢʔΫϦουڑ཭Ͱ)ΫΤϦը૾ͷ  ຒΊࠐΈ఺ʹ͍ۙॱʹྨࣅը૾ͱ͢Δ ը૾ݕࡧ ෼ྨʹՃ͑ͯະ஌ͷΫϥε΋ ʮະ஌ͷΫϥεʯͱͯ͠ରԠͰ͖Δ ը૾ࣝผ https://twitter.com/JYPETWICE_JAPAN/status/1156761692547629056 New?

Slide 29

Slide 29 text

ը૾ೝࣝ: ֤Ϋϥε͕·ͱ·Δಛ௃ྔۭؒͰࣝผ !29 ະ஌ͷΫϥε΋ʮະ஌ͷΫϥεʯͱͯ͠ରԠͰ͖Δ

Slide 30

Slide 30 text

ը૾ೝࣝ: ֤Ϋϥε͕·ͱ·Δಛ௃ྔۭؒͰࣝผ !30 ະ஌ͷΫϥε΋ʮະ஌ͷΫϥεʯͱͯ͠ରԠͰ͖Δ ʮֶश࣌ʹແ͍Ϋϥε͕ਪ࿦ʹग़ͯ͘Δʯ՝୊ઃఆʹ˕ • Face Recognition / Verification • Person Re-Identification • Image Retrieval ͳͲͳͲ...

Slide 31

Slide 31 text

!31 ଞʹ΋ Metric Learning for…

Slide 32

Slide 32 text

Adversarial Attack(ݸਓతʹ໘ന͔ͬͨͷͰ͝঺հ) !32 “Metric Learning for Adversarial Robustness”(NeurIPS’19) Adversarial Example͸ ಛ௃ྔۭؒʹ͓͍ͯ Ϋϥεͷڥք෇ۙʹҐஔ͍ͯ͠Δʁ(͔Βؒҧ͑Δʁ) Adversarial ExampleΛ ຊ౰ͷΫϥεʹ͍ۙํ޲ʹ ಈ͔͢Loss ը૾ͱAdversarialͳը૾Λಛ௃ྔۭؒʹ౤Ө͢Δͱ…ʁ

Slide 33

Slide 33 text

!33 “OpenGAN: Open Set Generative Adversarial Networks”(arXiv 2020/5/18) ҟৗݕ஌ “Unsupervised Anomaly Detection with Generative Adversarial Networks  to Guide Marker Discovery”(IPMI’17) GAN Distillation “Relational Knowledge Distillation”(CVPR’19)

Slide 34

Slide 34 text

౳ʑ… ͋͘·Ͱ͘͝Ұ෦ͷ࿩͔͍ͭܰ঺հͰ͢ͷͰ ͍Ζ͍Ζ୳ͨ͠Γࢼͨ͠Γͯ͠Έ͍ͯͩ͘͞ʂ !34

Slide 35

Slide 35 text

͓͠ͳ͕͖ • Metric Learningͬͯʁ • CVʹ͓͚ΔMetric LearningͷྲྀΕ • ຊ೔ͷ࿦จ঺հ !35

Slide 36

Slide 36 text

࢝૆: 2ͭͷσʔλ఺Λ͚ۙͮΔ/཭͢Contrastive Loss !36

Slide 37

Slide 37 text

࢝૆: 2ͭͷσʔλ఺Λ͚ۙͮΔ/཭͢Contrastive Loss !37 Deep CNN x1 x2 : : : : : : : :  : xN x1 x2 : : : : : : : :  : xN ಉ͡Ϋϥεͷը૾

Slide 38

Slide 38 text

࢝૆: 2ͭͷσʔλ఺Λ͚ۙͮΔ/཭͢Contrastive Loss !38 Deep CNN x1 x2 : : : : : : : :  : xN x1 x2 : : : : : : : :  : xN ಉ͡Ϋϥεͷը૾

Slide 39

Slide 39 text

࢝૆: 2ͭͷσʔλ఺Λ͚ۙͮΔ/཭͢Contrastive Loss !39 Deep CNN x1 x2 : : : : : : : :  : xN x1 x2 : : : : : : : :  : xN ΑΓ ۙͮ͘ ಉ͡Ϋϥεͷը૾

Slide 40

Slide 40 text

࢝૆: 2ͭͷσʔλ఺Λ͚ۙͮΔ/཭͢Contrastive Loss !40 Deep CNN x1 x2 : : : : : : : :  : xN x1 x2 : : : : : : : :  : xN ҧ͏Ϋϥεͷը૾

Slide 41

Slide 41 text

࢝૆: 2ͭͷσʔλ఺Λ͚ۙͮΔ/཭͢Contrastive Loss !41 Deep CNN x1 x2 : : : : : : : :  : xN x1 x2 : : : : : : : :  : xN ҧ͏Ϋϥεͷը૾

Slide 42

Slide 42 text

࢝૆: 2ͭͷσʔλ఺Λ͚ۙͮΔ/཭͢Contrastive Loss !42 Deep CNN x1 x2 : : : : : : : :  : xN x1 x2 : : : : : : : :  : xN ΑΓ ཭ΕΔ ҧ͏Ϋϥεͷը૾

Slide 43

Slide 43 text

࢝૆: 2ͭͷσʔλ఺Λ͚ۙͮΔ/཭͢Contrastive Loss !43 Deep CNN x1 x2 : : : : : : : :  : xN x1 x2 : : : : : : : :  : xN ҧ͏Ϋϥεͷը૾ “Dimensionality Reduction by Learning an Invariant Mapping” (CVPR’06)

Slide 44

Slide 44 text

࢝૆ͷࢠ(ʁ): 3ͭͷσʔλ఺Λ͚ۙͮΔ/཭͢Triplet Loss !44 [~]+: ώϯδؔ਺(= max(~, 0))

Slide 45

Slide 45 text

࢝૆ͷࢠ(ʁ): 3ͭͷσʔλ఺Λ͚ۙͮΔ/཭͢Triplet Loss !45 Deep CNN anchor: ͋Δσʔλ positive: anchorͱ ಉΫϥε negative: ผͷΫϥε [~]+: ώϯδؔ਺(= max(~, 0)) negative anchor positive

Slide 46

Slide 46 text

࢝૆ͷࢠ(ʁ): 3ͭͷσʔλ఺Λ͚ۙͮΔ/཭͢Triplet Loss !46 Deep CNN anchor: ͋Δσʔλ positive: anchorͱ ಉΫϥε negative: ผͷΫϥε [~]+: ώϯδؔ਺(= max(~, 0)) anchor positive negative dn dp dp ʹϚʔδϯαΛ଍ͯ͠΋ ·ͩdn ͷํ͕σΧ͘ͳͬͯ΄͍͠ • จ຺(ϥϕϧͱλεΫ)ʹؔ܎ͳ͘  ૬ରతʹۙ͘/ԕ͘͢Ε͹Α͍ • ૊Έ߹Θͤ਺͕๲େͰ  ֶश͕ऩଋ͠ͳ͍…

Slide 47

Slide 47 text

࢝૆ͷࢠ(ʁ): 3ͭͷσʔλ఺Λ͚ۙͮΔ/཭͢Triplet Loss !47 Deep CNN anchor: ͋Δσʔλ positive: anchorͱ ಉΫϥε negative: ผͷΫϥε [~]+: ώϯδؔ਺(= max(~, 0)) anchor positive negative dn dp dpʹϚʔδϯαΛ଍ͯ͠΋ dnΑΓখ͘͢͞Δ • จ຺(ϥϕϧͱλεΫ)ʹؔ܎ͳ͘  ૬ରతʹۙ͘/ԕ͘͢Ε͹Α͍ • ૊Έ߹Θͤ਺͕๲େͰ  ֶश͕ऩଋ͠ͳ͍… “Learning Fine-grained Image Similarity with Deep Ranking” (CVPR’14)

Slide 48

Slide 48 text

Contrastive / Triplet Loss͔Β೿ੜ→੝Γ্͕͖ͬͯͨ • ֶशʹ༗ޮͳϖΞΛ୳͢/࡞Δ • ͞Βʹߟྀ͢ΔΫϥε਺Λ֦ு • ۭؒΛ௒ٿ໘্ʹ੍໿ • ڑ཭ࢦඪΛ֯౓Ͱ • ୅ද఺ͰMetric Learning • Ϋϥε෼ྨܕωοτϫʔΫߏ଄ !48 ͳͲͳͲ…

Slide 49

Slide 49 text

Contrastive / Triplet Loss͔Β೿ੜ→੝Γ্͕͖ͬͯͨ !49 “FaceNet: A Unified Embedding for Face Recognition and Clustering”(CVPR’15) !49 !49 • ֶशʹ༗ޮͳϖΞΛ୳͢/࡞Δ • ͞Βʹߟྀ͢ΔΫϥε਺Λ֦ு • ۭؒΛ௒ٿ໘্ʹ੍໿ • ڑ཭ࢦඪΛ֯౓Ͱ • ୅ද఺ͰMetric Learning • Ϋϥε෼ྨܕωοτϫʔΫߏ଄ Tripletͷ૊Έ߹Θͤʹ͍ͭͯͷʮ೉͠͞ʯΛఆٛ Loss͕େ͖͘ͳΔSemi-hard NegativeϖΞΛޮ཰Α͘࡞੒

Slide 50

Slide 50 text

Contrastive / Triplet Loss͔Β೿ੜ→੝Γ্͕͖ͬͯͨ !50 “Improved Deep Metric Learning with Multi-class N-pair Loss Objective”(NIPS’16) NΫϥε͔Β2ը૾ͣͭ  औΓग़ͯ͠ಛ௃ྔʹม׵ N૊ͷηοτΛ࡞Γ  LossΛܭࢉ • ֶशʹ༗ޮͳϖΞΛ୳͢/࡞Δ • ͞Βʹߟྀ͢ΔΫϥε਺Λ֦ு • ۭؒΛ௒ٿ໘্ʹ੍໿ • ڑ཭ࢦඪΛ֯౓Ͱ • ୅ද఺ͰMetric Learning • Ϋϥε෼ྨܕωοτϫʔΫߏ଄

Slide 51

Slide 51 text

Contrastive / Triplet Loss͔Β೿ੜ→੝Γ্͕͖ͬͯͨ • ֶशʹ༗ޮͳϖΞΛ୳͢/࡞Δ • ͞Βʹߟྀ͢ΔΫϥε਺Λ֦ு • ۭؒΛ௒ٿ໘্ʹ੍໿ • ڑ཭ࢦඪΛ֯౓Ͱ • ୅ද఺ͰMetric Learning • Ϋϥε෼ྨܕωοτϫʔΫߏ଄ !51 “L2-constrained Softmax Loss for Discriminative Face Verification”(arXiv 2017/3) ໌ࣔతͳڑ཭࠷దԽ͸͠ͳ͍͕ ಛ௃ྔϕΫτϧΛϊϧϜͰׂͬͯ ಛ௃ྔ࣍ݩۭؒͷٿ໘্ʹ ಺ੵ(cosྨࣅ౓)Ͱදݱྗͷௐ੔ ֶशͷ೉қ౓΋εέʔϦϯάύϥϝʔλαͰௐ੔

Slide 52

Slide 52 text

Contrastive / Triplet Loss͔Β೿ੜ→੝Γ্͕͖ͬͯͨ !52 “Deep Metric Learning with Angular Loss”(ICCV’17) anchorɺpositiveɺ negativeΛಈ͔͢ ޲͖(ޯ഑)Λ ֯౓͔Βߟ͑Δ • ֶशʹ༗ޮͳϖΞΛ୳͢/࡞Δ • ͞Βʹߟྀ͢ΔΫϥε਺Λ֦ு • ۭؒΛ௒ٿ໘্ʹ੍໿ • ڑ཭ࢦඪΛ֯౓Ͱ • ୅ද఺ͰMetric Learning • Ϋϥε෼ྨܕωοτϫʔΫߏ଄

Slide 53

Slide 53 text

Contrastive / Triplet Loss͔Β೿ੜ→੝Γ্͕͖ͬͯͨ !53 • ֶशʹ༗ޮͳϖΞΛ୳͢/࡞Δ • ͞Βʹߟྀ͢ΔΫϥε਺Λ֦ு • ۭؒΛ௒ٿ໘্ʹ੍໿ • ڑ཭ࢦඪΛ֯౓Ͱ • ୅ද఺ͰMetric Learning • Ϋϥε෼ྨܕωοτϫʔΫߏ଄ “No Fuss Distance Metric Learning using proxies”(ICCV’17) positiveͱnegativeʹ ݸผͷσʔλ఺Ͱ͸ͳ͘ ֤Ϋϥεͷ୅ද஋ͷ ຒΊࠐΈϕΫτϧ(proxy) Λ༻͍Δ ѹ౗తऩଋͷ଎͞ ຊ೔ͷ࿦จͷϥΠόϧ

Slide 54

Slide 54 text

Contrastive / Triplet Loss͔Β೿ੜ→੝Γ্͕͖ͬͯͨ !54 • ֶशʹ༗ޮͳϖΞΛ୳͢/࡞Δ • ͞Βʹߟྀ͢ΔΫϥε਺Λ֦ு • ۭؒΛ௒ٿ໘্ʹ੍໿ • ڑ཭ࢦඪΛ֯౓Ͱ • ୅ද఺ͰMetric Learning • Ϋϥε෼ྨܕωοτϫʔΫߏ଄ “ArcFace: Additive Angular Margin Loss for Deep Face Recognition”(CVPR’19) ಛ௃ྔϕΫτϧͷޙஈʹશ݁߹૚Λ઀ଓ Softmaxؔ਺ͱCross Entropy LossͰ Metric LearningΛߦ͏(ਪ࿦࣌͸શ݁߹૚Λ֎͢) ຊ೔ͷ࿦จͷϥΠόϧ  Ͱ͸ͳ͍͕ඞಡʂ

Slide 55

Slide 55 text

͓·͚: CVPR’20ͰMetric Learningͷ࿩ͯ͠Δ࿦จ(ͷҰ෦) !55 •Deep Metric Learning via Adaptive Learnable Assessment •End-to-End Illuminant Estimation Based on Deep Metric Learning •Cross-Batch Memory for Embedding Learning •Optimizing Rank-Based Metrics With Blackbox Differentiation •Universal Weighting Metric Learning for Cross-Modal Matching •Fast(er) Reconstruction of Shredded Text Documents via  Self-Supervised Deep Asymmetric Metric Learning •Moving in the Right Direction: A Regularization for Deep Metric Learning •CurricularFace: Adaptive Curriculum Learning Loss for Deep Face Recognition •Circle Loss: A Unified Perspective of Pair Similarity Optimization

Slide 56

Slide 56 text

౳ʑ… ͋͘·Ͱ͘͝Ұ෦ͷ࿩͔͍ͭܰ঺հͰ͢ͷͰ ͍Ζ͍Ζ୳ͨ͠Γࢼͨ͠Γͯ͠Έ͍ͯͩ͘͞ʂ (࠶ܝ) !56

Slide 57

Slide 57 text

ࠓ೔ͷΰʔϧ(࠶ܝ) !57 ͷ͓ؾ࣋ͪΛཧղ͢Δ

Slide 58

Slide 58 text

ࠓ೔ͷΰʔϧ(࠶ܝ) !58 Triplet Loss • anchor: ͋Δσʔλ఺ • positive: anchorͱಉ͡Ϋϥεʹ ଐ͢Δผͷσʔλ఺ • negative: anchorɺpositiveͱ͸ ҟͳΔΫϥεͷσʔλ఺ ↓ • anchorͱpositive͕͖ۙͮɺ anchorͱnegative͕཭ΕΔLoss • σʔλͷ૊Έ߹Θͤ૯਺ଟ͗͢ • ֶशʹޮ͘ϖΞʹ౰ͨΓʹ͍͘

Slide 59

Slide 59 text

ࠓ೔ͷΰʔϧ(࠶ܝ) !59 N-pair Loss • anchor: ͋Δσʔλ఺ • positive: anchorͱಉ͡Ϋϥεʹ ଐ͢Δผͷσʔλ఺ • negatives: anchorɺpositiveͰ ͳ͍Ϋϥεͷσʔλ఺(1ͭͣͭ) • ۙ͞Λߟྀ ↓ • TripletΑΓ΋Ұ౓ʹશମΛݟΕΔ • ·ͩσʔλશମͱ͸ݴ͑ͳ͍  (ϛχόονͷதͷ͞ΒʹҰ෦)

Slide 60

Slide 60 text

ࠓ೔ͷΰʔϧ(࠶ܝ) !60 Lifted Structure Loss • anchor: ͋Δσʔλ఺ • positive: anchorͱಉ͡Ϋϥεʹ ଐ͢Δผͷσʔλ఺ • negatives: anchorɺpositiveͰ ͳ͍Ϋϥεͷશσʔλ఺ ↓ • N-pairΑΓ΋Ұ౓ʹશମΛݟΕΔ • ·ͩσʔλશମͱ͸ݴ͑ͳ͍  (anchorΫϥεͷҰ෦σʔλ΍ϛ χόον಺ͷΈͰݶఆత)

Slide 61

Slide 61 text

ࠓ೔ͷΰʔϧ(࠶ܝ) !61 Proxy-NCA Loss • anchor: ͋Δσʔλ఺ • proxy: ֤Ϋϥεʹ͖ͭ1ͭ͋Δ ʮ୅ද఺ʯ ↓ • anchorͱશproxyͰ  N-pair Lossతͳֶश • ऩଋ͕ΊͪΌͪ͘Ό଎͍ • σʔλશମͷۭؒత৘ใ͸  ׆͔͖͠Εͯͳ͍

Slide 62

Slide 62 text

ࠓ೔ͷΰʔϧ(࠶ܝ) !62 ࢒͢͸ຊ೔ͷ࿦จ Proxy Anchor Loss ͷΈ

Slide 63

Slide 63 text

ࠓ೔ͷΰʔϧ(࠶ܝ) !63 Proxy Anchor Loss • anchor: ͋ΔΫϥεͷproxy • positive: anchorͷproxyʹଐ͢ ΔΫϥεͷϛχόον಺શσʔλ ఺ • negative: anchorͱҟͳΔΫϥε ͷϛχόον಺શσʔλ఺ ↓ • ऩଋ͕͞ΒʹΊͪΌͪ͘Ό଎͍ • σʔλશମͷۭؒత৘ใΛ׆͔͠ ·͍ͬͯ͘Δ

Slide 64

Slide 64 text

ࠓ೔ͷΰʔϧ(࠶ܝ) !64 ͍͟ղઆ΁

Slide 65

Slide 65 text

͓͠ͳ͕͖ • Metric Learningͬͯʁ • CVʹ͓͚ΔMetric LearningͷྲྀΕ • ຊ೔ͷ࿦จ঺հ !65

Slide 66

Slide 66 text

ఏҊͯ͠Δ಺༰: Proxy Anchor Loss !66

Slide 67

Slide 67 text

ఏҊͯ͠Δ಺༰: Proxy Anchor Loss α: εέʔϦϯάύϥϝʔλ δ: Ϛʔδϯ ϛχόον಺ͷσʔλ શΫϥε෼ͷproxyͨͪͱ ͦͷݸ਺ proxy pͷΫϥεʹଐ͢Δ ϛχόον಺ͷશembedding (σʔλಛ௃ྔ)ϕΫτϧ embeddingϕΫτϧxͱ ରԠ͢ΔΫϥεͷproxyϕΫτϧpͷ ྨࣅ౓(ຊ࿦จͰ͸cosྨࣅ౓) શproxyͱ ͦͷݸ਺(=Ϋϥε਺) proxy pͷΫϥεʹଐ͞ͳ͍ ϛχόον಺ͷશembedding (σʔλಛ௃ྔ)ϕΫτϧ

Slide 68

Slide 68 text

ఏҊͯ͠Δ಺༰: Proxy Anchor Loss α: εέʔϦϯάύϥϝʔλ δ: Ϛʔδϯ ϛχόον಺ͷσʔλ શΫϥε෼ͷproxyͨͪͱ ͦͷݸ਺ proxy pͷΫϥεʹଐ͢Δ શembedding(σʔλಛ௃ྔ) ϕΫτϧ embeddingϕΫτϧxͱ ରԠ͢ΔΫϥεͷproxyϕΫτϧpͷ ྨࣅ౓(ຊ࿦จͰ͸cosྨࣅ౓) શproxyͱ ͦͷݸ਺(=Ϋϥε਺) proxy pͷΫϥεʹଐ͞ͳ͍ શembedding(σʔλಛ௃ྔ)ϕΫτϧ ʁ

Slide 69

Slide 69 text

Proxy Anchor Lossͷ͓ؾ࣋ͪΛཧղ͍ͨ͠ !69

Slide 70

Slide 70 text

Proxy Anchor Lossͷ͓ؾ࣋ͪΛཧղ͍ͨ͠ !70 “easier-to-interpret form”

Slide 71

Slide 71 text

“easier-to-interpret form” Proxy Anchor Lossͷ͓ؾ࣋ͪΛཧղ͍ͨ͠ !71 “smooth approximation of ReLU” ͭ·Γඍ෼Մೳ࿈ଓͳmax(0, z)

Slide 72

Slide 72 text

“easier-to-interpret form” Proxy Anchor Lossͷ͓ؾ࣋ͪΛཧղ͍ͨ͠ !72 “smooth approximation of max function” LogSumExp

Slide 73

Slide 73 text

“easier-to-interpret form” Proxy Anchor Lossͷ͓ؾ࣋ͪ !73 ͋Δproxy pͱhardest positive (p͔ΒҰ൪ԕ͍positive)͕ۙͮ͘ ͋Δproxy pͱhardest negative (p͔ΒҰ൪͍ۙnegative)͕཭ΕΔ “smooth approximation of max function”

Slide 74

Slide 74 text

Proxy Anchor Lossͷਅͷ͓ؾ࣋ͪ !74

Slide 75

Slide 75 text

Proxy Anchor Lossͷਅͷ͓ؾ࣋ͪ !75 proxyΛanchor(ج४)ʹͯ͠ positive΋negative΋ શͯͷྨࣅ౓ݟͯΔɺͷਤ

Slide 76

Slide 76 text

Proxy Anchor Lossͷਅͷ͓ؾ࣋ͪ !76 anchorͱ֤positiveͱͷྨࣅ౓͕ ߴ͍΄Ͳখ͘͞ͳͬͯخ͍߲͠ proxyΛanchor(ج४)ʹͯ͠ positive΋negative΋ શͯͷྨࣅ౓ݟͯΔɺͷਤ anchorͱ֤negativeͱͷྨࣅ౓͕ ௿͍΄Ͳখ͘͞ͳͬͯخ͍߲͠

Slide 77

Slide 77 text

!77 ͲΜͳͱ͜Ζ͕ྑ͘ͳͬͯΔʁ

Slide 78

Slide 78 text

࿦จᐌ͘1: ੑೳྑͯ͘ऩଋ଎͍ʂ !78

Slide 79

Slide 79 text

࿦จᐌ͘1: ੑೳྑͯ͘ऩଋ଎͍ʂ !79 Recall@1είΞ͕  Ұ൪ߴ͍ʂ ऩଋ͕଎͍ʂ

Slide 80

Slide 80 text

࿦จᐌ͘1: ੑೳྑͯ͘ऩଋ଎͍ʂ !80 σʔλͷۭؒతߏ଄΍෼෍Λ  Α͘ଊֶ͑ͨश͕Ͱ͖͍ͯΔ Recall@1είΞ͕  Ұ൪ߴ͍ʂ ऩଋ͕଎͍ʂ

Slide 81

Slide 81 text

࿦จᐌ͘1: ੑೳྑͯ͘ऩଋ଎͍ʂ !81 ୅ද఺Λ࢖͍ͳ͕Β σʔλશମ΋ݟΕ͍ͯΔ Recall@1είΞ͕  Ұ൪ߴ͍ʂ ऩଋ͕଎͍ʂ

Slide 82

Slide 82 text

࿦จᐌ͘1: ੑೳྑͯ͘ऩଋ଎͍ʂ !82 Recall@1είΞ͕  Ұ൪ߴ͍ʂ ऩଋ͕଎͍ʂ Recall@KείΞ • Retrieval(ݕࡧ)λεΫ΍Metric LearningͷੑೳΛଌΔͷʹΑ͘࢖ΘΕΔ • ֶशͨ͠ม׵(ωοτϫʔΫ)ʹςετσʔλΛશͯೖྗͯ͠embeddingԽͨ࣌͠ʹ  ֤σʔλ఺ʹ͍ͭͯɺͦͷۭؒ಺Ͱ͍ۙॱϥϯΩϯάTop-KҐͷதʹ  ಉ͡Ϋϥεͷσʔλ఺͕͋ΔΑ͏ͳ఺ͷׂ߹ • ྫ͑͹Recall@2ͳΒɺRecall@1(࠷ۙ๣఺͕ಉ͡ΫϥεͰ͋ΔΑ͏ͳ఺ͷׂ߹)ʹ  ʮ࠷دΓ͸ผΫϥεͷσʔλ఺͕ͩ2൪໨ʹ͍ۙ఺ͳΒಉ͡ΫϥεʯͰ͋ΔϠπΒΛ଍ͨ͠  είΞͱͳΔ • ʮະ஌ΫϥεΛ͖ͪΜͱ෼཭Ͱ͖Δ͔ʯΛଌΔͨΊɺσʔληοτΛ2෼ׂͯ͠  ֶशͱςετΛ׬શʹผͷΫϥεͰߦ͏(ͳ͔ͳ͔ϋʔυ)

Slide 83

Slide 83 text

࿦จᐌ͘2: ྨࣅ౓ʹԠͯ͡஋͕ͪΌΜͱมΘΔʂ !83 Proxy-NCA Loss Proxy Anchor Loss positive proxyʹ͍ͭͯ มԽ͕ݻఆ શσʔλ఺ʹ͍ͭͯ hard͞ʹԠֶͨ͡श

Slide 84

Slide 84 text

࿦จᐌ͘3: ૊Έ߹Θͤ਺͕(طଘݚڀͷதͰ)࠷খʂ !84 M: ֶशσʔλͷ૯਺ C: ֶशσʔλͷΫϥε਺ B: 1epochͷόον਺ U: 1Ϋϥε͋ͨΓͷproxy਺

Slide 85

Slide 85 text

Ϋϥε਺C << σʔλ૯਺M ࿦จᐌ͘3: ૊Έ߹Θͤ਺͕(طଘݚڀͷதͰ)࠷খʂ !85 M: ֶशσʔλͷ૯਺ C: ֶशσʔλͷΫϥε਺ B: 1epochͷόον਺ U: 1Ϋϥε͋ͨΓͷproxy਺ ϙΠϯτ

Slide 86

Slide 86 text

࿦จᐌ͘3: ૊Έ߹Θͤ਺͕(طଘݚڀͷதͰ)࠷খʂ !86 ૊Έ߹Θͤ૯਺͕ଟ͗ͯ͢  ֶशʹ༗ޮͳσʔλϖΞΛ Ҿ͖౰ͯΔͷʹ͔͔࣌ؒΔ Ϋϥε਺C << σʔλ૯਺M M: ֶशσʔλͷ૯਺ C: ֶशσʔλͷΫϥε਺ B: 1epochͷόον਺ U: 1Ϋϥε͋ͨΓͷproxy਺ ϙΠϯτ

Slide 87

Slide 87 text

!87 ࣮ݧͨ͠ΒSOTAͳΜͰ͠ΐ…ʁ

Slide 88

Slide 88 text

࣮ݧ݁Ռ: SOTAʂ !88

Slide 89

Slide 89 text

࣮ݧ݁Ռ: SOTAʂ !89 ௗ ं

Slide 90

Slide 90 text

࣮ݧ݁Ռ: SOTAʂ !90 ಛ௃ྔͷ ࣍ݩ਺Λ ͪΌΜͱ ߹Θͤͯ ൺֱʂ (݁ߏޮ͍ͯ͘Δ)

Slide 91

Slide 91 text

࣮ݧ݁Ռ: SOTAʂ !91 σΧ͍ը૾࢖ͬͯΔͷʹ΋߹ΘͤͯΔʂ(݁ߏޮ͍ͯ͘Δ)

Slide 92

Slide 92 text

࣮ݧ݁Ռ: SOTAʂ !92 backbone networkهࡌͯ͠Δ͠߹ΘͤͯΔʂ(݁ߏޮ͍ͯ͘Δ)

Slide 93

Slide 93 text

࣮ݧ݁Ռ: SOTAʂ !93 ஸೡʂ

Slide 94

Slide 94 text

࣮ݧ݁Ռ: SOTAʂ !94 Stanford Online Products (ΊͪΌͪ͘Ό͍ΖΜͳ෺) In-shop(ϑΝογϣϯΞΠςϜ)

Slide 95

Slide 95 text

࣮ݧ݁Ռ: SOTAʂ !95 ஸೡʂ ஸೡʂ

Slide 96

Slide 96 text

!96 ͚ͩͰ͸ͳ͘…ʂ

Slide 97

Slide 97 text

ΫΤϦը૾ͷྨࣅ౓ݕࡧ݁ՌTop4 !97 CUB-200 SOP In-shop Cars-196 ΋͸΍ ؒҧͬͯΔϠπ ਓ͕΍ͬͯ΋ Ϝζ͍͡ΌΜ…

Slide 98

Slide 98 text

ϋΠύϥߟ࡯ͦͷ1: όοναΠζ !98 ʮ1όον಺ͷσʔλ਺͕ଟ͍ํ͕  σʔλશମͷߏ଄ΛΑΓଊ͑ΒΕΔʯ ҙ֎ͱ ͔ࠩͭͳ͍ͳ…

Slide 99

Slide 99 text

ϋΠύϥߟ࡯ͦͷ2: ಛ௃ྔۭؒͷ࣍ݩ਺ !99 ʮେ͖͍ํ͕΋ͪΖΜੑೳྑ͍ ͕ ݕࡧ଎౓ͱ͔͸τϨʔυΦϑʯ σΧ͗͢Δͱ ੑೳԼ͕ΔΜͩ…

Slide 100

Slide 100 text

ϋΠύϥߟ࡯ͦͷ3: εέʔϦϯάύϥϝʔλαͱϚʔδϯδ !100 ʮεέʔϦϯάύϥϝʔλ͸ 16Ҏ্Ͱ҆ఆͯ͠Δɺ ͕ͦͬͪे෼ͳେ͖ͩ͞ͱ ϚʔδϯͰ͸͋Μ·Γ͕ࠩग़ͳ͍ʯ εέʔϦϯά ύϥϝʔλ ࢮ׆໰୊͡ΌΜ…

Slide 101

Slide 101 text

ϋΠύϥߟ࡯ͦͷ3: εέʔϦϯάύϥϝʔλαͱϚʔδϯδ !101 ʮεέʔϦϯάύϥϝʔλ͸ 16Ҏ্Ͱ҆ఆͯ͠Δɺ ͕ͦͬͪे෼ͳେ͖ͩ͞ͱ ϚʔδϯͰ͸͋Μ·Γ͕ࠩग़ͳ͍ʯ εέʔϦϯά ύϥϝʔλ ࢮ׆໰୊͡ΌΜ… • ෳ਺ͷೖྗಉ࢜ͷʮࠩΛڧௐ͢Δ౓߹͍ʯ • cosྨࣅ౓Λ࢖͏Metric LearningͰΑ͘ಋೖ͞ΕΔ  (ͦ΋ͦ΋ͷ஋Ҭ͕-1 ≦ cosx ≦ 1ͳͷͰ͕͖ࠩͭʹ͍ͨ͘Ί) • DistillationΛఏҊͨ͠࿦จͰొ৔ͨ͠softmaxԹ౓ύϥϝʔλΛٯ਺ʹͨ͠΋ͷ ※͍͔ͭ͘ͷ࿦จΛݟ͖ͯͨײ͡20~60͙Β͍͕ଥ౰ͳΠϝʔδͩͬͨͷͰೲಘ εέʔϦϯάύϥϝʔλ “Distilling the Knowledge in a Neural Network”(NIPS’14 Workshop)

Slide 102

Slide 102 text

͞ΒʹarXiv൛Ͱ͸… !102

Slide 103

Slide 103 text

Appendix1: backbone networkͱը૾αΠζ !103 ʮ࣮ݧ͸Inception-BNͰ΍͚ͬͨͲResNetͩͱ΋ͬͱੌ͍ɺ GPUϝϞϦ࣍ୈͰը૾͸σΧ͍ํ͕ྑ͍ʯ ͦΓΌͦ͏ͩ…

Slide 104

Slide 104 text

Appendix2: ΫΤϦը૾ͷྨࣅ౓ݕࡧ݁Ռ(vs Proxy-NCA) !104 CUB-200 SOP In-shop Cars-196

Slide 105

Slide 105 text

Appendix3: ςετσʔλͷ෼෍ͲΜͳײ͡ʁ(t-SNEՄࢹԽ) !105 CUB-200 SOP In-shop Cars-196

Slide 106

Slide 106 text

͞Βʹ͞Βʹ… !106

Slide 107

Slide 107 text

͔ͳΓ៉ྷͳஶऀPyTorch࣮૷ʂ !107 https://github.com/tjddus9597/Proxy-Anchor-CVPR2020 • LossΫϥε͔Βtrain/testίʔυͷ࣮૷͸΋ͪΖΜ • σʔληοτ4छͷDatasetΫϥε࣮૷΋͋Γ • backbone networkͨͪͷ࣮૷΋͋Γ • طଘݚڀͷLossͨͪ΋ࢼͤΔΑ͏ʹͳͬͯΔ  (Pytorch-Metric-Learningύοέʔδ͕ඞཁ) ※͜ͷൃද·Ͱʹಈ͔͓͖͔ͯͨͬͨ͠ͷͰ͕࣌ؒ͢଍Γͣʀʀ

Slide 108

Slide 108 text

౳ʑ Deep Metric Learning in CVͷݱঢ়͕ ͱͯ΋ྑ͍ײ͡ʹ·ͱ·͍ͬͯΔҹ৅Ͱ͢ ͜ͷࢿྉยखʹ͝ҰಡΛʂ !108

Slide 109

Slide 109 text

·ͱΊ !109 •Metric Learning(ಛʹCV෼໺)ͷ࿩ •ຊ೔ͷ࿦จ: Proxy Anchor Loss ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ

Slide 110

Slide 110 text

!110