Unsupervised Domain Adaptation by Backpropagation

2018/11/22 PaperFriday @ CyberAgent, AI Lab

Transcript

1. Unsupervised Domain Adaptation by Backpropagation גࣜձࣾαΠόʔΤʔδΣϯτ ΞυςΫຊ෦ɹAI Lab Kazuki Taniguchi
2. Paper Detail ICML2015 http://proceedings.mlr.press/v37/ganin15.pdf
3. Motivation -BCFMFE
   EBUB (े෼ͳσʔλ͕͋Ε͹…) Deep Learning࠷ߴ!! Situation: A -BSHF
   EBUB
4. Synthetic Training Data : ֶशͷͨΊʹ߹੒ը૾ͱͯ͠ੜ੒͞Εͨ Situation: B : ࣮ࡍʹ༧ଌ͍ͨ͠ը૾ -BCFMFE
   EBUB -BCFMFE
   EBUB
   4ZOUIFUJD
   EBUB -BSHF
   EBUB 4NBMM
   EBUB : 8 : 6 : 4 : 1
5. Domain Problem Կ͔͕ҧ͏?? ࣮ࡍʹ༧ଌ͍ͨ͠ը૾ ≠ ֶशʹ༻͍Δը૾
6. Domain Adaptation ֶशͱςετͷσʔλͷ෼෍ͷҧ͍(shift)͕ଘࡏ͢ΔϞσϧΛֶश͢Δ Domain Adaptation shift Target Domain Source Domain Source Domain͔ΒTarget DomainΛ༧ଌ͍ͨ͠
7. Unsupervised Domain Adaptation 6OMBCFMFE
   EBUB -BCFMFE
   EBUB Unlabeled dataΛ༧ଌ͍ͨ͠!! Situation: C : 8 : 6 : ? : ?
8. Related works
9. Related works • Subspace alignment • source subspace͔Βtarget subspaceͷม׵MΛֶश͢Δ Fernando, Basura, Habrard, Amaury, Sebban, Marc, and Tuytelaars, Tinne. Unsupervised visual domain adaptation using subspace alignment. In ICCV, 2013. Xs , Xt : eigenvectors • simple to setup • for experiments
10. Related works • Generative adversarial nets (GAN) Goodfellow, Ian, Pouget-Abadie, Jean, Mirza, Mehdi, Xu, Bing, Warde-Farley, David, Ozair, Sherjil, Courville, Aaron, and Bengio, Yoshua. Generative adversarial nets. In NIPS, 2014. D(x) DiscriminatorΛὃͤΔը૾Λ࡞Δ!! ຊ෺ِ͔෺ΛݟۃΊΔͧ!! G(z) ʮຊ෺ͷը૾ʯΛ”ຊ෺”ͱࣝผ ʮِ෺ͷը૾ʯΛ”ِ෺”ͱࣝผ → G͸Dʹؒҧ͑ͯ΄͍͠ → D͸ਖ਼ࣝ͘͠ผͰ͖Ε͹ྑ͍
11. Related works • Deep Adaptation Network Long, Mingsheng and Wang, Jianmin. Learning transferable features with deep adaptation networks. CoRR, abs/1502.02791, 2015. • shallow • optimized by SGD but complex domainͷࣝผΛؒҧ͑ΔΑ͏ʹ͢Δ
12. Deep Domain Adaptation
13. Proposed Architecture
14. Notation yi ∈ Y (Y = {1,2,...,L}) xi ∈ X di ∈ {0,1} xi ∼ S(x, y) if di = 0 xi ∼ T(x, y) if di = 1 : Input Data : Domain Label : Source Domain͔Βαϯϓϧ͞Εͨσʔλ : Target Domain͔Βαϯϓϧ͞Εͨσʔλ : Label (yi is known if di = 0 else unknown)
15. Feature Representation • ϥϕϧ༧ଌޡࠩΛ࠷খʹ͢Δ • Source DomainͱTarget Domainʹରͯ͠ෆมʹͳΔ f = Gf (x; θf ) (f ∈ ℝD)
16. Label prediction • Source DomainͷσʔλʹͷΈൃੜ͢Δ • ࣮ࡍͷ༧ଌΛ୲౰͢Δ y = Gy (f; θy )
17. Domain Invariant • શͯͷσʔλʹ͍ͭͯυϝΠϯ෼ྨΛߦ͏ • υϝΠϯΛʮΑΓؒҧ͑ΔΑ͏ʹʯֶश͢Δ d = Gd (f; θd ) = Adversarial
18. Loss function E(θf , θy , θd ) = ∑ i=1,..,N Ly (Gy (Gf (xi ; θf ); θy ), yi ) − λ ∑ i=1,..,N Ld (Gd (Gf (xi ; θf ); θd ), di ) Label prediction Domain Invariant Ly : label prediction loss(e . g . multinomial) Ld : domain classification loss(e . g . logistic)
19. Optimization ( ̂ θf , ̂ θy ) = argminθf ,θy E(θf , θy , ̂ θd ) ̂ θd = argmaxθd E( ̂ θf , ̂ θy , θd ) SGD θf ← θf − μ( δLi y δθf − λ δLi d δθf ) θy ← θy − μ( δLi y δθy ) θd ← θd − μ( δLi d δθd )
20. Optimization θf ← θf − μ( δLi y δθf − λ δLi d δθf ) δLi y δθf − λ δLi d δθf Researcher !!
21. Gradient reversal layer (GRL) Rλ (x) = x δRλ (x) δx = − λI ˜ E(θf , θy , θd ) = ∑ i=1,..,N Ly (Gy (Gf (xi ; θf ); θy ), yi ) − λ ∑ i=1,..,N Ld (Gd (Rλ (Gf (xi ; θf )); θd ), di )
22. Model Summary • Domain labelΛ࢖ͬͨadversarialͳlossΛ௥Ճ • Gradient reversal layer(GRL)Λ௥Ճ͢Δ͜ͱͰ࣮૷͕༰қʹ SGDͰֶश͢Δ͜ͱ͕Մೳ
23. Experiments
24. Image Datasets Source Domain Target Domain Training Test
25. Office Datasets ←ͷΑ͏ͳ঎඼ը૾Λ ɹɾDSLR ɹɾamazon.com ɹɾweb camera ͰࡱӨͨ͠σʔληοτ (https://people.eecs.berkeley.edu/~jhoffman/domainadapt/) 2817 labeled images 31 categories Ұ൪σʔληοτͷେ͖͍υϝΠϯ
26. Comparisons • Baseline • source domainͷσʔλͰֶश • Subspace Alignment (SA) • (Fernando et al., 2013) • Train-on-target • target domainͷσʔλͰֶश (upper bound)
27. Results (1) Classification accuracies
28. Results (2) Classification accuracies
29. Results (3) Real: ࣮ࡍͷը૾430ຕ (labeled) Syn: ߹੒ը૾100,000ຕ (labeled) Adapted: target domainͷը૾31,000ຕ (unlabeled)
30. Visualizations (t-SNE)
31. Discussion
32. Discussion • Unsupervised Domain AdaptationͰDeepͳಛ௃நग़Λ༻͍ ͯɺߴਫ਼౓ͳϞσϧΛֶश͢Δ͜ͱ͕Ͱ͖ͨ • GRLΛಋೖ͢Δ͜ͱͰɺجຊతͳDLϥΠϒϥϦͰ΋༰қʹ Scalableʹֶश͢Δ͜ͱ͕Ͱ͖Δ • Future worksͱͯ͠semi-supervisedͳઃఆ΍΋ͬͱେ͖ͳλ εΫͰධՁ͢Δ͜ͱΛڍ͍͛ͯΔ
33. fin. 5IBOLT
    UP
    ʮ͍Β͢ͱ΍ʯ