Unsupervised Domain Adaptation by Backpropagation גࣜձࣾαΠόʔΤʔδΣϯτ ΞυςΫຊ෦ɹAI Lab Kazuki Taniguchi 

Paper Detail ICML2015 http://proceedings.mlr.press/v37/ganin15.pdf 

Motivation -BCFMFE
EBUB (े෼ͳσʔλ͕͋Ε͹…) Deep Learning࠷ߴ!! Situation: A -BSHF
EBUB 

Synthetic Training Data : ֶशͷͨΊʹ߹੒ը૾ͱͯ͠ੜ੒͞Εͨ Situation: B : ࣮ࡍʹ༧ଌ͍ͨ͠ը૾ -BCFMFE
EBUB -BCFMFE
EBUB
4ZOUIFUJD
EBUB -BSHF
EBUB 4NBMM
EBUB : 8 : 6 : 4 : 1 

Domain Problem Կ͔͕ҧ͏?? ࣮ࡍʹ༧ଌ͍ͨ͠ը૾ ≠ ֶशʹ༻͍Δը૾ 

Domain Adaptation ֶशͱςετͷσʔλͷ෼෍ͷҧ͍(shift)͕ଘࡏ͢ΔϞσϧΛֶश͢Δ Domain Adaptation shift Target Domain Source Domain Source Domain͔ΒTarget DomainΛ༧ଌ͍ͨ͠ 

Unsupervised Domain Adaptation 6OMBCFMFE
EBUB -BCFMFE
EBUB Unlabeled dataΛ༧ଌ͍ͨ͠!! Situation: C : 8 : 6 : ? : ? 

Related works 

Related works • Subspace alignment • source subspace͔Βtarget subspaceͷม׵MΛֶश͢Δ Fernando, Basura, Habrard, Amaury, Sebban, Marc, and Tuytelaars, Tinne. Unsupervised visual domain adaptation using subspace alignment. In ICCV, 2013. Xs , Xt : eigenvectors • simple to setup • for experiments 

Related works • Generative adversarial nets (GAN) Goodfellow, Ian, Pouget-Abadie, Jean, Mirza, Mehdi, Xu, Bing, Warde-Farley, David, Ozair, Sherjil, Courville, Aaron, and Bengio, Yoshua. Generative adversarial nets. In NIPS, 2014. D(x) DiscriminatorΛὃͤΔը૾Λ࡞Δ!! ຊ෺ِ͔෺ΛݟۃΊΔͧ!! G(z) ʮຊ෺ͷը૾ʯΛ”ຊ෺”ͱࣝผ ʮِ෺ͷը૾ʯΛ”ِ෺”ͱࣝผ → G͸Dʹؒҧ͑ͯ΄͍͠ → D͸ਖ਼ࣝ͘͠ผͰ͖Ε͹ྑ͍ 

Related works • Deep Adaptation Network Long, Mingsheng and Wang, Jianmin. Learning transferable features with deep adaptation networks. CoRR, abs/1502.02791, 2015. • shallow • optimized by SGD but complex domainͷࣝผΛؒҧ͑ΔΑ͏ʹ͢Δ 

Deep Domain Adaptation 

Proposed Architecture 

Notation yi ∈ Y (Y = {1,2,...,L}) xi ∈ X di ∈ {0,1} xi ∼ S(x, y) if di = 0 xi ∼ T(x, y) if di = 1 : Input Data : Domain Label : Source Domain͔Βαϯϓϧ͞Εͨσʔλ : Target Domain͔Βαϯϓϧ͞Εͨσʔλ : Label (yi is known if di = 0 else unknown) 

Feature Representation • ϥϕϧ༧ଌޡࠩΛ࠷খʹ͢Δ • Source DomainͱTarget Domainʹରͯ͠ෆมʹͳΔ f = Gf (x; θf ) (f ∈ ℝD) 

Label prediction • Source DomainͷσʔλʹͷΈൃੜ͢Δ • ࣮ࡍͷ༧ଌΛ୲౰͢Δ y = Gy (f; θy ) 

Domain Invariant • શͯͷσʔλʹ͍ͭͯυϝΠϯ෼ྨΛߦ͏ • υϝΠϯΛʮΑΓؒҧ͑ΔΑ͏ʹʯֶश͢Δ d = Gd (f; θd ) = Adversarial 

Loss function E(θf , θy , θd ) = ∑ i=1,..,N Ly (Gy (Gf (xi ; θf ); θy ), yi ) − λ ∑ i=1,..,N Ld (Gd (Gf (xi ; θf ); θd ), di ) Label prediction Domain Invariant Ly : label prediction loss(e . g . multinomial) Ld : domain classification loss(e . g . logistic) 

Optimization ( ̂ θf , ̂ θy ) = argminθf ,θy E(θf , θy , ̂ θd ) ̂ θd = argmaxθd E( ̂ θf , ̂ θy , θd ) SGD θf ← θf − μ( δLi y δθf − λ δLi d δθf ) θy ← θy − μ( δLi y δθy ) θd ← θd − μ( δLi d δθd ) 

Optimization θf ← θf − μ( δLi y δθf − λ δLi d δθf ) δLi y δθf − λ δLi d δθf Researcher !! 

Gradient reversal layer (GRL) Rλ (x) = x δRλ (x) δx = − λI ˜ E(θf , θy , θd ) = ∑ i=1,..,N Ly (Gy (Gf (xi ; θf ); θy ), yi ) − λ ∑ i=1,..,N Ld (Gd (Rλ (Gf (xi ; θf )); θd ), di ) 

Model Summary • Domain labelΛ࢖ͬͨadversarialͳlossΛ௥Ճ • Gradient reversal layer(GRL)Λ௥Ճ͢Δ͜ͱͰ࣮૷͕༰қʹ SGDͰֶश͢Δ͜ͱ͕Մೳ 

Experiments 

Image Datasets Source Domain Target Domain Training Test 

Office Datasets ←ͷΑ͏ͳ঎඼ը૾Λ ɹɾDSLR ɹɾamazon.com ɹɾweb camera ͰࡱӨͨ͠σʔληοτ (https://people.eecs.berkeley.edu/~jhoffman/domainadapt/) 2817 labeled images 31 categories Ұ൪σʔληοτͷେ͖͍υϝΠϯ 

Comparisons • Baseline • source domainͷσʔλͰֶश • Subspace Alignment (SA) • (Fernando et al., 2013) • Train-on-target • target domainͷσʔλͰֶश (upper bound) 

Results (1) Classification accuracies 

Results (2) Classification accuracies 

Results (3) Real: ࣮ࡍͷը૾430ຕ (labeled) Syn: ߹੒ը૾100,000ຕ (labeled) Adapted: target domainͷը૾31,000ຕ (unlabeled) 

Visualizations (t-SNE) 

Discussion 

Discussion • Unsupervised Domain AdaptationͰDeepͳಛ௃நग़Λ༻͍ ͯɺߴਫ਼౓ͳϞσϧΛֶश͢Δ͜ͱ͕Ͱ͖ͨ • GRLΛಋೖ͢Δ͜ͱͰɺجຊతͳDLϥΠϒϥϦͰ΋༰қʹ Scalableʹֶश͢Δ͜ͱ͕Ͱ͖Δ • Future worksͱͯ͠semi-supervisedͳઃఆ΍΋ͬͱେ͖ͳλ εΫͰධՁ͢Δ͜ͱΛڍ͍͛ͯΔ 

fin. 5IBOLT
UP
ʮ͍Β͢ͱ΍ʯ