Unsupervised Domain Adaptation by Backpropagation

Unsupervised Domain Adaptation by Backpropagation

2018/11/22 PaperFriday @ CyberAgent, AI Lab

F2bd408c57c20505bd4202c9e28ecd59?s=128

Kazuki Taniguchi

November 22, 2018
Tweet

Transcript

  1. Unsupervised Domain Adaptation by Backpropagation גࣜձࣾαΠόʔΤʔδΣϯτ ΞυςΫຊ෦ɹAI Lab Kazuki Taniguchi

  2. Paper Detail ICML2015 http://proceedings.mlr.press/v37/ganin15.pdf

  3. Motivation -BCFMFEEBUB (े෼ͳσʔλ͕͋Ε͹…) Deep Learning࠷ߴ!! Situation: A -BSHFEBUB

  4. Synthetic Training Data : ֶशͷͨΊʹ߹੒ը૾ͱͯ͠ੜ੒͞Εͨ Situation: B : ࣮ࡍʹ༧ଌ͍ͨ͠ը૾ -BCFMFEEBUB

    -BCFMFEEBUB 4ZOUIFUJDEBUB -BSHFEBUB 4NBMMEBUB : 8 : 6 : 4 : 1
  5. Domain Problem Կ͔͕ҧ͏?? ࣮ࡍʹ༧ଌ͍ͨ͠ը૾ ≠ ֶशʹ༻͍Δը૾

  6. Domain Adaptation ֶशͱςετͷσʔλͷ෼෍ͷҧ͍(shift)͕ଘࡏ͢ΔϞσϧΛֶश͢Δ Domain Adaptation shift Target Domain Source Domain

    Source Domain͔ΒTarget DomainΛ༧ଌ͍ͨ͠
  7. Unsupervised Domain Adaptation 6OMBCFMFEEBUB -BCFMFEEBUB Unlabeled dataΛ༧ଌ͍ͨ͠!! Situation: C :

    8 : 6 : ? : ?
  8. Related works

  9. Related works • Subspace alignment • source subspace͔Βtarget subspaceͷม׵MΛֶश͢Δ Fernando,

    Basura, Habrard, Amaury, Sebban, Marc, and Tuytelaars, Tinne. Unsupervised visual domain adaptation using subspace alignment. In ICCV, 2013. Xs , Xt : eigenvectors • simple to setup • for experiments
  10. Related works • Generative adversarial nets (GAN) Goodfellow, Ian, Pouget-Abadie,

    Jean, Mirza, Mehdi, Xu, Bing, Warde-Farley, David, Ozair, Sherjil, Courville, Aaron, and Bengio, Yoshua. Generative adversarial nets. In NIPS, 2014. D(x) DiscriminatorΛὃͤΔը૾Λ࡞Δ!! ຊ෺ِ͔෺ΛݟۃΊΔͧ!! G(z) ʮຊ෺ͷը૾ʯΛ”ຊ෺”ͱࣝผ ʮِ෺ͷը૾ʯΛ”ِ෺”ͱࣝผ → G͸Dʹؒҧ͑ͯ΄͍͠ → D͸ਖ਼ࣝ͘͠ผͰ͖Ε͹ྑ͍
  11. Related works • Deep Adaptation Network Long, Mingsheng and Wang,

    Jianmin. Learning transferable features with deep adaptation networks. CoRR, abs/1502.02791, 2015. • shallow • optimized by SGD but complex domainͷࣝผΛؒҧ͑ΔΑ͏ʹ͢Δ
  12. Deep Domain Adaptation

  13. Proposed Architecture

  14. Notation yi ∈ Y (Y = {1,2,...,L}) xi ∈ X

    di ∈ {0,1} xi ∼ S(x, y) if di = 0 xi ∼ T(x, y) if di = 1 : Input Data : Domain Label : Source Domain͔Βαϯϓϧ͞Εͨσʔλ : Target Domain͔Βαϯϓϧ͞Εͨσʔλ : Label (yi is known if di = 0 else unknown)
  15. Feature Representation • ϥϕϧ༧ଌޡࠩΛ࠷খʹ͢Δ • Source DomainͱTarget Domainʹରͯ͠ෆมʹͳΔ f =

    Gf (x; θf ) (f ∈ ℝD)
  16. Label prediction • Source DomainͷσʔλʹͷΈൃੜ͢Δ • ࣮ࡍͷ༧ଌΛ୲౰͢Δ y = Gy

    (f; θy )
  17. Domain Invariant • શͯͷσʔλʹ͍ͭͯυϝΠϯ෼ྨΛߦ͏ • υϝΠϯΛʮΑΓؒҧ͑ΔΑ͏ʹʯֶश͢Δ d = Gd (f;

    θd ) = Adversarial
  18. Loss function E(θf , θy , θd ) = ∑

    i=1,..,N Ly (Gy (Gf (xi ; θf ); θy ), yi ) − λ ∑ i=1,..,N Ld (Gd (Gf (xi ; θf ); θd ), di ) Label prediction Domain Invariant Ly : label prediction loss(e . g . multinomial) Ld : domain classification loss(e . g . logistic)
  19. Optimization ( ̂ θf , ̂ θy ) = argminθf

    ,θy E(θf , θy , ̂ θd ) ̂ θd = argmaxθd E( ̂ θf , ̂ θy , θd ) SGD θf ← θf − μ( δLi y δθf − λ δLi d δθf ) θy ← θy − μ( δLi y δθy ) θd ← θd − μ( δLi d δθd )
  20. Optimization θf ← θf − μ( δLi y δθf −

    λ δLi d δθf ) δLi y δθf − λ δLi d δθf Researcher !!
  21. Gradient reversal layer (GRL) Rλ (x) = x δRλ (x)

    δx = − λI ˜ E(θf , θy , θd ) = ∑ i=1,..,N Ly (Gy (Gf (xi ; θf ); θy ), yi ) − λ ∑ i=1,..,N Ld (Gd (Rλ (Gf (xi ; θf )); θd ), di )
  22. Model Summary • Domain labelΛ࢖ͬͨadversarialͳlossΛ௥Ճ • Gradient reversal layer(GRL)Λ௥Ճ͢Δ͜ͱͰ࣮૷͕༰қʹ SGDͰֶश͢Δ͜ͱ͕Մೳ

  23. Experiments

  24. Image Datasets Source Domain Target Domain Training Test

  25. Office Datasets ←ͷΑ͏ͳ঎඼ը૾Λ ɹɾDSLR ɹɾamazon.com ɹɾweb camera ͰࡱӨͨ͠σʔληοτ (https://people.eecs.berkeley.edu/~jhoffman/domainadapt/) 2817

    labeled images 31 categories Ұ൪σʔληοτͷେ͖͍υϝΠϯ
  26. Comparisons • Baseline • source domainͷσʔλͰֶश • Subspace Alignment (SA)

    • (Fernando et al., 2013) • Train-on-target • target domainͷσʔλͰֶश (upper bound)
  27. Results (1) Classification accuracies

  28. Results (2) Classification accuracies

  29. Results (3) Real: ࣮ࡍͷը૾430ຕ (labeled) Syn: ߹੒ը૾100,000ຕ (labeled) Adapted: target

    domainͷը૾31,000ຕ (unlabeled)
  30. Visualizations (t-SNE)

  31. Discussion

  32. Discussion • Unsupervised Domain AdaptationͰDeepͳಛ௃நग़Λ༻͍ ͯɺߴਫ਼౓ͳϞσϧΛֶश͢Δ͜ͱ͕Ͱ͖ͨ • GRLΛಋೖ͢Δ͜ͱͰɺجຊతͳDLϥΠϒϥϦͰ΋༰қʹ Scalableʹֶश͢Δ͜ͱ͕Ͱ͖Δ •

    Future worksͱͯ͠semi-supervisedͳઃఆ΍΋ͬͱେ͖ͳλ εΫͰධՁ͢Δ͜ͱΛڍ͍͛ͯΔ
  33. fin. 5IBOLTUPʮ͍Β͢ͱ΍ʯ