Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Unsupervised Domain Adaptation by Backpropagation

Unsupervised Domain Adaptation by Backpropagation

2018/11/22 PaperFriday @ CyberAgent, AI Lab

F2bd408c57c20505bd4202c9e28ecd59?s=128

Kazuki Taniguchi

November 22, 2018
Tweet

More Decks by Kazuki Taniguchi

Other Decks in Research

Transcript

  1. Unsupervised Domain Adaptation by Backpropagation גࣜձࣾαΠόʔΤʔδΣϯτ ΞυςΫຊ෦ɹAI Lab Kazuki Taniguchi

  2. Paper Detail ICML2015 http://proceedings.mlr.press/v37/ganin15.pdf

  3. Motivation -BCFMFEEBUB (े෼ͳσʔλ͕͋Ε͹…) Deep Learning࠷ߴ!! Situation: A -BSHFEBUB

  4. Synthetic Training Data : ֶशͷͨΊʹ߹੒ը૾ͱͯ͠ੜ੒͞Εͨ Situation: B : ࣮ࡍʹ༧ଌ͍ͨ͠ը૾ -BCFMFEEBUB

    -BCFMFEEBUB 4ZOUIFUJDEBUB -BSHFEBUB 4NBMMEBUB : 8 : 6 : 4 : 1
  5. Domain Problem Կ͔͕ҧ͏?? ࣮ࡍʹ༧ଌ͍ͨ͠ը૾ ≠ ֶशʹ༻͍Δը૾

  6. Domain Adaptation ֶशͱςετͷσʔλͷ෼෍ͷҧ͍(shift)͕ଘࡏ͢ΔϞσϧΛֶश͢Δ Domain Adaptation shift Target Domain Source Domain

    Source Domain͔ΒTarget DomainΛ༧ଌ͍ͨ͠
  7. Unsupervised Domain Adaptation 6OMBCFMFEEBUB -BCFMFEEBUB Unlabeled dataΛ༧ଌ͍ͨ͠!! Situation: C :

    8 : 6 : ? : ?
  8. Related works

  9. Related works • Subspace alignment • source subspace͔Βtarget subspaceͷม׵MΛֶश͢Δ Fernando,

    Basura, Habrard, Amaury, Sebban, Marc, and Tuytelaars, Tinne. Unsupervised visual domain adaptation using subspace alignment. In ICCV, 2013. Xs , Xt : eigenvectors • simple to setup • for experiments
  10. Related works • Generative adversarial nets (GAN) Goodfellow, Ian, Pouget-Abadie,

    Jean, Mirza, Mehdi, Xu, Bing, Warde-Farley, David, Ozair, Sherjil, Courville, Aaron, and Bengio, Yoshua. Generative adversarial nets. In NIPS, 2014. D(x) DiscriminatorΛὃͤΔը૾Λ࡞Δ!! ຊ෺ِ͔෺ΛݟۃΊΔͧ!! G(z) ʮຊ෺ͷը૾ʯΛ”ຊ෺”ͱࣝผ ʮِ෺ͷը૾ʯΛ”ِ෺”ͱࣝผ → G͸Dʹؒҧ͑ͯ΄͍͠ → D͸ਖ਼ࣝ͘͠ผͰ͖Ε͹ྑ͍
  11. Related works • Deep Adaptation Network Long, Mingsheng and Wang,

    Jianmin. Learning transferable features with deep adaptation networks. CoRR, abs/1502.02791, 2015. • shallow • optimized by SGD but complex domainͷࣝผΛؒҧ͑ΔΑ͏ʹ͢Δ
  12. Deep Domain Adaptation

  13. Proposed Architecture

  14. Notation yi ∈ Y (Y = {1,2,...,L}) xi ∈ X

    di ∈ {0,1} xi ∼ S(x, y) if di = 0 xi ∼ T(x, y) if di = 1 : Input Data : Domain Label : Source Domain͔Βαϯϓϧ͞Εͨσʔλ : Target Domain͔Βαϯϓϧ͞Εͨσʔλ : Label (yi is known if di = 0 else unknown)
  15. Feature Representation • ϥϕϧ༧ଌޡࠩΛ࠷খʹ͢Δ • Source DomainͱTarget Domainʹରͯ͠ෆมʹͳΔ f =

    Gf (x; θf ) (f ∈ ℝD)
  16. Label prediction • Source DomainͷσʔλʹͷΈൃੜ͢Δ • ࣮ࡍͷ༧ଌΛ୲౰͢Δ y = Gy

    (f; θy )
  17. Domain Invariant • શͯͷσʔλʹ͍ͭͯυϝΠϯ෼ྨΛߦ͏ • υϝΠϯΛʮΑΓؒҧ͑ΔΑ͏ʹʯֶश͢Δ d = Gd (f;

    θd ) = Adversarial
  18. Loss function E(θf , θy , θd ) = ∑

    i=1,..,N Ly (Gy (Gf (xi ; θf ); θy ), yi ) − λ ∑ i=1,..,N Ld (Gd (Gf (xi ; θf ); θd ), di ) Label prediction Domain Invariant Ly : label prediction loss(e . g . multinomial) Ld : domain classification loss(e . g . logistic)
  19. Optimization ( ̂ θf , ̂ θy ) = argminθf

    ,θy E(θf , θy , ̂ θd ) ̂ θd = argmaxθd E( ̂ θf , ̂ θy , θd ) SGD θf ← θf − μ( δLi y δθf − λ δLi d δθf ) θy ← θy − μ( δLi y δθy ) θd ← θd − μ( δLi d δθd )
  20. Optimization θf ← θf − μ( δLi y δθf −

    λ δLi d δθf ) δLi y δθf − λ δLi d δθf Researcher !!
  21. Gradient reversal layer (GRL) Rλ (x) = x δRλ (x)

    δx = − λI ˜ E(θf , θy , θd ) = ∑ i=1,..,N Ly (Gy (Gf (xi ; θf ); θy ), yi ) − λ ∑ i=1,..,N Ld (Gd (Rλ (Gf (xi ; θf )); θd ), di )
  22. Model Summary • Domain labelΛ࢖ͬͨadversarialͳlossΛ௥Ճ • Gradient reversal layer(GRL)Λ௥Ճ͢Δ͜ͱͰ࣮૷͕༰қʹ SGDͰֶश͢Δ͜ͱ͕Մೳ

  23. Experiments

  24. Image Datasets Source Domain Target Domain Training Test

  25. Office Datasets ←ͷΑ͏ͳ঎඼ը૾Λ ɹɾDSLR ɹɾamazon.com ɹɾweb camera ͰࡱӨͨ͠σʔληοτ (https://people.eecs.berkeley.edu/~jhoffman/domainadapt/) 2817

    labeled images 31 categories Ұ൪σʔληοτͷେ͖͍υϝΠϯ
  26. Comparisons • Baseline • source domainͷσʔλͰֶश • Subspace Alignment (SA)

    • (Fernando et al., 2013) • Train-on-target • target domainͷσʔλͰֶश (upper bound)
  27. Results (1) Classification accuracies

  28. Results (2) Classification accuracies

  29. Results (3) Real: ࣮ࡍͷը૾430ຕ (labeled) Syn: ߹੒ը૾100,000ຕ (labeled) Adapted: target

    domainͷը૾31,000ຕ (unlabeled)
  30. Visualizations (t-SNE)

  31. Discussion

  32. Discussion • Unsupervised Domain AdaptationͰDeepͳಛ௃நग़Λ༻͍ ͯɺߴਫ਼౓ͳϞσϧΛֶश͢Δ͜ͱ͕Ͱ͖ͨ • GRLΛಋೖ͢Δ͜ͱͰɺجຊతͳDLϥΠϒϥϦͰ΋༰қʹ Scalableʹֶश͢Δ͜ͱ͕Ͱ͖Δ •

    Future worksͱͯ͠semi-supervisedͳઃఆ΍΋ͬͱେ͖ͳλ εΫͰධՁ͢Δ͜ͱΛڍ͍͛ͯΔ
  33. fin. 5IBOLTUPʮ͍Β͢ͱ΍ʯ