Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Supervised PCAとその周辺

Supervised PCAとその周辺

Daisuke Yoneoka

November 14, 2023
Tweet

More Decks by Daisuke Yoneoka

Other Decks in Research

Transcript

  1. Supervised PCA ͱͦͷपล Daisuke Yoneoka February 22, 2015 Daisuke Yoneoka

    Supervised PCA ͱͦͷपล February 22, 2015 1 / 11
  2. Notations Latent: zi ∈ RL Observed: xi ∈ RD Outcome:

    yi ∈ RD PCA: ৴߸ղੳͱ͔ͩͱ Karhunen Loeve transform ͱ͔ͱ΋ݺ͹ΕΔΒ͍͠ Factor analysis: p(zi ) = N(zi |µ0 , Σ0 ), p(xi |zi , θ) = N(W zi + µ, Ψ) W ͸ D × L Ͱ factor loading matrix Ψ ͸ D × D Ͱର֯ߦྻ ͱ͘ʹ Ψ = σ2I ͷͱ͖ probabilistic PCA(PPCA) or sensible PCA (Roweis 1997) PCA ͷઆ໌͸লུ Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 2 / 11
  3. ಋೖ ϖΞʹͳͬͯΔσʔλͱ͔Λ௿࣍ݩʹຒΊࠐΈ͍ͨΈ͍ͨͳཁ๬ ͜ͷཁ๬Λ Virtanen 2010, ʹԊͬͯ Latent Gaussian model Ͱղܾ͍ͨ͠

    ҎԼͷख๏Λ Murphy ʹԊͬͯ֓આ supervised PCA discriminative supervised PCA partial least square canonical correlation analysis Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 3 / 11
  4. Supervised PCA (SPCA) ࣜ͸ҎԼͷ༷ʹͳΔ. Yu et al. (2006) Ͱ͸ Supervised

    PCA Ͱ, West (2003) Ͱ͸ Bayesian factor regression ͱݺ͹Ε͍ͯΔ. p(zi) = N(0, IL) p(yi |zi) = N(wT y zi + µy, σ2 y ) p(xi |zi ) = N(Wx zi + µx , σ2 x ID ) PCA ͱͷ૬ҧ఺: PCA ͸ x ʹ͔͠஫໨͍ͯ͠ͳ͍͕, SPCA ͸ y ΋ߟྀͯ͠ ͍Δ. Joint Gaussian ͳͷͰ, yi |xi ∼ N(xT i w, σy + wT y Cwy), where w = Ψ−1WxCwy , Ψ = σ2 x ID and C−1 = I + W T x Ψ−1Wx ͕ܭࢉՄೳ. z Ͱ৚͚݅ͭΔͱ x ͱ y ͸ಠཱ: p(y, x|z) = p(y|z)p(x|z) ΛܭࢉͰ͸࢖͏ Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 5 / 11
  5. Supervised PCA ͱ Zeller ͷ g-prior West (2003) ͸, SPCA

    ͱ Zeller ͷ g-prior ͷؔ܎Λ໌Β͔ʹͨ͠. p(wy ) = N(0, (1/gΣ2)−1) SVD of X ͱͯ͠ X = RV T , V T V = I ͱ͠, RT R = Σ2 = diag(σ2 j ) ͸ಛ ҟ஋Λର֯ʹฒ΂ͨ΋ͷͱ͢Δͱ σ2 x → 0 ͷۃݶͰ p(w) = N(0, gV −T Σ−2V −1) = N(0, g(XT X)−1) Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 6 / 11
  6. Information bottleneck x Λ࣍ݩॖ໿ͯ͠ y Λ༧ଌ͠Α͏ͱ͢ΔΞΠσΟΞ͸৘ใཧ࿦Ͱ͸ҎԼͷΑ͏ʹ දݱՄೳ. ҎԼΛ࠷খԽ͢ΔΑ͏ͳ p(z|z) Λൃݟ͍ͨ͠

    I(X; Z) − βI(X; Y ), ͨͩ͠, I(X; Y ) ͸૬ޓ৘ใྔ I(X : Y ) = H(X) − H(X|Y ) = H(X) + H(Y ) − H(X, Y ), ͨͩ͠ H(X) = Ep log 1 p(X) β ≥ 0 ͸ information bottleneck Ͱॖ໿౓߹͍ͱ༧ଌੑೳͷ tradeoff Λௐ੔ Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 7 / 11
  7. Discriminant supervised PCA SPCA ͷ໰୊͸ p(x|z) ͱ p(y|z) ʹಉ͡ॏΈΛ͔͚͍ͯΔ఺. Rish

    et al. (2008) ͸͜ΕΛղܾ. ॏΈ αx ͱ αy ΛҎԼͷΑ͏ʹಋೖ. l(θ) = Πip(yi |ηiy)αy p(xi |ηix)αx , ͨͩ͠, ηim = Wmzm α ͸ exponential family ͳΒ͹ noise variance ͱͯ͠ղऍՄೳ. ྫ͑͹σʔλ͕ Gaussian: l(θ) ∝ Πi exp(− 1 2 αx ∥xT i − ηix ∥2) Note: α ͷਪఆ͸໬౓ͷ normalizing constant ͕ͦͷ౎౓มԽ͢ΔͷͰɺࠔ ೉Ͱ͋Δ. Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 8 / 11
  8. Partial least square (PLS) ܭྔܦࡁͷํͰ͸༗໊. Ϟσϧ͸ҎԼ. p(zi) = N(zs i

    |0, ILs )N(zx i |0, ILx ) p(yi |zi ) = N(W T y zs i + µy , σ2IDy ) p(xi |zi ) = N(Wx zs i + Bx zx i + µx , σ2IDx ) ΞΠσΟΞ͸, zi Λڞ௨ͷ zs i ͱ zx i ʹ෼ղ͢Δ͜ͱ. vi = (yi , xi ) ͷ৚݅෇͖෼෍: p(vi |θ) = N(vi |W zi + µ, σI)N(zi |0, I)dzi = N(vi |µ, W W T + σI) where W = Wy 0 Wx By and W W T = Wy W T y Wx W T x Wx W T x Wx W T x + Bx BT x Note: Latent ͳΫϥεͷ࣍ݩ͸, zs i ͕ڞมྔʹಛ༗ͷ෼ࢄΛଊ͑ͯ͠·Θͳ ͍Α͏ʹे෼େ͖ΊʹऔΔඞཁ͕͋Δ. Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 9 / 11
  9. PLS ͷΞϧΰϦζϜ PCA ͷ݁ՌΛ regression ʹೖΕΔํ๏ (Primary component regression) ͱؔ܎͠

    ͍ͯΔ͕, ͪΐ ͬͱҧ͏. Solution path ͸෼ࢄ͕େ͖͘, ͔ͭ y ͱ૬ؔͷߴ͍ํ޲Λ୳ࡧ͍ͯ͠Δ cf. PCR ͸ಛ௃ྔͷ෼ࢄΛେ͖͘͢Δ͜ͱ͚ͩʹ஫໨͍ͯ͠Δ. Frank et al.(1993) ʹΑΔͱ Ridge ճؼΑΓ༧ଌੑೳͰ͸ྼΔ͕௿࣍ݩ΁ͷ ॖ໿͕Մೳ. ਪఆʹ͸ Wold (1975) ͷ NIPALS ΞϧΰϦζϜ͕༗໊ (Hastie et al. (2001)). Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 10 / 11
  10. Canonical correlation analysis (CCA) ਖ਼४૬ؔ෼ੳͱ͍͏໊લͰֶ෦ͷ࣌ͱ͔ʹशͬͨΑ͏ͳؾ͕͢Δ. ࣜ͸ҎԼ (Bach and Jordan, (2005)).

    p(zi ) = N(zs i |0, ILs )N(zx i |0, ILx )N(zy i |0, ILy p(yi |zi ) = N(yi |By zy i + Wy zs i + µy , σ2IDy ) p(xi |zi ) = N(xi |Bx zx i + Wx zs i + µx , σ2IDx ) PLS Λ synmetric ʹͨ͠΋ͷ. ͭ·Γ, zi Λڞ௨ͷ zs i ͱ zx i ͱ zy i ʹ෼ղ͢Δ ͜ͱ. vi ͷ৚݅෇͖෼෍: p(vi |θ) = N(vi |W zi + µ, σI)N(zi |0, I)dzi = N(vi |µ, W W T + σID) where W = Wx Bx 0 Wy 0 By and W W T = Wx W T x + Bx BT x Wx W T y Wy W T y Wy W T y + By BT y MLE Λ EM Ͱղ͘ classic ͳ non-probabilistic ͳ݁ՌͱҰக͢Δ (Bach and Jordan, (2005)) Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 11 / 11