Share
Embed
Transcript
11 slides
-
Supervised PCA ͱͦͷपล Daisuke Yoneoka February 22, 2015 Daisuke Yoneoka
Supervised PCA ͱͦͷपล February 22, 2015 1 / 11 -
Notations Latent: zi ∈ RL Observed: xi ∈ RD Outcome:
yi ∈ RD PCA: ৴߸ղੳͱ͔ͩͱ Karhunen Loeve transform ͱ͔ͱݺΕΔΒ͍͠ Factor analysis: p(zi ) = N(zi |µ0 , Σ0 ), p(xi |zi , θ) = N(W zi + µ, Ψ) W D × L Ͱ factor loading matrix Ψ D × D Ͱର֯ߦྻ ͱ͘ʹ Ψ = σ2I ͷͱ͖ probabilistic PCA(PPCA) or sensible PCA (Roweis 1997) PCA ͷઆ໌লུ Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 2 / 11 -
ಋೖ ϖΞʹͳͬͯΔσʔλͱ͔Λ࣍ݩʹຒΊࠐΈ͍ͨΈ͍ͨͳཁ ͜ͷཁΛ Virtanen 2010, ʹԊͬͯ Latent Gaussian model Ͱղܾ͍ͨ͠
ҎԼͷख๏Λ Murphy ʹԊͬͯ֓આ supervised PCA discriminative supervised PCA partial least square canonical correlation analysis Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 3 / 11 -
Supervised PCA (SPCA) ࣜҎԼͷ༷ʹͳΔ. Yu et al. (2006) Ͱ Supervised
PCA Ͱ, West (2003) Ͱ Bayesian factor regression ͱݺΕ͍ͯΔ. p(zi) = N(0, IL) p(yi |zi) = N(wT y zi + µy, σ2 y ) p(xi |zi ) = N(Wx zi + µx , σ2 x ID ) PCA ͱͷ૬ҧ: PCA x ʹ͔͍ͯ͠͠ͳ͍͕, SPCA y ߟྀͯ͠ ͍Δ. Joint Gaussian ͳͷͰ, yi |xi ∼ N(xT i w, σy + wT y Cwy), where w = Ψ−1WxCwy , Ψ = σ2 x ID and C−1 = I + W T x Ψ−1Wx ͕ܭࢉՄೳ. z Ͱ͚݅ͭΔͱ x ͱ y ಠཱ: p(y, x|z) = p(y|z)p(x|z) ΛܭࢉͰ͏ Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 5 / 11 -
Supervised PCA ͱ Zeller ͷ g-prior West (2003) , SPCA
ͱ Zeller ͷ g-prior ͷؔΛ໌Β͔ʹͨ͠. p(wy ) = N(0, (1/gΣ2)−1) SVD of X ͱͯ͠ X = RV T , V T V = I ͱ͠, RT R = Σ2 = diag(σ2 j ) ಛ ҟΛର֯ʹฒͨͷͱ͢Δͱ σ2 x → 0 ͷۃݶͰ p(w) = N(0, gV −T Σ−2V −1) = N(0, g(XT X)−1) Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 6 / 11 -
Information bottleneck x Λ࣍ݩॖͯ͠ y Λ༧ଌ͠Α͏ͱ͢ΔΞΠσΟΞใཧͰҎԼͷΑ͏ʹ දݱՄೳ. ҎԼΛ࠷খԽ͢ΔΑ͏ͳ p(z|z) Λൃݟ͍ͨ͠
I(X; Z) − βI(X; Y ), ͨͩ͠, I(X; Y ) ૬ޓใྔ I(X : Y ) = H(X) − H(X|Y ) = H(X) + H(Y ) − H(X, Y ), ͨͩ͠ H(X) = Ep log 1 p(X) β ≥ 0 information bottleneck Ͱॖ߹͍ͱ༧ଌੑೳͷ tradeoff Λௐ Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 7 / 11 -
Discriminant supervised PCA SPCA ͷ p(x|z) ͱ p(y|z) ʹಉ͡ॏΈΛ͔͚͍ͯΔ. Rish
et al. (2008) ͜ΕΛղܾ. ॏΈ αx ͱ αy ΛҎԼͷΑ͏ʹಋೖ. l(θ) = Πip(yi |ηiy)αy p(xi |ηix)αx , ͨͩ͠, ηim = Wmzm α exponential family ͳΒ noise variance ͱͯ͠ղऍՄೳ. ྫ͑σʔλ͕ Gaussian: l(θ) ∝ Πi exp(− 1 2 αx ∥xT i − ηix ∥2) Note: α ͷਪఆͷ normalizing constant ͕ͦͷมԽ͢ΔͷͰɺࠔ Ͱ͋Δ. Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 8 / 11 -
Partial least square (PLS) ܭྔܦࡁͷํͰ༗໊. ϞσϧҎԼ. p(zi) = N(zs i
|0, ILs )N(zx i |0, ILx ) p(yi |zi ) = N(W T y zs i + µy , σ2IDy ) p(xi |zi ) = N(Wx zs i + Bx zx i + µx , σ2IDx ) ΞΠσΟΞ, zi Λڞ௨ͷ zs i ͱ zx i ʹղ͢Δ͜ͱ. vi = (yi , xi ) ͷ͖݅: p(vi |θ) = N(vi |W zi + µ, σI)N(zi |0, I)dzi = N(vi |µ, W W T + σI) where W = Wy 0 Wx By and W W T = Wy W T y Wx W T x Wx W T x Wx W T x + Bx BT x Note: Latent ͳΫϥεͷ࣍ݩ, zs i ͕ڞมྔʹಛ༗ͷࢄΛଊ͑ͯ͠·Θͳ ͍Α͏ʹेେ͖ΊʹऔΔඞཁ͕͋Δ. Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 9 / 11 -
PLS ͷΞϧΰϦζϜ PCA ͷ݁ՌΛ regression ʹೖΕΔํ๏ (Primary component regression) ͱؔ͠
͍ͯΔ͕, ͪΐ ͬͱҧ͏. Solution path ࢄ͕େ͖͘, ͔ͭ y ͱ૬ؔͷߴ͍ํΛ୳ࡧ͍ͯ͠Δ cf. PCR ಛྔͷࢄΛେ͖͘͢Δ͜ͱ͚ͩʹ͍ͯ͠Δ. Frank et al.(1993) ʹΑΔͱ Ridge ճؼΑΓ༧ଌੑೳͰྼΔ͕࣍ݩͷ ॖ͕Մೳ. ਪఆʹ Wold (1975) ͷ NIPALS ΞϧΰϦζϜ͕༗໊ (Hastie et al. (2001)). Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 10 / 11 -
Canonical correlation analysis (CCA) ਖ਼४૬ؔੳͱ͍͏໊લͰֶ෦ͷ࣌ͱ͔ʹशͬͨΑ͏ͳؾ͕͢Δ. ࣜҎԼ (Bach and Jordan, (2005)).
p(zi ) = N(zs i |0, ILs )N(zx i |0, ILx )N(zy i |0, ILy p(yi |zi ) = N(yi |By zy i + Wy zs i + µy , σ2IDy ) p(xi |zi ) = N(xi |Bx zx i + Wx zs i + µx , σ2IDx ) PLS Λ synmetric ʹͨ͠ͷ. ͭ·Γ, zi Λڞ௨ͷ zs i ͱ zx i ͱ zy i ʹղ͢Δ ͜ͱ. vi ͷ͖݅: p(vi |θ) = N(vi |W zi + µ, σI)N(zi |0, I)dzi = N(vi |µ, W W T + σID) where W = Wx Bx 0 Wy 0 By and W W T = Wx W T x + Bx BT x Wx W T y Wy W T y Wy W T y + By BT y MLE Λ EM Ͱղ͘ classic ͳ non-probabilistic ͳ݁ՌͱҰக͢Δ (Bach and Jordan, (2005)) Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 11 / 11