Supervised PCA ͱͦͷपล Daisuke Yoneoka February 22, 2015 Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 1 / 11

Notations Latent: zi ∈ RL Observed: xi ∈ RD Outcome: yi ∈ RD PCA: ৴߸ղੳͱ͔ͩͱ Karhunen Loeve transform ͱ͔ͱ΋ݺ͹ΕΔΒ͍͠ Factor analysis: p(zi ) = N(zi |µ0 , Σ0 ), p(xi |zi , θ) = N(W zi + µ, Ψ) W ͸ D × L Ͱ factor loading matrix Ψ ͸ D × D Ͱର֯ߦྻ ͱ͘ʹ Ψ = σ2I ͷͱ͖ probabilistic PCA(PPCA) or sensible PCA (Roweis 1997) PCA ͷઆ໌͸লུ Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 2 / 11

ಋೖ ϖΞʹͳͬͯΔσʔλͱ͔Λ௿࣍ݩʹຒΊࠐΈ͍ͨΈ͍ͨͳཁ๬ ͜ͷཁ๬Λ Virtanen 2010, ʹԊͬͯ Latent Gaussian model Ͱղܾ͍ͨ͠ ҎԼͷख๏Λ Murphy ʹԊͬͯ֓આ supervised PCA discriminative supervised PCA partial least square canonical correlation analysis Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 3 / 11

άϥϑతઆ໌ (Murphy, 2012) Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 4 / 11

Supervised PCA (SPCA) ࣜ͸ҎԼͷ༷ʹͳΔ. Yu et al. (2006) Ͱ͸ Supervised PCA Ͱ, West (2003) Ͱ͸ Bayesian factor regression ͱݺ͹Ε͍ͯΔ. p(zi) = N(0, IL) p(yi |zi) = N(wT y zi + µy, σ2 y ) p(xi |zi ) = N(Wx zi + µx , σ2 x ID ) PCA ͱͷ૬ҧ఺: PCA ͸ x ʹ͔͠஫໨͍ͯ͠ͳ͍͕, SPCA ͸ y ΋ߟྀͯ͠ ͍Δ. Joint Gaussian ͳͷͰ, yi |xi ∼ N(xT i w, σy + wT y Cwy), where w = Ψ−1WxCwy , Ψ = σ2 x ID and C−1 = I + W T x Ψ−1Wx ͕ܭࢉՄೳ. z Ͱ৚͚݅ͭΔͱ x ͱ y ͸ಠཱ: p(y, x|z) = p(y|z)p(x|z) ΛܭࢉͰ͸࢖͏ Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 5 / 11

Supervised PCA ͱ Zeller ͷ g-prior West (2003) ͸, SPCA ͱ Zeller ͷ g-prior ͷؔ܎Λ໌Β͔ʹͨ͠. p(wy ) = N(0, (1/gΣ2)−1) SVD of X ͱͯ͠ X = RV T , V T V = I ͱ͠, RT R = Σ2 = diag(σ2 j ) ͸ಛ ҟ஋Λର֯ʹฒ΂ͨ΋ͷͱ͢Δͱ σ2 x → 0 ͷۃݶͰ p(w) = N(0, gV −T Σ−2V −1) = N(0, g(XT X)−1) Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 6 / 11

Information bottleneck x Λ࣍ݩॖ໿ͯ͠ y Λ༧ଌ͠Α͏ͱ͢ΔΞΠσΟΞ͸৘ใཧ࿦Ͱ͸ҎԼͷΑ͏ʹ දݱՄೳ. ҎԼΛ࠷খԽ͢ΔΑ͏ͳ p(z|z) Λൃݟ͍ͨ͠ I(X; Z) − βI(X; Y ), ͨͩ͠, I(X; Y ) ͸૬ޓ৘ใྔ I(X : Y ) = H(X) − H(X|Y ) = H(X) + H(Y ) − H(X, Y ), ͨͩ͠ H(X) = Ep log 1 p(X) β ≥ 0 ͸ information bottleneck Ͱॖ໿౓߹͍ͱ༧ଌੑೳͷ tradeoff Λௐ੔ Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 7 / 11

Discriminant supervised PCA SPCA ͷ໰୊͸ p(x|z) ͱ p(y|z) ʹಉ͡ॏΈΛ͔͚͍ͯΔ఺. Rish et al. (2008) ͸͜ΕΛղܾ. ॏΈ αx ͱ αy ΛҎԼͷΑ͏ʹಋೖ. l(θ) = Πip(yi |ηiy)αy p(xi |ηix)αx , ͨͩ͠, ηim = Wmzm α ͸ exponential family ͳΒ͹ noise variance ͱͯ͠ղऍՄೳ. ྫ͑͹σʔλ͕ Gaussian: l(θ) ∝ Πi exp(− 1 2 αx ∥xT i − ηix ∥2) Note: α ͷਪఆ͸໬౓ͷ normalizing constant ͕ͦͷ౎౓มԽ͢ΔͷͰɺࠔ ೉Ͱ͋Δ. Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 8 / 11

Partial least square (PLS) ܭྔܦࡁͷํͰ͸༗໊. Ϟσϧ͸ҎԼ. p(zi) = N(zs i |0, ILs )N(zx i |0, ILx ) p(yi |zi ) = N(W T y zs i + µy , σ2IDy ) p(xi |zi ) = N(Wx zs i + Bx zx i + µx , σ2IDx ) ΞΠσΟΞ͸, zi Λڞ௨ͷ zs i ͱ zx i ʹ෼ղ͢Δ͜ͱ. vi = (yi , xi ) ͷ৚݅෇͖෼෍: p(vi |θ) = N(vi |W zi + µ, σI)N(zi |0, I)dzi = N(vi |µ, W W T + σI) where W = Wy 0 Wx By and W W T = Wy W T y Wx W T x Wx W T x Wx W T x + Bx BT x Note: Latent ͳΫϥεͷ࣍ݩ͸, zs i ͕ڞมྔʹಛ༗ͷ෼ࢄΛଊ͑ͯ͠·Θͳ ͍Α͏ʹे෼େ͖ΊʹऔΔඞཁ͕͋Δ. Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 9 / 11

PLS ͷΞϧΰϦζϜ PCA ͷ݁ՌΛ regression ʹೖΕΔํ๏ (Primary component regression) ͱؔ܎͠ ͍ͯΔ͕, ͪΐ ͬͱҧ͏. Solution path ͸෼ࢄ͕େ͖͘, ͔ͭ y ͱ૬ؔͷߴ͍ํ޲Λ୳ࡧ͍ͯ͠Δ cf. PCR ͸ಛ௃ྔͷ෼ࢄΛେ͖͘͢Δ͜ͱ͚ͩʹ஫໨͍ͯ͠Δ. Frank et al.(1993) ʹΑΔͱ Ridge ճؼΑΓ༧ଌੑೳͰ͸ྼΔ͕௿࣍ݩ΁ͷ ॖ໿͕Մೳ. ਪఆʹ͸ Wold (1975) ͷ NIPALS ΞϧΰϦζϜ͕༗໊ (Hastie et al. (2001)). Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 10 / 11

Canonical correlation analysis (CCA) ਖ਼४૬ؔ෼ੳͱ͍͏໊લͰֶ෦ͷ࣌ͱ͔ʹशͬͨΑ͏ͳؾ͕͢Δ. ࣜ͸ҎԼ (Bach and Jordan, (2005)). p(zi ) = N(zs i |0, ILs )N(zx i |0, ILx )N(zy i |0, ILy p(yi |zi ) = N(yi |By zy i + Wy zs i + µy , σ2IDy ) p(xi |zi ) = N(xi |Bx zx i + Wx zs i + µx , σ2IDx ) PLS Λ synmetric ʹͨ͠΋ͷ. ͭ·Γ, zi Λڞ௨ͷ zs i ͱ zx i ͱ zy i ʹ෼ղ͢Δ ͜ͱ. vi ͷ৚݅෇͖෼෍: p(vi |θ) = N(vi |W zi + µ, σI)N(zi |0, I)dzi = N(vi |µ, W W T + σID) where W = Wx Bx 0 Wy 0 By and W W T = Wx W T x + Bx BT x Wx W T y Wy W T y Wy W T y + By BT y MLE Λ EM Ͱղ͘ classic ͳ non-probabilistic ͳ݁ՌͱҰக͢Δ (Bach and Jordan, (2005)) Daisuke Yoneoka Supervised PCA ͱͦͷपล February 22, 2015 11 / 11