Wakariyasui PR ch6 sec1-3

chapter6 section1-3 intro-ml at intro-ml

Transcript

1. Θ͔Γ΍͍͢ύλʔϯೝࣝ ୈ 6 ষ ಛ௃ۭؒͷม׵ ࣡૔جจ ஜ೾େֶ ৘ใֶ܈ ஌ࣝ৘ใɾਤॻֶؗྨ ख௩ݚڀࣨ 01/22/2017
2. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ Contents 1 ಛ௃બ୒ͱಛ௃ۭؒͷม׵ 2 ಛ௃ྔͷਖ਼نԽ 3 KL ల։ 2 / 43
3. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ 1 ಛ௃બ୒ͱಛ௃ۭؒͷม׵ 2 ಛ௃ྔͷਖ਼نԽ 3 KL ల։ 3 / 43
4. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ύλʔϯೝࣝ (Pattern recognition) (෮श) ύλʔϯೝࣝͷྲྀΕ ೝࣝ (Recognition)(pp.1-2) 1 લॲཧ (Preprocessing) 2 ಛ௃நग़ (Feature extraction) 3 ࣝผ (Classfication) ࠓճ͸ಛ௃நग़ͷ෦෼ͷ࿩Ͱ͢ɽ 4 / 43
5. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ಛ௃நग़ (Feature extraction) ಛ௃நग़ ๲େͳ৘ใΛ࣋ͭݪύλʔϯ͔Βࣝผʹඞཁͳຊ࣭తͳಛ௃ͷΈΛ நग़͢Δ ྫ: ੑผͷࣝผͷͨΊͷಛ௃நग़ ਎௕ ൅ͷ௕͞ ෰ͷ৭ etc... 5 / 43
6. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ಛ௃நग़ͷ໰୊ ಛ௃நग़ͷ໰୊ 1 ಛ௃ϕΫτϧͷ֤੒෼ͷεέʔϦϯά 2 ಛ௃ۭؒͷ࣍ݩ਺ ໰୊ 1 ͷྫ: ਎௕Λ km, ൅ͷ௕͞Λ mm ͷεέʔϧ (ई౓)(Scale) Ͱܭଌ ໰୊ 2 ͷྫ: 1000 ݸͷಛ௃Λநग़ (1. ਎௕, 2. ൅ͷ௕͞, 3. ෰ͷ৭,ʜ, 1000. ଍ͷେ͖͞) 6 / 43
7. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ಛ௃நग़ͷ໰୊ 1. ಛ௃ϕΫτϧͷ֤੒෼ͷεέʔϦϯά ໰୊఺ ύλʔϯؒͷྨࣅੑͱಛ௃্ۭؒͷڑ཭͕Ұக͠ͳ͍ ಛ௃ͷॏΈ͕ዞҙతʹภΔ ݪҼ ಛ௃ϕΫτϧͷ੒෼ͷ୯Ґ (εέʔϧ) Λม͑Δͱ෼෍͕ม ΘΔ վળํ๏ ಛ௃ϕΫτϧͷਖ਼نԽ (Normalization) 7 / 43
8. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ಛ௃நग़ͷ໰୊ 2. ಛ௃ۭؒͷ࣍ݩ਺ ໰୊఺ ಛ௃ྔ͕େ͖͗͢Δͱޡࣝผ཰্͕ঢ͠΍͍͢ ݪҼ ૬ؔͷߴ͍ಛ௃ͷ૊͕ࠞೖ͢ΔՄೳੑ͕ߴ·Δ (਎௕ͱ଍ͷେ ͖͞) ࣍ݩͷढ͍ (Curse of dimensionality): ܭࢉྔ͕গͳ͘ͱ΋࣍ݩ d ͷ΂͖৐ͷΦʔμʔ (p.67) ώϡʔζͷݱ৅ (Hughes phenomena): ύλʔϯ਺͕༗ݶݸͷ ࣌ʹ࣍ݩΛ૿΍͗͢͠Δͱޡࣝผ཰্ঢ (όΠΞεʹىҼ) 8 / 43
9. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ಛ௃நग़ͷ໰୊ 2. ಛ௃ۭؒͷ࣍ݩ਺ վળํ๏ ಛ௃બ୒: ͋Δج४ʹଇͬͯಛ௃ۭؒͷ࣍ݩ࡟ݮ (Dimensionality reduction) Λߦ͏ d ࣍ݩͷಛ௃ϕΫτϧ͔Β༗༻ͳ੒෼ͷΈर͍ग़ͯ͠ ˜ d(< d) ࣍ ݩͷಛ௃ϕΫτϧΛߏ੒͢Δํ๏ (d ͕େ͖͍ͱ͖ܭࢉྔ͕ ๲େ) ݪಛ௃ϕΫτϧΛಛఆͷج४ͷ΋ͱʹΑΓখ͞ͳ࣍ݩͷಛ௃ϕ Ϋτϧʹม׵͢Δํ๏ (KL ల։) 9 / 43
10. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ಛ௃ۭؒͷม׵ (Transformation of feature space) ਖ਼نԽ΍ಛ௃બ୒ͳͲɼݪಛ௃ϕΫτϧΛͦͷޙͷॲཧʹదͨ͠ܗ ʹม׵͢Δૢ࡞ ઢܗม׵ y ม׵ޙͷϕΫτϧ = AT ม׵ߦྻ (Transformation matrix) x ݪಛ௃ϕΫτϧ 10 / 43
11. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ 1 ಛ௃બ୒ͱಛ௃ۭؒͷม׵ 2 ಛ௃ྔͷਖ਼نԽ 3 KL ల։ 11 / 43
12. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ಛ௃ྔͷਖ਼نԽ ਖ਼نԽ ύλʔϯ૬ޓͷڑ཭Λ࠷খԽ d ࣍ݩಛ௃্ۭؒͷ n ݸͷύλʔϯू߹ͷதͰɼp ൪໨ͷύλʔϯ xp ʹਖ਼نԽͷͨΊͷม׵ߦྻ A Λ༻͍ͯઢܗม׵͠ਖ਼نԽ͞Εͨ ύλʔϯΛ yp ͱ͢Δɽ yp = ATxp xp = (xp1, xp2, · · · , xpd)T yp = (yp1, yp2, · · · , ypd)T A =      a11 0 a22 ... 0 ad      12 / 43
13. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ p ൪໨ͷύλʔϯͷଞͷύλʔϯͱͷฏۉڑ཭ rp n ݸͷύλʔϯू߹ͷதͷ p ൪໨ͷύλʔϯͱɼଞͷ (n − 1) ݸͷύλʔϯͱͷ ฏۉڑ཭Λ rp ͱ͢Δɽ rp 2 = 1 n − 1 n ∑ q=1 d ∑ j=1 (ypj − yqj)2 ∥yp − yq ∥2 = d ∑ j=1 (ypj − yqj)2 13 / 43
14. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ύλʔϯ૬ޓͷڑ཭ R ਖ਼نԽޙͷ֤ύλʔϯײͷ ฏۉڑ཭Λ R ͱ͢Δ (ύ λʔϯ૬ޓͷڑ཭)ɽ ཁૉ͝ͱʹॻ͘ͱ ypj = ajxpj ͱͳΔɽ͜͜ Ͱ ¯ x ͸ x ͷू߹ฏۉͰ ͋Δɽ R2 = 1 n n ∑ p=1 rp 2 = 1 n(n − 1) n ∑ p=1 n ∑ q=1 d ∑ j=1 (ypj − yqj)2 = · · · = 2n n − 1 d ∑ j=1 a2 j ( ¯ x2 j − ¯ xj 2) 14 / 43
15. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ෆภ෼ࢄ σ2 j ͜͜Ͱ j ൪໨ͷಛ௃ xj ͷ ෆภ෼ࢄ σ2 j ͸ӈͷΑ͏ʹ ͳΔɽ ͪͳΈʹɼ ¯ x2 j = 1 n n ∑ p=1 xpj 2 ͱͳΔɽ(ಛ௃ j ʹؔͯ͠ɼ ཁૉͷ 2 ৐ͷύλʔϯ ฏۉ) σ2 j = 1 n − 1 n ∑ p=1 (xpj − ¯ xj)2 = 1 n − 1 n ∑ p=1 (xpj 2 − 2xpj ¯ xj + ¯ xj 2) = 1 n − 1 (n ¯ x2 j − 2n ¯ xj 2 + n ¯ xj 2) = n n − 1 ( ¯ x2 j − ¯ xj 2) 15 / 43
16. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ෼ࢄ σ2 j Λ༻͍ͨύλʔϯ૬ޓͷڑ཭ ͭ·Γɼύλʔϯ૬ޓͷڑ ཭ͷ 2 ৐ R2 ͸෼ࢄ σ2 j Λ ༻͍ͯӈͷΑ͏ʹදͤΔɽ ͜ΕΛ࠷খԽ!! R2 = 2 d ∑ j=1 a2 j σ2 j 16 / 43
17. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ໰୊ͷઃఆ R2 Λ੍໿৚݅ͷ΋ͱ࠷খ Խ͢Δɽ ੍໿৚݅͸ɼಛ௃ۭؒͷ୯ Ґ௒ཱํମͷମੵ (௒ମੵ) Λɼਖ਼نԽͷલޙͰҰఆʹ อͭͱ͍͏৚݅ʹ૬౰ɽ min. R2 = 2 d ∑ j=1 a2 j σ2 j subject. to d ∏ j=1 aj = 1 17 / 43
18. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ࠷దԽ໰୊Λղ͘ ϥάϥϯδϡͷະఆ৐਺๏ Λ༻͍ͯ (Appendix ࢀ র)R2 Λ࠷খԽ͢Δ aj Λ ٻΊΔɽ ϥάϥϯδϡؔ਺ L Λɼ ϥάϥϯδϡະఆ৐਺ λ Λ ༻͍ͯͭ͘Δɽ aj Ͱภඍ෼͠ 0 ͱ͓͘ɽ L = 2 d ∑ j=1 a2 j σ2 j − λ( d ∏ j=1 aj − 1) ∂L ∂aj = 0 18 / 43
19. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ࠷దԽ໰୊Λղ͘ ภඍ෼ͷ݁Ռ = 0 ͱஔ͍ ͨࣜͷ྆ลʹ aj Λ͔͚Ҡ ߲͢Δɽ·ͨɼ੍໿৚݅Α Γӈล͸ λ ʹͳΔɽΑͬ ͯɼaj Λ λ Λ༻͍ͯද͢ ͜ͱ͕Ͱ͖Δɽ 4ajσ2 j − λ d ∏ k̸=j ak = 0 4aj 2σ2 = λ d ∏ j=1 aj =1 aj = √ λ 2σj 19 / 43
20. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ࠷దԽ໰୊Λղ͘ ੍໿৚݅ͷࣜͷ aj ʹ √ λ 2σj Λ୅ೖ͢Δɽ྆ลΛ 2 d ৐ ͢Δɽ ͕ͨͬͯ͠ɼaj Λ σj Λ༻ ͍ͯදͤΔɽ ͜ΕͰม׵ߦྻ A ͕ܭࢉ Ͱ͖Δɽ d ∏ j=1 √ λ 2σj = 1 √ λ d = 2d d ∏ j=1 σj λ = 4( d ∏ j=1 σj)2 d aj = 1 σj ( d ∏ k=1 σk)1 d ಛ௃࣠ j Λඪ४ภࠩ σj Ͱਖ਼نԽ 20 / 43
21. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ಛ௃ྔͷਖ਼نԽͷ·ͱΊ ਖ਼نԽ ύλʔϯ૬ޓͷڑ཭Λ࠷খԽ ⇔ ֤ಛ௃࣠Λඪ४ภࠩͰਖ਼نԽ͠ɼ ฏۉͷपΓͷ෼ࢄΛ౳͘͢͠ΔΑ͏ͳม׵ߦྻ A Ͱઢܗม׵ yp = ATxp A =      a11 0 a22 ... 0 ad      aj = 1 σj ( d ∏ k=1 σk)1 d ಛ௃࣠ j Λඪ४ภࠩ σj Ͱਖ਼نԽ 21 / 43
22. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ 1 ಛ௃બ୒ͱಛ௃ۭؒͷม׵ 2 ಛ௃ྔͷਖ਼نԽ 3 KL ల։ 22 / 43
23. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ [͓͞Β͍] ઢܕۭؒ ઢܕۭؒͷجఈͱ෦෼ۭؒ ఆٛ (ઢܕۭؒͷجఈ) ઢܕۭؒ V ͷݩͷ૊ {a1, a2, · · · , an} ͕࣍ͷ 2 ͭͷੑ࣭Λຬͨ͢ ͱ͖ɼ͜ΕΛ V ͷجఈͱ͍͏ɽ a1, a2, · · · , an ͸ઢܗಠཱͰ͋Δ V ͷ೚ҙͷݩ x ͸ɼa1, a2, · · · , an ͷઢܗ݁߹ͰදͤΔ ఆٛ (෦෼ۭؒ) ઢܕۭؒ V ͷۭू߹Ͱͳ͍෦෼ू߹ W ͕ɼ V ͱಉ͘͡࿨ͱεΧ ϥʔഒͷԋࢉʹରͯ͠ɼઢܗۭؒʹͳ͍ͬͯΔͱ͖ɼW Λ V ͷ෦ ෼ۭؒͱ͍͏ɽ 23 / 43
24. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ KL ల։ [1] ࣍ݩ࡟ݮͷͨΊͷج४ KL ల։ (KL expansion) ઢܕۭؒʹ͓͚Δಛ௃ϕΫτϧͷ෼෍Λ࠷΋Αۙ͘ࣅ͢Δ෦෼ۭؒ (subspace) ΛٻΊΔํ๏ (ओ੒෼෼ੳ (Principal component) ͱ΄ͱ ΜͲ౳Ձ) KL ల։ (KL expansion) ʹΑΔ࣍ݩ࡟ݮͷධՁج४ ෼ࢄ (Variance) ࠷େج४ ฏۉೋ৐ޡࠩ (Mean-square error) ࠷খج४ 24 / 43
25. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ෼ࢄ࠷େج४ʹجͮ͘ KL ల։ [2] ෼ࢄ࠷େج४ ม׵ޙͷ ˜ d ࣍ݩ෦෼ۭؒʹ͓͍ͯύλʔϯͷ͹Β͖͕ͭΑΓେ͖͚ Ε͹ͦͷ෦෼ۭؒ͸ݪಛ௃ۭؒͰͷύλʔϯͷ෼෍ͷಛ௃ΛΑΓΑ ͘อଘۭͨؒ͠Ͱ͋ΔͱΈͳ͢͜ͱ͕Ͱ͖Δɽ ͭ·Γɼม׵ޙͷύλʔϯ෼෍ͷ෼ࢄΛ࠷େʹ͢Ε͹Α͍ɽ ෼ࢄ࠷େج४ʹجͮ͘ KL ల։ ˜ d ࣍ݩ෦෼ۭؒͷม׵ޙͷύλʔϯͷ෼ࢄ ˜ σ2(A) Λ࠷େʹ͢Δม׵ ߦྻ A ΛٻΊΔɽͨͩ͠ɼA ͷཁૉϕΫτϧ͸ਖ਼ن௚ަجఈͰ ͋Δɽ Max. ˜ σ2(A) subject. to ATA = I 25 / 43
26. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ݩͷ d ࣍ݩಛ௃ۭ͔ؒΒ ˜ d ࣍ݩ෦෼ۭؒ΁ͷม׵ d ࣍ݩಛ௃্ۭؒͷಛ௃ϕΫτϧ xp Λม׵ߦྻ A Λ༻͍ͯ ˜ d ࣍ݩ ෦෼্ۭؒͷ y ʹઢܗࣸ૾͢Δɽ y = ATx A = (u1, u2, · · · , u ˜ d ) ͨͩ͠ɼ(u1, u2, · · · , u ˜ d ) ͸ਖ਼ن௚ަجఈͰ͋ΔͷͰ ui Tuj = δij. ͕ͨͬͯ͠ɼ ATA = I 26 / 43
27. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ४උ ύλʔϯ਺ nɼݪಛ௃ۭؒ Ͱͷύλʔϯฏۉ m Λ༻ ͍ͯɼ෦෼ۭؒͰͷύλʔ ϯͷฏۉ ˜ m ͱɼύλʔϯ ू߹ͷݪಛ௃ۭؒʹ͓͚Δ ڞ෼ࢄߦྻΛΛද͢ɽ ·ͨɼTrA ͸ਖ਼ํߦྻ A ͷର֯੒෼ͷ࿨Ͱ͋Δɽ ˜ m = 1 n ∑ y∈Y y = 1 n ∑ x∈X ATx = ATm Σ = 1 n ∑ x∈X (x − m)(x − m)T xTx = Tr(xxT) 27 / 43
28. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ෦෼ۭؒͰͷύλʔϯͷ෼ࢄ ˜ σ2(A) ෦෼ۭؒͰͷύλʔϯͷ෼ࢄ ˜ σ2(A) Λ෦෼ۭؒͰͷύλʔϯฏۉ ˜ m ͱύλʔϯू߹ͷݪಛ௃ۭؒʹ͓͚Δڞ෼ࢄߦྻ Σ Λ༻͍ͯ ද͢ɽ ˜ σ2(A) = 1 n ∑ y∈Y (y − ˜ m)T(y − ˜ m) = 1 n ∑ x∈X (AT(x − m))T (AT(x − m)) = 1 n ∑ x∈X Tr(AT(x − m)(AT(x − m))T ) = Tr(AT 1 n ∑ x∈X ((x − m)(x − m)T)A) = Tr(ATΣA) 28 / 43
29. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ໰୊ͷઃఆ ˜ σ2(A) Λ੍໿৚݅ͷ΋ͱ࠷ খԽ͢Δɽ ੍໿৚݅͸ɼجఈͷਖ਼ن௚ ަੑɽ Max. ˜ σ2(A) = Tr(ATΣA) subject. to ATA = I 29 / 43
30. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ࠷దԽ໰୊Λղ͘ ˜ σ2(A) Λ੍໿৚݅ͷ΋ͱ࠷ খԽ͢Δɽ ϥάϥϯδϡͷະఆ৐਺๏ ͷΑ͏ʹɼ੍໿ͷ߲Λ௥Ճ ͨؔ͠਺Λఆٛ͠ɼA Ͱภ ඍ෼ͨ͠ 0 ͱ͓͘ɽΛ ͸ ˜ d ࣍ݩର֯ߦྻͰ͋Δɽ J(A) = Tr(ATΣA) − Tr((ATA − I)Λ) A Ͱภඍ෼ͨ͠ 0 ͱ͓͘ͱɼ 2ΣA − 2AΛ = 0. ͨͩ͠ɼ(n, m) ߦྻ Xɼn ࣍ਖ਼ํߦྻ Y ʹ͍ͭͯɼ ∂ ∂X Tr(XTY X) = (Y + Y T)X. ∂ ∂X Tr(XTXY ) = X(Y T + Y ). 30 / 43
31. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ࠷దԽ໰୊Λղ͘ ͕ͨͬͯ͠ɼରশߦྻͷݻ ༗஋໰୊Ͱ͋Γɼ ΣA = AΛ ATΣA = Λ ͱͳΓɼA ͸ Σ Λର֯Խ ͢ΔߦྻͰ͋Δɽ ΑͬͯɼΛ ͸ Σ ͷ d ݸͷ ݻ༗஋ λi(λ1 ≥ · · · ≥ λd) ͔ΒͳΔ ˜ (d) ࣍ݩର֯ߦྻ Ͱ͋Δɽͭ·Γɼ෼ࢄ ˜ σ2A Λ࠷େʹ͢Δʹ͸ɼ্Ґ ˜ (d) ݸͷݻ༗஋ͷ૯࿨ͱ ͳΔɽ max{˜ σ2A} = max{Tr(ATΣA)} = max{TrΛ} = ˜ d ∑ i=1 λi 31 / 43
32. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ෼ࢄ࠷େج४ʹجͮ͘ KL ల։ͷ·ͱΊ [2] ෼ࢄ࠷େج४ ෼ࢄ࠷େج४ʹجͮ͘ KL ల։ ˜ d ࣍ݩ෦෼ۭؒͷม׵ޙͷύλʔϯͷ෼ࢄ ˜ σ2(A) Λ࠷େʹ͢Δม׵ ߦྻ A ΛٻΊΔɽͨͩ͠ɼA ͷཁૉϕΫτϧ͸ਖ਼ن௚ަجఈͰ ͋Δɽ Max. ˜ σ2(A) subject. to ATA = I Αͬͯɼύλʔϯू߹ͷݪಛ௃ۭؒʹ͓͚Δڞ෼ࢄߦྻ Σ ͷ্Ґ ˜ d ݸͷݻ༗஋ λ1, · · · , λ˜ d ʹରԠ͢Δ ˜ d ݸͷਖ਼ن௚ަݻ༗ϕΫτϧΛ ྻͱ͢Δߦྻ͕ A Ͱ͋Δɽ 32 / 43
33. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ฏۉೋ৐ޡࠩ࠷খج४ʹجͮ͘ KL ల։ [2] ฏۉೋ৐ޡࠩ࠷খج४ ฏۉೋ৐ޡࠩج४ʹجͮ͘ KL ల։ ˜ d ࣍ݩ෦෼ۭؒͷม׵ޙͷύλʔϯΛݪಛ௃ۭؒʹࣸ૾ͨ͠΋ͷͱ ݪಛ௃ۭؒͷύλʔϯͱͷࠩΛ࠷খʹ͢Δม׵ߦྻ A ΛٻΊΔɽͨ ͩ͠ɼA ͷཁૉϕΫτϧ͸ਖ਼ن௚ަجఈͰ͋Δɽ min. ε2(A) subject. to ATA = I 33 / 43
34. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ४උ ύλʔϯ਺ nɼݪಛ௃ۭؒ Ͱͷύλʔϯฏۉ m Λ༻ ͍ͯɼ෦෼ۭؒͰͷύλʔ ϯͷฏۉ ˜ m ͱɼύλʔϯ ू߹ͷݪಛ௃ۭؒʹ͓͚Δ ڞ෼ࢄߦྻΛΛද͢ɽ ·ͨɼR ͸ࣗݾ૬ؔߦྻΛ ද͢ɽ ͜ͷؒʹ͸ӈͷΑ͏ͳؔ܎ ͕͋Δɽ ˜ m = ATm Σ = 1 n ∑ x∈X (x − m)(x − m)T R = 1 n ∑ x∈X xxT Σ = 1 n ∑ x∈X (x−m)(x−m)T = R−mmT 34 / 43
35. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ม׵ A ʹΑΔฏۉೋ৐ޡࠩ ε2(A) y(= Ax) Λݩͷ࠲ඪͰݟͨ Ay ͱ x ͱͷڑ཭ (ޡࠩ) Λ༻͍ͨฏۉ ೋ৐ޡࠩΛࣗݾ૬ؔߦྻ R Ͱද͢ɽ ε2(A) = 1 n ∑ (Ay − x)T(Ay − x) = 1 n ∑ (AATx − x)T (AATx − x) = 1 n ∑ ((AATx)TAATx − xTAATx − (AATx)Tx + xTx) = 1 n ∑ (xAATAATx − xTAATx − xTAATx + xTx) = 1 n ∑ (xTx − xTAATx) = 1 n ∑ (Tr(xTx) − Tr(ATxxTA)) = TrR − Tr(ATRA) 35 / 43
36. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ໰୊ͷઃఆ ε2(A) Λ੍໿৚݅ͷ΋ͱ࠷ খԽ͢Δɽ ੍໿৚݅͸ɼجఈͷਖ਼ن௚ ަੑɽ ͜Ε͸ɼ੍໿ͷ΋ͱ Tr(ATRA) Λ࠷େԽ͢Δ ͜ͱͱ౳ՁͰ͋Δɽ min. ε2(A) = TrR − Tr(ATRA) subject. to ATA = I ⇔ Max. Tr(ATRA) subject. to ATA = I 36 / 43
37. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ࠷దԽ໰୊Λղ͘ ෼ࢄ࠷େԽج४ͱಉ༷ʹ R ͷݻ༗஋Λ λi(λ1 ≥ · · · ≥ λd) ͱͯ͠ ͜ͷ࠷దԽ໰୊Λղ͘ɽ max{Tr(ATRA)} = max{TrΛ} = ˜ d ∑ i=1 λi ͱͳΔɽ ͭ·Γɼ min{ε2(A)} = TrR − max{Tr(ATRA)} = TrR − max{TrΛ} = TrR − ˜ d ∑ i=1 λi ͱͳΓɼR ͷ্Ґ ˜ d ݸͷݻ༗஋ λ1, · · · , λ˜ d ʹରԠ͢Δਖ਼ن௚ަݻ༗ϕΫ τϧΛྻͱ͢Δߦྻ͕ A ͱͳΔɽ 37 / 43
38. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ෼ࢄ࠷େج४ͱͷൺֱ ෼ࢄ࠷େԽج४Ͱٻ·Δ෦ ෼ۭؒͱલड़ͷΑ͏ͳฏۉ ೋ৐ޡࠩ࠷খج४ͰٻΊͨ ෦෼ۭؒ͸ҟͳΔɽ ݪҼ ޡࠩΛٻΊΔࡍͷج४͕ ݪ఺ ಉ෦෼ۭؒʹ͢Δํ๏ ෦෼ۭؒͷ࠲ඪͷج४Λݪ ఺͔ΒฏۉϕΫτϧ m ʹ ͢Δ m ͚ͩฏߦҠಈ ࣗݾ૬ؔߦྻΛฏۉϕΫτϧ͚ͩฏߦҠ ಈͨ͠΋ͷ͕ڞ෼ࢄߦྻͰ͋ΔͷͰɼ Σ = R ͱͳΓɼ ε2(A) = TrR − Tr(ATRA) = TrR − Tr(ATΣA) ͱͳΔɽ͕ͨͬͯ͠ɼ͜ͷͱ͖ٻ·Δ A ͸෼ࢄ࠷େج४ʹΑͬͯٻ·Δ A ʹ౳ ͍͠ɽ 38 / 43
39. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ฏۉೋ৐ޡࠩ࠷খج४ʹ͓͚Δ࠷దͳฏߦҠಈ ฏߦҠಈ m ͸࠷దͳ ͷ͔ʁ ฏߦҠಈΛ x0 ͱͯ͠ A Λ ٻΊͯΈΔɽ y = AT(x − x0) ΑΓɼy Λݪಛ௃্ۭؒͰΈΔͱ Ay + x0 ͱͳΔɽ ·ͨɼਖ਼ن௚ަجఈͷ੍໿ ΑΓɼ QTQ = Q Q = I − AAT ͱ͢Δͱɼ ฏۉೋ৐ޡࠩ ε2(A) ͸ӈͷ Α͏ʹͳΔɽ (Ay + x0) − x = AAT(x − x0) + x0 − x = (AAT − I)(x − x0) ε2(A, x0) = 1 n ∑ x∈X (Q(x − x0))TQ(x − x0) = 1 n ∑ x∈X (x − x0)TQTQ(x − x0) = 1 n ∑ x∈X (x − x0)TQ(x − x0) 39 / 43
40. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ฏۉೋ৐ޡࠩ࠷খج४ʹ͓͚Δ࠷దͳฏߦҠಈ ͜͜Ͱ͸ɼ௚ަࣹӨߦྻ ͕ɼP = AAT Ͱ͋Γɼ Q = I − AAT = I − P ͸ɼA ͷுΔ෦෼ۭؒͷ௚ ަิۭؒ΁ͷ௚ަࣹӨߦྻ ͱͳΔɽ ͜͜Ͱɼε2(x0) Λ x0 Ͱภඍ෼͠ 0 ͱஔ ͘ͱ ∂ε2(x0) ∂x0 = 1 n ∑ (2Qx0 − Qx) = 2Q(x0 − m) = 0 ͱͳΓɼ Qx0 = Qm. 40 / 43
41. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ฏۉೋ৐ޡࠩ࠷খج४ʹ͓͚Δ࠷దͳฏߦҠಈ ͜ΕΛ,ε2(A, x0) ʹ୅ೖ͢ ΔͱɼӈͷΑ͏ʹͳΔɽ ͕ͨͬͯ͠ɼݪ఺Λ m Ͱ ฏߦҠಈ্ͨ͠Ͱ ε2(A) Λ ࠷খʹ͢Δม׵ߦྻ A ͸ Σ ͷ্Ґ ˜ d ݸͷݻ༗஋ λ1, · · · , λ˜ d ʹରԠ͢Δ ˜ d ݸ ͷਖ਼ن௚ަݻ༗ϕΫτϧΛ ྻͱ͢ΔߦྻͱͳΔɽ ε2(A) = 1 n ∑ x∈X (Q(x − m))TQ(x − m) = 1 n ∑ x∈X (x − m)TQTQ(x − m) = 1 n ∑ x∈X (x − m)TQ(x − m) = 1 n ∑ x∈X (x − x0)T(I − AAT)(x − x0) = TrΣ − Tr(ATΣA) 41 / 43
42. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ ฏۉೋ৐ޡࠩ࠷খج४ʹجͮ͘ KL ల։ͷ·ͱΊ [2] ฏۉೋ৐ޡࠩ࠷খج४ ฏۉೋ৐ޡࠩ࠷খج४ʹجͮ͘ KL ల։ ˜ d ࣍ݩ෦෼ۭؒͷݪಛ௃ۭؒʹ͓͚Δม׵ޙͷύλʔϯͱͷฏۉೋ ৐ޡࠩ ε2(A) Λ࠷খʹ͢Δม׵ߦྻ A ΛٻΊΔɽͨͩ͠ɼA ͷཁ ૉϕΫτϧ͸ਖ਼ن௚ަجఈͰ͋Δɽ min. ε2(A) subject. to ATA = I Αͬͯɼύλʔϯू߹ͷݪಛ௃ۭؒʹ͓͚Δ m ͚ͩฏߦҠಈ͠ ͨࣗݾ૬ؔߦྻ R ͷ্Ґ ˜ d ݸͷݻ༗஋ λ1, · · · , λ˜ d ʹରԠ͢Δ ˜ d ݸ ͷਖ਼ن௚ަݻ༗ϕΫτϧΛྻͱ͢Δߦྻ͕ A Ͱ͋Δɽ 42 / 43
43. ಛ௃બ୒ͱಛ௃ۭؒͷม׵ ಛ௃ྔͷਖ਼نԽ KL ల։ KL ల։ʹΑΔ࣍ݩ࡟ݮͷ·ͱΊ KL ల։ʹΑΔ࣍ݩ࡟ݮ KL ల։ʹΑΔ࣍ݩ࡟ݮ ڞ෼ࢄߦྻ Σ ͷ্Ґݻ༗஋ʹରԠ͢Δਖ਼ن௚ަݻ༗ϕΫτϧΛجఈ ͱͨ͠෦෼ۭؒΛ༻͍ͨํ๏ 43 / 43
44. Appendix: ౳੍ࣜ໿෇ತܭը໰୊ ౳੍ࣜ໿෇ತܭը໰୊ Max. f(x) subject. to g(x) = 0 ղ๏: ϥάϥϯδϡͷະఆ৐਺๏ ϥάϥϯδϡؔ਺ LL Λ༻ҙ͢Δɽ L(x, λ) = f(x) + λg(x) ͜ͷؔ਺ͷ x ʹؔ͢Δภඍ෼͕ 0 ͱͳΓɼ͔ͭ༩͑ΒΕ੍ͨ໿͕ຬ ͨ͞ΕΔͱ͖ɼ࠷దͳղ͕ಘΒΕΔɽ 44 / 43
45. Appendix: ౳੍ࣜ໿෇ತܭը໰୊ ͭ·Γɼ࿈ཱํఔࣜ ∇xf(x) + λ∇xg(x) = 0 g(x) = 0 Λղ͘͜ͱʹͳΔɽ ղੳతʹղ͘ ਺஋తʹղ͘ (ޯ഑߱Լ๏ɼχϡʔτϯ๏ͳͲ) 45 / 43
46. Appendix: ౳੍ࣜ໿෇ತܭը໰୊ ͳʹ͕ߦΘΕ͍ͯΔͷ͔? Ұͭ໨ͷ౳ࣜ͸ҎԼͷΑ͏ʹมܗͰ͖Δɽ ∇xf(x) = −λ∇xg(x) ͜Ε͔Βɼf(x) ͷ౳ߴઢͷ๏ઢϕΫτϧ͕ɼg(x) ͷ๏ઢϕΫτϧ ͷεΧϥʔ λ ഒͰ͋Γɼͦͷํ޲͕౳͍͜͠ͱ͕Θ͔Δɽͭ·Γɼ g(x) ͕౳ߴઢʹ઀͍ͯ͠ΔͨΊͷඞཁ৚݅Λද͍ͯ͠Δɽ·ͨɼ੍ ໿৚݅ʹΑΓ g(x) = 0 ʹຊ౰ʹղ͕৐͍ͬͯΔ͔ͱ͍͏ඞཁ৚ ͕݅ՃΘΓɼ࠷దղΛಋग़͢Δඞཁ৚݅ͱͳΔɽ 46 / 43