AICの原論文Akaike(1973)の勉強用資料

ブログ用の資料です．
http://seetheworld1992.hatenablog.com/entry/2016/12/13/100758

Transcript

1. AIC ͷݪ࿦จ Akaike(1973) ͷษڧ༻ࢿྉ @seetheworld1992 2016/12/1
2. Table of Contents 0. ͸͡Ίʹ 1. AIC ͷಋग़ 2. ܭࢉաఔ
3. 0. ͸͡Ίʹ [1/2] ຊࢿྉ͸ɼAIC ͷݪ࿦จ Akaike(1973) Ͱ AIC ͷಋग़Λษڧ͢Δࡍ ͷॿ͚ͱͳΔΑ͏ʹɼ·ͨܭࢉաఔͷه࿥ͱͳΔ͜ͱΛ໨తͱ͠ ͯ࡞੒ͨ͠ɽ࡞੒ํ਑͸ҎԼͰ͋Δɽ ▶ AIC ͷಋग़෦෼ͷΈΛ·ͱΊΔɽ ▶ ͳΔ΂͘ Akaike(1973) ͷਐߦΛͳͧΔɽ ▶ ͳΔ΂͘ Akaike(1973) ௨Γͷه߸Λ༻͍Δɽ ▶ ܭࢉաఔ΋ͳΔ΂͘ஸೡʹॻ͘ 1ɽ ▶ ඞཁʹԠͯ͡આ໌ΛՃ͑Δɽ ▶ 1 ষͰ͸ AIC ͷಋग़ʹ͍ͭͯه͢ɽ ▶ 2 ষ͸ 1 ষͰলུͨ͠ܭࢉաఔΛه͢ɽ1 ষʹ͋Δ਺ࣜͷ͏ ͪ٭஫ʹޙड़ͱॻ͔Εͨ΋ͷ͸ৄࡉͳܭࢉաఔΛ 2 ষʹه͢ɽ 1࡞੒ऀͷྗෆ଍ʹΑΓᐆດͳ෦෼΋͍͔ͭ͋͘Δ...
4. 0. ͸͡Ίʹ [2/2] ࢀߟจݙ ຊࢿྉΛ࡞੒͢Δʹ͋ͨΓɼҎԼͷจݙΛࢀߟʹͨ͠ɽ Akaike, H., ”Information theory and an extension of the maximum likelihood principle”, Proceedings of the 2nd International Symposium on Information Theory, Petrov, B. N., and Caski, F. (eds.), Akadimiai Kiado, Budapest: 267-281 (1973) J.deLeeuw,”Introduction to Akaike (1973) Information Theory and an Extension of the Maximum Likelihood Principle”,Breakthroughs in Statistics, Part of the series Springer Series in Statistics, Samuel, K., and Norman L, J. (eds.),Springer New York, New York: 599-609 (1992)
5. 1. AIC ͷಋग़ [1/12] Ϟσϧબ୒໰୊ͷઃఆ ֬཰ີ౓ؔ਺ f(x|θ) ͕͋Δͱ͠ɼਅͷ֬཰෼෍ g(x) = f(x|¯ θ) Λ ਪଌ͢Δ໰୊Λߟ͑Δ 2ɽf(x|¯ θ) ʹै͏֬཰ม਺Λ XɼͲͷσʔ λ͕ಘΒΕΔ͔Λҙຯ͢Δ֬཰ม਺Λ Z ͱ͠ɼX ͱ Z ͸ޓ͍ʹಠ ཱͰ͋Δͱ͢Δɽ͜ͷͱ͖ɼ࠷ऴతͳਪఆ஋ͱͯ͠ظ଴ର਺໬౓ EZEX log f(X|θ) = EZ ∫ f(x|¯ θ) log f(x|θ)dx (1) Λ࠷େʹ͢Δਪఆྔ θ Λ࠾༻͢Δ͜ͱʹ͢Δ 3ɽAIC ͸ͦͷΑ͏ ͳ θ Λબ୒͢ΔͨΊͷࢦඪͰ͋ΔɽAIC ͕খ͍͞ϞσϧΛબ୒͢ Δ͜ͱ͸ (1) Λେ͖͘͢Δ θ Λબ୒͢Δ͜ͱͱ౳ՁͱͳΔɽҎ্ ͷઃఆʹΑΓɼຊࢿྉʹ͓͚ΔϞσϧબ୒ͷ໰୊͸ύϥϝʔλͷ ࣍ݩΛܾఆ͢Δ໰୊ͱͯ͠ղऍͰ͖Δɽ 2AIC ͸૬ҟͳΔ֬཰ີ౓ؔ਺ (Ϟσϧ) ͷൺֱʹ࢖༻͢Δ͜ͱ΋ՄೳͰ͋Δ ͕ɼ͜͜Ͱ͸ f ͸֬ఆ͍ͯ͠Δ΋ͷͱ͢Δɽ·ͨɼਅͷ֬཰෼෍͸ f(·|ΘL ) Ͱ ͸දݱͰ͖ͳ͍ؔ਺ܗͰ͋Δ͜ͱ΋͋Ζ͏͕ɼ͜͜Ͱ͸ g(·) ∈ f(·|ΘL ) Ͱ͋Δ ͱԾఆ͢ΔɽΘL ͷఆٛ͸ޙड़ɽ 3࠷໬ਪఆͷ֦ுͱղऍͰ͖Δɽޙड़͢Δ͕ɼϕΫτϧ θ ͸ύϥϝʔλۭؒ ͷཁૉͰ͋Δɽ֬཰ม਺ X, Z ʹؔ͢Δظ଴஋Λ EX , EZ ͱه͢ɽ
6. 1. AIC ͷಋग़ [2/12] ࠷໬ਪఆͷར༻ ଛࣦؔ਺ W ͱϦεΫؔ਺ R Λಋೖ͢Δɽ W(θ, ¯ θ) ≡ −2 ∫ f(x|¯ θ) log ( f(x|θ) f(x|¯ θ) ) dx (2) R(θ, ¯ θ) ≡ EZW(θ, ¯ θ) (3) ͢ΔͱલεϥΠυͷٞ࿦ΑΓ (3) Λ࠷খʹ͢Δ θ Λબ୒͢Ε͹Α ͍ɽ࣍ʹσʔλ਺Λ N ͱ͢Δͱେ਺ͷ๏ଇΑΓ DN (θ, ¯ θ) ≡ − 2 N N ∑ i=1 log ( f(xi|θ) f(xi|¯ θ) ) p −→ W(θ, ¯ θ) (4) Ͱ͋ΔͷͰɼ(1/N) ∑ N i=1 log(f(xi|θ)/f(xi|¯ θ)) Λ࠷େʹ͢Δ θ = ˆ θ(Z) ͸ (2) Λ࠷খԽ͢Δ 4ɽҎޙɼͦͷ ˆ θ(Z) ͷԼͰ (3)(͢ͳ Θͪ R(ˆ θ(Z), ¯ θ)) ͷΑ͍ۙࣅΛಋ͍͍ͯ͘ɽ 4͢ͳΘͪ ˆ θ(Z) ͸࠷໬ਪఆྔͰ͋ΔͷͰσʔλ Z ʹґଘ͢Δ͜ͱΛ໌ࣔ͢ Δه๏ͱ͍ͯ͠Δɽ·ͨ ¯ θ ͕ະ஌Ͱ΋ܭࢉՄೳͰ͋Δɽ
7. 1. AIC ͷಋग़ [3/12] ύϥϝʔλۭؒͷఆٛ ࿩͕લޙ͢Δ͕ɼύϥϝʔλϕΫτϧ θ ͸ύϥϝʔλۭؒ ΘL ͷ ཁૉͰ͋Δͱ͢Δ 5ɽ θ ≡ (θ1, · · · , θL)′ ∈ ΘL (5) ੍໿ΛೖΕͨ෦෼ύϥϝʔλۭؒ Θk ͱͦͷཁૉ kθ Λఆٛ͢Δ 6ɽ Θk ≡ {θ ∈ ΘL|θk+1 = θk+2 = · · · = θK = 0} (6) kθ ≡ (kθ1, · · · , kθk, 0, · · · , 0)′ ∈ Θk (7) ҎԼͷΑ͏ͳ࠷໬ਪఆྔΛఆ͓ٛͯ͘͠ɽ ˆ θ(Z) ≡ arg max θ∈ΘL l(θ) (8) k ˆ θ(Z) ≡ arg max θ∈Θk l(θ) (9) l(θ) ≡ 1 N N ∑ i=1 log f(xi|θ) (10) 5͜͜Ͱ ′ ͸సஔΛҙຯ͢ΔɽL ͸ඇৗʹେ͖͍ਖ਼ͷ੔਺ͱߟ͓͚͑ͯ͹Α͍ɽ 6ఆٛΑΓ Θ1 ⊂ Θ2 ⊂ · · · ⊂ Θk ⊂ · · · ⊂ ΘL ͕੒Γཱͭɽ
8. 1. AIC ͷಋग़ [4/12] ϦεΫؔ਺ͷۙࣅ લεϥΠυͷఆٛΑΓɼҎԼͷۙࣅΛԾఆ͢Δɽ R(ˆ θ(Z), ¯ θ) = EZW(ˆ θ(Z), ¯ θ) ≈ EZW(k ˆ θ(Z), ¯ θ) (11) ࣍ʹҎԼͷ֬཰ऩଋΛԾఆ͢Δ 7ɽ DN (k ˆ θ(Z), ˆ θ(Z)) = − 2 N N ∑ i=1 log ( f(xi|k ˆ θ(Z)) f(xi|ˆ θ(Z)) ) p −→ W(k ˆ θ(Z), ¯ θ) (12) Αͬͯ N → ∞ Ͱ͸ R(ˆ θ(Z), ¯ θ) ΛҎԼͷ࠷ӈลͰۙࣅͰ͖Δ͕ (12) ΛԾఆ͢ΔͷͰภΓ (όΠΞε) ͕ଘࡏ͢Δɽ R(ˆ θ(Z), ¯ θ) ≈ EZW(k ˆ θ(Z), ¯ θ) ≈ EZDN (k ˆ θ(Z), ˆ θ(Z)) + Bias (13) Αͬͯ W(k ˆ θ(Z), ¯ θ) ͱ DN (k ˆ θ(Z), ˆ θ(Z)) ΛධՁ͢Ε͹Αͦ͞͏Ͱ ͋Δɽ·ͣ͸લऀͷఆࣜԽʹਐΉɽ 7େ਺ͷ๏ଇΛߟ͑Δࡍʹ ¯ θ Λ࠷໬ਪఆྔ ˆ θ(Z) Ͱ୅༻͍ͯ͠Δͱ͍͑Δɽ
9. 1. AIC ͷಋग़ [5/12] Fisher ৘ใߦྻͳͲͷఆٛ ҎԼͷه๏Λಋೖ͢Δ 8ɽ ||θ(1) − θ(0)||2 C(¯ θ) ≡ (θ(1) − θ(0))′C(¯ θ)(θ(1) − θ(0)) (14) < θ(3)−θ(2), θ(1)−θ(0) >C(¯ θ) ≡ (θ(3)−θ(2))′C(¯ θ)(θ(1)−θ(0)) (15) C(θ) ͸ Fisher ৘ใߦྻͰ͋Γ (l, m) ཁૉ͸ҎԼͰ͋Δ 9ɽ Clm(θ(0)) ≡ ∫ ( ∂f(x|θ) ∂θl 1 f(x|θ) ) ( ∂f(x|θ) ∂θm 1 f(x|θ) ) f(x|θ)dx θ=θ(0) = − ∫ ∂2 log f(x|θ) ∂θl∂θm f(x|θ)dx θ=θ(0) (16) 8Ҏޙɼθ(·) ∈ ΘL ͱ͢Δɽ 9Ҏޙɼl, m ∈ {1, 2, · · · , L} ͱ͢Δɽ࠷ӈลͷಋग़͸ޙड़ɽ
10. 1. AIC ͷಋग़ [6/12] ଛࣦؔ਺ͷۙࣅ ଛࣦؔ਺ W ͸ҎԼͷೋ࣍ܗࣜͰۙࣅͰ͖ 10ɼ݁Ռͱͯ͠લεϥ ΠυͰಋೖͨ͠ه๏ͰදݱͰ͖Δɽ W(θ, ¯ θ) = −2 ∫ f(x|¯ θ) log ( f(x|θ) f(x|¯ θ) ) dx ≈ (θ − ¯ θ)′C(¯ θ)(θ − ¯ θ) = ||θ − ¯ θ||2 C(¯ θ) (17) ¯ θ(∈ ΘL) ͷ Θk ʹର͢ΔࣹӨ k ¯ θ Λಋೖ͢Δɽ k ¯ θ ≡ arg min kθ∈Θk ||kθ − ¯ θ||2 C(¯ θ) (18) ͢ΔͱࡾฏํͷఆཧΑΓҎԼΛಘΔ 11ɽ W(k ˆ θ(Z), ¯ θ) = ||k ˆ θ(Z) − ¯ θ||2 C(¯ θ) = ||k ¯ θ − ¯ θ||2 C(¯ θ) + ||k ˆ θ(Z) − k ¯ θ||2 C(¯ θ) (19) 10Fisher ৘ใߦྻ C(·) ͸ਖ਼ఆ஋ରশߦྻɽςΠϥʔల։ͯ͠ 3 ࣍Ҏ߱ͷ߲Λ ແࢹ͢ΔɽW ͕े෼׈Β͔Ͱ͋Δ͜ͱΛԾఆ͍ͯ͠Δɽۙࣅͷಋग़͸ޙड़ɽ 11࠷ӈลୈ 1 ߲͸ ΘK ʹ௚ަ͢ΔϕΫτϧͷ௕͞Ͱɼୈ 2 ߲͸ ΘK ͷཁૉϕ Ϋτϧͷ௕͞ͱղऍͰ͖Δɽ
11. 1. AIC ͷಋग़ [7/12] ࣍ʹɼDN (k ˆ θ(Z), ˆ θ(Z)) ΛҎԼͷΑ͏ʹมܗ͢Δͱɼ࠷ӈลͷೋ ࣍ܗࣜͷࠩͰۙࣅͰ͖Δ 12ɽ NDN (k ˆ θ(Z), ˆ θ(Z)) = −2 N ∑ i=1 log ( f(xi|k ˆ θ(Z)) f(xi|ˆ θ(Z)) ) = −2 [ − N ∑ i=1 log ( f(xi|k ¯ θ) f(xi|k ˆ θ(Z)) ) + N ∑ i=1 log ( f(xi|k ¯ θ) f(xi|ˆ θ(Z)) )] ≈ −2 [ − 1 2 √ N(k ¯ θ − k ˆ θ(Z))′G(k ˆ θ(Z)) √ N(k ¯ θ − k ˆ θ(Z)) ] − 2 [ 1 2 √ N(k ¯ θ − ˆ θ(Z))′G(ˆ θ(Z)) √ N(k ¯ θ − ˆ θ(Z)) ] = √ N(k ¯ θ − ˆ θ(Z))′(−G(ˆ θ(Z))) √ N(k ¯ θ − ˆ θ(Z)) − √ N(k ¯ θ − k ˆ θ(Z))′(−G(k ˆ θ(Z))) √ N(k ¯ θ − k ˆ θ(Z)) (20) 12ςΠϥʔల։ͯ͠ 3 ࣍Ҏ߱ͷ߲Λແࢹ͢Δɽۙࣅͷಋग़͸ޙड़ɽ
12. 1. AIC ͷಋग़ [8/12] ͜͜Ͱ N → ∞ ͱ͢Δʹ͋ͨΓɼ √ N(k ¯ θ − ¯ θ) ͕༗քͰ͋ΔΑ͏ ʹ N ͱ k ΛͱΕΔͱ͢Δ 13ɽ͜ͷԾఆͷԼͰɼ࠷໬ਪఆྔ k ˆ θ(Z), ˆ θ(Z) ͷ઴ۙਖ਼نੑͱ k ˆ θ(Z), ˆ θ(Z) ͸ͦΕͧΕ k ¯ θ, ¯ θ ͷҰக ਪఆྔͰ͋Δ͜ͱɼ͞Βʹߦྻ −G(·) ͕ Fisher ৘ใߦྻ C(·) ʹ ֬཰ऩଋ͢Δ͜ͱ 14 ͔ΒҎԼΛಘΔɽ NDN (k ˆ θ(Z), ˆ θ(Z)) ≈ √ N(ˆ θ(Z) − k ¯ θ)′(−G(ˆ θ(Z))) √ N(ˆ θ(Z) − k ¯ θ) − √ N(k ˆ θ(Z) − k ¯ θ)′(−G(k ˆ θ(Z))) √ N(k ˆ θ(Z) − k ¯ θ) p −→ √ N(ˆ θ(Z) − k ¯ θ)′C(¯ θ) √ N(ˆ θ(Z) − k ¯ θ) − √ N(k ˆ θ(Z) − k ¯ θ)′C(¯ θ) √ N(k ˆ θ(Z) −k ¯ θ) = N||ˆ θ(Z) − k ¯ θ||2 C(¯ θ) − N||k ˆ θ(Z) − k ¯ θ||2 C(¯ θ) (21) 13͜ͷԾఆΑΓɼN → ∞ ͷͱ͖͋Δʢ࣮਺ͷʣL ࣍ݩϕΫτϧ M ʹ͍ͭͯ √ N(k ¯ θ − ¯ θ) = M Ͱ͋ΔͷͰ k ¯ θ = ¯ θ + 1 √ N M ≈ ¯ θ Ͱ͋Δɽ 14͜ͷಋग़͸ޙड़ɽ
13. 1. AIC ͷಋग़ [9/12] Ҿ͖ଓ͖ҎԼ͕ܭࢉͰ͖Δ 15ɽ NDN (k ˆ θ(Z), ˆ θ(Z)) ≈ N||(ˆ θ(Z) − ¯ θ) − (k ¯ θ − ¯ θ)||2 C(¯ θ) − N||k ˆ θ(Z) − k ¯ θ||2 C(¯ θ) = N||ˆ θ(Z) − ¯ θ||2 C(¯ θ) − 2N < ˆ θ(Z) − ¯ θ, k ¯ θ − ¯ θ >C(¯ θ) +N||k ¯ θ − ¯ θ||2 C(¯ θ) − N||k ˆ θ(Z) − k ¯ θ||2 C(¯ θ) ͜Εͱ (19) ΛลʑҾ͍ͯ੔ཧ͢ΔͱҎԼΛಘΔɽ NW(k ˆ θ(Z), ¯ θ) ≈ NDN (k ˆ θ(Z), ˆ θ(Z)) − ( N||ˆ θ(Z) − ¯ θ||2 C(¯ θ) − N||k ˆ θ(Z) − k ¯ θ||2 C(¯ θ) ) + N||k ˆ θ(Z) − k ¯ θ||2 C(¯ θ) − 2N < ˆ θ(Z) − ¯ θ, k ¯ θ − ¯ θ >C(¯ θ) (22) 15΋͠ ¯ θ ∈ Θk ͳΒ͹ k ¯ θ = ¯ θ ͳͷͰ࠷ӈลͷୈ 2 ߲ͱୈ 3 ߲͸ 0 ʹͳΔɽ
14. 1. AIC ͷಋग़ [10/12] (22) ͷ྆ลͷσʔλ Z ʹ͍ͭͯͷظ଴஋ EZ ΛͱΔͱɼ N||ˆ θ(Z) − ¯ θ||2 C(¯ θ) − N||k ˆ θ(Z) − k ¯ θ||2 C(¯ θ) ͱ N||k ˆ θ(Z) − k ¯ θ||2 C(¯ θ) ͸ ͦΕͧΕࣗ༝౓ L − k, k ͷΧΠೋ৐෼෍ʹ઴ۙతʹै͏ 16 ͜ͱͱɼ N < ˆ θ(Z) − ¯ θ, k ¯ θ − ¯ θ >C(¯ θ) ͸ฏۉ 0ɼඪ४ภࠩ √ N||k ¯ θ − ¯ θ||C(¯ θ) ͷਖ਼ن෼෍ʹ઴ۙతʹै͏ 17 ͜ͱ͔ΒҎԼΛಘΔ 18ɽ EZ ( W(k ˆ θ(Z), ¯ θ) ) ≈ 1 N [ EZ ( NDN (k ˆ θ(Z), ˆ θ(Z)) ) − (L − k) + k ] = 1 N [ EZ ( −2 N ∑ i=1 log ( f(xi|k ˆ θ(Z)) f(xi|ˆ θ(Z)) )) + 2k − L ] (23) 16͜ͷࣄ࣮ͷಋग़͸ޙड़ɽ 17͜ͷࣄ࣮ͷಋग़͸ޙड़ɽ 18ӈลͷ (1/N)(2k − L) ͸ϦεΫؔ਺Λର਺໬౓ൺͰۙࣅͨ͜͠ͱʹΑΔภ Γ (όΠΞε) Ͱ͋Δɽ
15. 1. AIC ͷಋग़ [11/12] ࠷ऴతʹɼ(23) Λ࠷খʹ͢Δ࣍ݩΛબ୒͢Δࢦඪͱͯ͠ AIC ΛҎ ԼͷΑ͏ʹఆΊΔ͜ͱ͕Ͱ͖Δ 19ɽ AIC(k) = −2 N ∑ i=1 log f(xi|k ˆ θ(Z)) + 2k (24) 19Ԡ༻্ L ͸ඇৗʹେ͖͍஋ͱͳΓ໌֬ʹఆٛͰ͖ͳ͍͔΋͠Εͳ͍ɽ͍· ͸ݻఆͨ͠ f ʹؔ͢Δ࣍ݩ k Λબ୒͢Δ໰୊ʹ͍ͭͯߟ͍͑ͯΔͷͰɼL ٴͼ L ʹґଘ͢ΔྔͰ͋Δ log(f(xi |ˆ θ(Z))) ͸શͯͷϞσϧ (࣍ݩ) ʹ͍ͭͯಉ͡஋ ͱͳΓɼ͜ΕΒΛϞσϧબ୒ࢦඪʹؚΊΔඞཁ͸ͳ͍ɽ
16. 1. AIC ͷಋग़ [12/12] ಋग़͔ΒΘ͔Δ͜ͱ ͜Ε·Ͱݟ͖ͯͨ͜ͱ͔ΒɼҎԼ͕Θ͔Δɽ ▶ AIC Λ࠷΋খ͘͢͞Δ k Λબ୒͢Δ͜ͱ͸ɼਅͷ෼෍͔Βൃ ੜ͢ΔͲΜͳσʔλʹରͯ͠΋ (֬཰ີ౓ؔ਺ f Λݻఆͨ͠ ঢ়ଶͰ) ฏۉతʹ࠷΋໬΋Β͍͠ θ Λબ୒͢Δ͜ͱʹ౳ՁͰ ͋Δɽ ▶ N → ∞ ΛԾఆ͍ͯ͠ΔͨΊɼσʔλ਺͕খ͍͞৔߹ʹ͸༗ ޮͰ͸ͳͦ͞͏Ͱ͋Δɽ ▶ ࠷໬ਪఆͷ઴ۙਖ਼نੑΛ༻͍͍ͯΔͷͰɼϑΟ ογϟʔ৘ใ ߦྻ͕ਖ਼ଇͰ͋Δ͜ͱ͕ඞཁͰ͋Γɼͦ͏Ͱͳ͍৔߹ʹ͸༗ ޮͰ͸ͳͦ͞͏Ͱ͋Δɽ
17. 2. ܭࢉաఔ/Fisher ৘ใߦྻͷผදݱ [1/1] (16) ͷ࠷ӈลͷಋग़Λه͢ɽҎޙɼه๏ f(x|θ) = fθ Λ༻͍Δɽ − ∫ ( ∂ log fθ ∂θl∂m ) fθdx = − ∫ ∂ ∂θl ( ∂ log fθ ∂θl ) fθdx = − ∫ ∂ ∂θl ( 1 fθ ∂fθ ∂θl ) fθdx = − ∫ ( − 1 f2 θ ∂fθ ∂θl ∂fθ ∂θm + 1 fθ ∂2f ∂θl∂θm ) fθdx = ∫ ( ∂fθ ∂θl 1 fθ ) ( ∂fθ ∂θm 1 fθ ) fθdx − ∫ ∂2f ∂θl∂θm dx = ∫ ( ∂fθ ∂θl 1 fθ ) ( ∂fθ ∂θm 1 fθ ) fθdx − ∂2 ∂θl∂θm ∫ fθdx = ∫ ( ∂fθ ∂θl 1 fθ ) ( ∂fθ ∂θm 1 fθ ) fθdx − ∂2 ∂θl∂θm 1 = Clm(θ)
18. 2. ܭࢉաఔ/ଛࣦؔ਺ͷۙࣅ [1/6] (17) Ͱ༻͍Δؔ܎ࣜɼ͢ͳΘͪଛࣦؔ਺ W(θ, θ(0)) ͷςΠϥʔల ։ʹΑΔ 2 ࣍ͷ߲·ͰͷۙࣅΛه͢ɽΦ(r) ≡ log r ͱͯ͠ҎԼͷ ৘ใྔΛఆٛ͠ɼͦͷ఺ θ(0) ·ΘΓͷςΠϥʔల։Λߟ͑Δɽ I(θ, θ(0); Φ) ≡ − 1 2 W(θ, θ(0)) = − ∫ f(x|θ(0)) log ( f(x|θ(0)) f(x|θ) ) dx = ∫ f(x|θ(0))Φ ( f(x|θ) f(x|θ(0)) ) dx 0 ࣍ͷ߲͸ҎԼͷΑ͏ʹ؆୯Ͱ͋Δɽ I(θ(0), θ(0); Φ) = Φ(1) ∫ f(x|θ(0))dx = Φ(1) = 0
19. 2. ܭࢉաఔ/ଛࣦؔ਺ͷۙࣅ [2/6] θ ͷ l ൪໨ͷཁૉʹ͍ͭͯภඍ෼͢ΔͱҎԼͱͳΔɽ ∂ ∂θl I(θ, θ(0); Φ) θ=θ(0) = ∫ ∂ ∂θl ( Φ ( f(x|θ) f(x|θ(0)) ) f(x|θ) ) dx θ=θ(0) = ∫ ( d dr Φ(r) ∂r ∂θl ) θ=θ(0) f(x|θ(0))dx = ∫ ( d dr Φ(r) ∂fθ ∂θl 1 fθ ) θ=θ(0) fθ(0) dx = ˙ Φ(r) θ=θ(0) ∫ ( ∂fθ ∂θl ) θ=θ(0) dx = ˙ Φ(1) (∫ ∂fθ ∂θl dx ) θ=¯ θ = ˙ Φ(1) ( ∂ ∂θl ∫ fθdx ) θ=θ(0) = ˙ Φ(1) ( ∂ ∂θl 1 ) θ=θ(0) = 0
20. 2. ܭࢉաఔ/ଛࣦؔ਺ͷۙࣅ [3/6] લεϥΠυΑΓɼ1 ࣍ͷ߲͸ҎԼͱͳΔɽ ( ∂ ∂θ1 I(θ, θ(0); Φ), · · · , ∂ ∂θL I(θ, θ(0); Φ) ) θ=θ(0) (θ − θ(0)) = 0 ࣍ʹɼ(l, m) ੒෼͕ҎԼͰ༩͑ΒΕΔߦྻ H(θ) Λఆٛ͢Δɽ Hlm(θ(0)) ≡ ∂2 ∂θl∂θm I(θ, θ(0); Φ) θ=θ(0) 2 ࣍ͷ߲͸ҎԼͰ͋Δɽ࣍εϥΠυҎ߱Ͱ Hlm(θ(0)) ͷܭࢉաఔ Λه͢ɽ 1 2 (θ − θ(0))′H(θ(0))(θ − θ(0))
21. 2. ܭࢉաఔ/ଛࣦؔ਺ͷۙࣅ [4/6] Hlm(θ(0)) = ∂2 ∂θl∂θm I(θ, θ(0); Φ) θ=θ(0) = ∫ ∂2 ∂θl∂θm ( Φ ( f(x|θ) f(x|θ(0)) ) f(x|θ(0)) ) θ=θ(0) dx = ∫ ∂ ∂θl ( ∂ ∂θm ( Φ ( f(x|θ) f(x|θ(0)) )) f(x|θ(0)) ) θ=θ(0) dx = ∫ ∂ ∂θl ( d dr Φ(r) ∂r ∂θm ) θ=θ(0) f(x|θ(0))dx = ∫ [ d2 dr2 Φ(r) ∂r ∂θl ∂r ∂θm + ( d dr Φ(r) ∂2r ∂θl∂θm )] θ=θ(0) f(x|θ(0))dx = ∫ [ d2 dr2 Φ(r) ∂f(x|θ) ∂θl 1 f(x|θ(0)) ∂f(x|θ) ∂θm 1 f(x|θ(0)) ] θ=θ(0) f(x|θ(0))dx + ∫ ( d dr Φ(r) ∂ ∂θl ( ∂f(x|θ) ∂θm 1 f(x|θ(0)) )) θ=θ(0) f(x|θ(0))dx
22. 2. ܭࢉաఔ/ଛࣦؔ਺ͷۙࣅ [5/6] = d2 dr2 Φ(r) θ=θ(0) ∫ [( ∂fθ ∂θl 1 fθ(0) ) ( ∂fθ ∂θm 1 fθ(0) )] θ=θ(0) fθ(0) dx + d dr Φ(r) θ=θ(0) ∫ ( ∂ ∂θl ( ∂fθ ∂θm )) θ=θ(0) dx = ¨ Φ(1) ∫ [( ∂fθ ∂θl 1 fθ(0) ) ( ∂fθ ∂θm 1 fθ(0) )] θ=θ(0) fθ(0) dx + ˙ Φ(1) ∫ ( ∂2fθ ∂θl∂θm ) θ=θ(0) dx = ¨ Φ(1)Clm(θ(0)) + ˙ Φ(1) ( ∂2 ∂θl∂θm ∫ fθdx ) θ=θ(0) = ¨ Φ(1)Clm(θ(0)) + ˙ Φ(1) ( ∂2 ∂θl∂θm 1 ) θ=θ(0) = ¨ Φ(1)Clm(θ(0))
23. 2. ܭࢉաఔ/ଛࣦؔ਺ͷۙࣅ [6/6] Φ(r) = log r, ˙ Φ(r) = 1/r, ¨ Φ(r) = −1/r2 ΑΓ ¨ Φ(1) = −1 Ͱ͋Δ ͷͰҎԼΛಘΔɽ Hlm(θ(0)) = −Clm(θ(0)) H(θ(0)) = −C(θ(0)) ैͬͯ 2 ࣍ͷ߲͸ҎԼͱͳΔɽ 1 2 (θ − θ(0))′H(θ)(θ − θ(0)) = − 1 2 (θ − θ(0))′C(θ)(θ − θ(0)) 0 ࣍ɼ1 ࣍ͷ߲͕ 0 Ͱ͋Δ͔Β 3 ࣍Ҏ߱ͷ߲Λແࢹ͢Δͱ࠷ऴతʹ ҎԼΛಘΔ 20ɽ I(θ, θ(0); Φ) = ∫ f(x|θ(0))Φ ( f(x|θ) f(x|θ(0)) ) dx ≈ − 1 2 (θ − θ(0))′C(θ)(θ − θ(0)) 20KL μΠόʔδΣϯεΛςΠϥʔల։͢Δͱ Fisher ৘ใߦྻ͕ग़ݱ͢Δ͜ͱ ͕Θ͔Δɽ
24. 2. ܭࢉաఔ/ର਺໬౓ൺͷۙࣅ [1/2] (20) Ͱ࢖͏ؔ܎ࣜɼ͢ͳΘͪର਺໬౓ൺͷۙࣅʹ͍ͭͯه͢ɽҎ Լͷ࠷໬ਪఆྔ ˆ θ ͷ·ΘΓ 21 ͷςΠϥʔల։Λߟ͑Δ 22ɽ N ∑ i=1 log f(xi|θ) 0 ࣍ͷ߲͸ ∑ N i=1 log f(xi|ˆ θ) Ͱ͋Δɽ࣍ʹ 1 ࣍ͷ߲Ͱ͋Δ͕ɼθ ͷ l ൪໨ͷཁૉʹ͍ͭͯͷภඍ෼͸࠷໬ਪఆྔ ˆ θ ʹ͍ͭͯߟ͍͑ͯ Δ͔Β 0 Ͱ͋Γɼ1 ࣍ͷ߲͸ 0 ͱͳΔɽ N ∑ i=1 ∂ log f(xi|θ) ∂θl θ=ˆ θ = 0 ( N ∑ i=1 ∂ log f(xi|θ) ∂θ1 , · · · , N ∑ i=1 ∂ log f(xi|θ) ∂θL ) θ=ˆ θ (θ − ˆ θ) = 0 21͜͜Ͱ͸ ˆ θ ∈ ΘL ͱ͢Δ͕ɼˆ θ ∈ ΘK ͷ৔߹΋ಉ༷ʹߟ͑ͯΑ͍ɽ 22ඞཁͳͷͰ͜ͷԾఆΛ͓͘ɽ
25. 2. ܭࢉաఔ/ର਺໬౓ൺͷۙࣅ [2/2] ࣍͸ 2 ࣍ͷ߲Ͱ͋Δɽ(l, m) ੒෼͕ҎԼͰ༩͑ΒΕΔߦྻ G(θ) Λ ఆٛ͢Δͱɼ(16) ͱେ਺ͷ๏ଇΑΓ N → ∞ Ͱ Fisher ৘ใߦྻ (ͷ −1 ഒ) ʹ֬཰ऩଋ͢Δɽ Glm(ˆ θ) ≡ 1 N N ∑ i=1 ∂2 log f(xi|θ) ∂θl∂θm θ=ˆ θ p −→ −Clm(ˆ θ) G(ˆ θ) p −→ −C(ˆ θ) (25) 3 ࣍Ҏ߱ͷ߲Λແࢹ͢ΔͱςΠϥʔల։ͷ݁Ռ͸ҎԼͱͳΔɽ N ∑ i=1 log f(xi|θ) ≈ N ∑ i=1 log f(xi|ˆ θ) + 1 2 √ N(θ − ˆ θ)′G(ˆ θ) √ N(θ − ˆ θ) ର਺໬౓ൺͷۙࣅ͸ҎԼͰ͋Δ 23ɽ N ∑ i=1 log f(xi|θ) log f(xi|ˆ θ) ≈ 1 2 √ N(θ − ˆ θ)′G(ˆ θ) √ N(θ − ˆ θ) 23࠷໬ਪఆྔΛ༻͍Δͱ͍͏ԾఆͷԼͰͷ݁ՌͰ͋Δɽ
26. 2. ܭࢉաఔ/ภΓ (όΠΞε) ͷ઴ۙ෼෍ [1/7] (23) Ͱ࢖͏ؔ܎ʹ͍ͭͯه͢ɽҎԼͰ͸ 1 ൪໨͔Β k ൪໨ͷཁૉ ͕౳͍͠ (ۙࣅͰ͖Δ) ͜ͱΛҙຯ͢ΔϕΫτϧʹର͢Δԋࢉࢠ =k (≈k) Λ༻͍Δɽ ͸͡Ίʹ N||ˆ θ(Z) − ¯ θ||2 C(¯ θ) − N||k ˆ θ(Z) − k ¯ θ||2 C(¯ θ) ͕ࣗ༝౓ L − k ͷΧΠೋ৐෼෍ʹ઴ۙతʹै͏͜ͱΛࣔ͢ɽҎԼͷؔ਺ 24 ͷ k ˆ θ ٴͼ ˆ θ ͷ·ΘΓͷςΠϥʔల։Λ (2 ͭಉ࣌ʹ) ߟ͑Δɽ k ¯ θ ∈ ΘK Ͱ͋ΔͷͰ 1 ൪໨͔Β k ൪໨ͷཁૉʹ͍ͭͯͷΈߟ͑Ε͹Α͍ɽ 1 √ N N ∑ i=1 ∂ log f(xi|k ¯ θ) ∂ k ¯ θ 24͜ͷؔ਺͸ L ࣍ݩྻϕΫτϧͰ͋Δɽ͜͜Ͱ͸ k ¯ θ Λม਺ͱͯ͠ߟ͍͑ͯΔɽ
27. 2. ܭࢉաఔ/ภΓ (όΠΞε) ͷ઴ۙ෼෍ [2/7] 0 ࣍ͷ߲͸ k ˆ θ, ˆ θ ͕࠷໬ਪఆྔͰ͋Δ͜ͱ (8)(9) ͔ΒͲͪΒ΋θϩ ϕΫτϧͰ͋Δɽ 1 √ N N ∑ i=1 ∂ log f(xi|k ¯ θ) ∂ k ¯ θ k ¯ θ= k ˆ θ =k 1 √ N N ∑ i=1 ∂ log f(xi|k ¯ θ) ∂ k ¯ θ k ¯ θ=ˆ θ =k 0 1 ࣍ͷ߲͸ͦΕͧΕҎԼͰ͋Δɽ √ N 1 N N ∑ i=1 ∂2 log f(xi|k ¯ θ) ∂ k ¯ θ ∂ k ¯ θ′ k ¯ θ= k ˆ θ (k ¯ θ − k ˆ θ) √ N 1 N N ∑ i=1 ∂2 log f(xi|k ¯ θ) ∂ k ¯ θ ∂ k ¯ θ′ k ¯ θ=ˆ θ (k ¯ θ − ˆ θ)
28. 2. ܭࢉաఔ/ภΓ (όΠΞε) ͷ઴ۙ෼෍ [3/7] ςΠϥʔల։ͷ݁Ռ͸ 2 ࣍Ҏ߱ͷ߲Λແࢹ͢ΔͱҎԼͱͳΔɽ 1 √ N N ∑ i=1 ∂ log f(xi|k ¯ θ) ∂ k ¯ θ ≈k √ N 1 N N ∑ i=1 ∂2 log f(xi|k ¯ θ) ∂ k ¯ θ ∂ k ¯ θ′ k ¯ θ=k ˆ θ (k ¯ θ − k ˆ θ) ≈k √ N 1 N N ∑ i=1 ∂2 log f(xi|k ¯ θ) ∂ k ¯ θ ∂ k ¯ θ′ k ¯ θ=ˆ θ (k ¯ θ − ˆ θ)
29. 2. ܭࢉաఔ/ภΓ (όΠΞε) ͷ઴ۙ෼෍ [4/7] (21) ͰߦͬͨΑ͏ʹɼ͜͜Ͱ N → ∞ ͱ͢Δʹ͋ͨΓɼ √ N(k ¯ θ − ¯ θ) ͕༗քͰ͋ΔΑ͏ʹ N ͱ k ΛͱΕΔͱ͢Δ 25ɽ͜ͷ ৚݅ͷԼͰɼ࠷໬ਪఆྔ k ˆ θ(Z), ˆ θ(Z) ͷ઴ۙਖ਼نੑͱ k ˆ θ(Z), ˆ θ(Z) ͸ͦΕͧΕ k ¯ θ, ¯ θ ͷҰகਪఆྔͰ͋Δ͜ͱɼ͞Βʹ (25) ͔ΒҎԼ ΛಘΔɽ 1 √ N N ∑ i=1 ∂ log f(xi|k ¯ θ) ∂ k ¯ θ ≈k C(¯ θ) √ N(k ˆ θ − k ¯ θ) ≈k C(¯ θ) √ N(ˆ θ − k ¯ θ) (26) 25͜ͷ৚݅ΑΓɼN → ∞ ͷͱ͖͋Δ࣮਺ M ʹ͍ͭͯ √ N(k ¯ θ − ¯ θ) = M Ͱ ͋ΔͷͰ k ¯ θ = ¯ θ + 1 √ M ≈ ¯ θ Ͱ͋Δɽ
30. 2. ܭࢉաఔ/ภΓ (όΠΞε) ͷ઴ۙ෼෍ [5/7] (26) ʹ (27) Λ༻͍ͯҎԼͱͳΓɼ k ˆ θ − k ¯ θ ͸઴ۙతʹ ˆ θ − ¯ θ ͷࣹ ӨͰ͋Δ͜ͱ͕Θ͔Δ 26ɽ C(¯ θ) √ N(k ˆ θ − k ¯ θ) ≈k C(¯ θ) √ N(ˆ θ − ¯ θ) ࣍εϥΠυΑΓ N||k ˆ θ(Z) − k ¯ θ||2 C(¯ θ) ͸ࣗ༝౓ k ͷΧΠೋ৐෼෍ ʹ઴ۙతʹै͏͜ͱ͕Θ͔Γɼಉ༷ͷٞ࿦ʹΑΓ N||ˆ θ(Z) − ¯ θ||2 C(¯ θ) ͸ࣗ༝౓ L ͷΧΠೋ৐෼෍ʹ઴ۙతʹै͏ɽ Αͬͯ k ˆ θ − k ¯ θ ͸઴ۙతʹ ˆ θ − ¯ θ ͷࣹӨͰ͋Δ͜ͱ͔Β N||ˆ θ(Z) − ¯ θ||2 C(¯ θ) − N||k ˆ θ(Z) − k ¯ θ||2 C(¯ θ) ͸ࣗ༝౓ L − k ͷΧΠೋ ৐෼෍ʹ઴ۙతʹै͏ 27ɽ 26ཁૉͰॻ͘ͱҎԼͱͳΔɽAkaike(1973) ʹ͋Θͤͯූ߸Λ൓స͍ͯ͠Δɽ C(l, 1) √ N(k ¯ θ1 − k ˆ θ1 ) + · · · + C(l, k) √ N(k ¯ θk − k ˆ θk ) ≈ C(l, 1) √ N(¯ θ1 − ˆ θ1 ) + · · · + C(l, k) √ N(¯ θk − ˆ θk ) + · · · +C(l, L) √ N(¯ θL − ˆ θL ), l = 1, 2, · · · , k 27͍·͍ͪᐆດͰ͋Δ͕͜ͷ͘Β͍ʹ͓ͯ͘͠..
31. 2. ܭࢉաఔ/ภΓ (όΠΞε) ͷ઴ۙ෼෍ [6/7] ࣍ʹɼN||k ˆ θ(Z) − k ¯ θ||2 C(¯ θ) ͕ࣗ༝౓ k ͷΧΠೋ৐෼෍ʹ઴ۙతʹ ै͏͜ͱ͸ N||k ˆ θ(Z) − k ¯ θ||2 C(¯ θ) = √ N(k ˆ θ − k ¯ θ)′C(¯ θ) √ N(k ˆ θ − k ¯ θ) ͱɼ࠷໬ਪఆྔͷ઴ۙਖ਼نੑΑΓ N → ∞ Ͱ √ N(ˆ θ − ¯ θ) d −→ N(0, C(¯ θ)−1) Ͱ͋Δ͜ͱɼ͞Βʹ k ˆ θ − k ¯ θ ͸઴ۙ తʹ ˆ θ − ¯ θ ͷࣹӨͰ͋Δ͜ͱ͔ΒΘ͔Δ 28ɽ 28N(·, ·) ͸ਖ਼ن෼෍Ͱ͋ΔɽL ࣍ݩͷ֬཰ม਺ X ʹ͍ͭͯɼ෼ࢄڞ෼ࢄߦྻ Σ ͱͯ͠ɼX ∼ N(0, Σ) ⇒ X′Σ−1X ∼ χ2 L Ͱ͋Δ͜ͱΛ༻͍Δɽχ2 L ͸ࣗ༝౓ L ͷΧΠೋ৐෼෍Ͱ͋Δɽ
32. 2. ܭࢉաఔ/ภΓ (όΠΞε) ͷ઴ۙ෼෍ [7/7] ࣍ʹɼ N < ˆ θ(Z) − ¯ θ, k ¯ θ − ¯ θ >C(¯ θ) ͕ฏۉ 0ɼඪ४ภࠩ √ N||k ¯ θ − ¯ θ||C(¯ θ) ͷਖ਼ن෼෍ʹ઴ۙతʹै͏͜ͱΛه͢ɽ࠷໬ਪఆ ྔͷ઴ۙਖ਼نੑΑΓ N → ∞ Ͱ √ N(ˆ θ(Z) − ¯ θ) d −→ N(0, C(¯ θ)−1) Ͱ͋Δ͜ͱ͔ΒҎԼͷ݁ՌΛಘΔɽ N < ˆ θ(Z) − ¯ θ, k ¯ θ − ¯ θ >2 C(¯ θ) = √ N(k ¯ θ − ¯ θ)′C(¯ θ) √ N(ˆ θ(Z) − ¯ θ) d −→ √ N(k ¯ θ − ¯ θ)′C(¯ θ)N(0, C(¯ θ)−1) = √ NN(0, ((k ¯ θ − ¯ θ)′C(¯ θ))C(¯ θ)−1((k ¯ θ − ¯ θ)′C(¯ θ))′) = √ NN(0, ((k ¯ θ − ¯ θ)′C(¯ θ)(k ¯ θ − ¯ θ)) = √ NN(0, ||k ¯ θ − ¯ θ||2 C(¯ θ) ) = N(0, N||k ¯ θ − ¯ θ||2 C(¯ θ) )
33. 2. ܭࢉաఔ/ਅͷύϥϝʔλ ¯ θ ͱࣹӨ k ¯ θ ͷؔ܎ࣜ [1/1] (26) ʹ࢖͏ؔ܎ࣜʹ͍ͭͯه͢ɽ k ¯ θ ͷఆٛ (18) ΑΓɼ1 ൪໨͔Β k ൪໨ͷཁૉ͕ 0 Ͱ͋Δ x ∈ ΘL ʹΑΓҎԼͷؔ܎͕ಘΒΕΔɽ ¯ θ = k ¯ θ + x ྆ลʹ C Λ৐ͯ͡ҎԼΛಘΔ 29ɽ C(¯ θ) k ¯ θ = C(¯ θ)¯ θ + C(¯ θ)x =k C(¯ θ)¯ θ (27) 29ཁૉͰॻ͘ͱҎԼͱͳΔɽ C(l, 1)k ¯ θ1 + · · · + C(l, k)k ¯ θk = C(l, 1)¯ θ1 + · · · + C(l, k)¯ θk + · · · + C(l, L)¯ θL , l = 1, 2, · · · , k
34. ऴΘΓͰ͢ɽ