Slide 1
Slide 1 text
ୈ 5 ճ PRML ηϛφʔ
ࡔޱ ྒี
1 / 51
Slide 2
Slide 2 text
0. ࠓճͷηϛφʔʹ͍ͭͯ
ࠓճͷηϛφʔͰɺPRML ͷୈ 10 ষͷۙࣅਪ๏Λ͓͍ͨ͠͠ͱ
ࢥ͍·͢ɻ
·ͨɺ͜ΕΒͷΛઆ໌͢ΔͨΊʹඞཁͳ༧උࣝΛղઆ͠·͢ɻ
(PRML 9.4)
ͳ͓ҙͱͯ͠ɺຊεϥΠυͷࣜ൪߸ͱ PRML ͷࣜ൪߸ҟͳΓ·
͢ͷͰɺ͝ҙ͍ͩ͘͞ɻ
2 / 51
Slide 3
Slide 3 text
࣍
1. ༧උࣝ (PRML 9.4)
2. ۙࣅਪ๏
2-1. มਪ (PRML 10.1)
2-2. ྫɿมࠞ߹Ψε (PRML 10.2)
2-3. ہॴతมਪ๏ (PRML 10.5)
2-4. มϩδεςΟοΫճؼ (PRML 10.6)
3 / 51
Slide 4
Slide 4 text
1. ༧උࣝ
EM ΞϧΰϦζϜ؍ଌରͰ͋Δ؍ଌม X = (x1
, · · · , xN
)T ͱ؍
ଌରͰͳ͍જࡏม Z = (z1
, · · · , zN
)T ΛؚΉܥ (۩ମྫͱͯࠞ͠߹
Ψε͕͋Δ) ͷ࠷ਪఆΛߦ͏ࡍͷख๏ͷҰͭͰ͋ΓɺE εςο
ϓͱ M εςοϓͱ͍͏ 2 ͭͷεςοϓΛ෮తʹ࣮ߦ͢Δ͜ͱͰ
ؔΛ࠷େԽ͢Δɻ
͜͜ͰɺҰൠతͳ EM ΞϧΰϦζϜʹ͍ͭͯ؆୯ʹઆ໌͢Δɻ
(PRML 9.4 ʹରԠ)
4 / 51
Slide 5
Slide 5 text
1. ༧උࣝ
·ͣɺਪఆ͍ͨ͠ύϥϝʔλϕΫτϧΛ θ ͱ͠ɺN ݸͷ؍ଌม
X = (x1
, · · · , xN
)T ʹରԠ͢Δજࡏม Z = (z1
, · · · , zN
)T Λ֬ม
ͱ͢Δҙͷ֬ؔ q(Z) Λಋೖ͢Δͱɺ؍ଌσʔλ X ͷର
ؔ (ෆશσʔλͷରؔ)ln p(X|θ) ҎԼͷΑ͏ʹͳΔɻ
ln p(X|θ)
=
{ ∑
Z
q(Z)
}
=1
ln p(X|θ)
=
∑
Z
q(Z) ln
{
p(X, Z|θ)
p(Z|X, θ)
}
=
∑
Z
q(Z) ln
{
p(X, Z|θ)
q(Z)
·
q(Z)
p(Z|X, θ)
}
=
∑
Z
q(Z) ln
{
p(X, Z|θ)
q(Z)
}
−
∑
Z
q(Z) ln
{
p(Z|X, θ)
q(Z)
}
(1.1)
5 / 51
Slide 6
Slide 6 text
1. ༧උࣝ
͜͜Ͱɺ๏ఆཧ
p(X, Z|θ) = p(Z|X, θ)p(X|θ) (1.2)
Λ༻͍ͨɻ
·ͨɺ
L(q, θ) =
∑
Z
q(Z) ln
{
p(X, Z|θ)
q(Z)
}
(1.3)
KL(q∥p) = −
∑
Z
q(Z) ln
{
p(Z|X, θ)
q(Z)
}
(1.4)
ͱ͢Εɺ(1.1)
ln p(X|θ) = L(q, θ) + KL(q∥p) (1.5)
ͱͳΔɻ(KL(q∥p) KL μΠόʔδΣϯε)
6 / 51
Slide 7
Slide 7 text
1. ༧උࣝ
͜͜ͰɺPRML ͷ 1.6.1 ΑΓɺKL μΠόʔδΣϯε KL(q∥p) ඇෛͰ
͋Δɻ(KL(q∥p) ≥ 0 Ͱ͋Γɺ߸ q(Z) = p(Z|X, θ) ͷͱ͖ͷΈཱ
͢Δɻ)
Ҏ্ΑΓɺln p(X|θ) ≥ L(q, θ) ͱͳΓɺL(q, θ) ln p(X|θ) ͷԼݶͰ͋
Δɻ(Լਤ)
7 / 51
Slide 8
Slide 8 text
1. ༧උࣝ
ղ (1.5) Λར༻ͯ͠ EM ΞϧΰϦζϜΛఆٛ͠ɺͦΕ͕ର
ln p(X|θ) Λ࠷େԽ͢Δ͜ͱΛࣔ͢ɻ
ͦͷͨΊʹ·ͣɺύϥϝʔλͷΛ θold ʹॳظԽ͢Δɻ
E εςοϓͰɺL(q, θold) Λ θold Λݻఆ͠ͳ͕Β q(Z) ʹ͍ͭͯ࠷େ
Խ͢Δɻ(͜Ε͕ͷͪʹग़ͯ͘Δม๏ʹରԠ)
͜ͷ࠷େԽ؆୯Ͱɺର ln p(X|θold) q(Z) ʹґଘ͠ͳ͍ͷͰɺ
q(Z) ΛมԽͤͯ͞ ln p(X|θold) มԽ͠ͳ͍ɻ
͜ΕΑΓɺKL(q∥p) = 0 ͭ·Γ q(Z) = p(Z|X, θold) ͷͱ͖ɺL(q, θold)
࠷େʹͳΔɻ
͜ͷͱ͖ɺԼਤͷΑ͏ʹ ln p(X|θold) = L(q, θold) ͱͳΔɻ
8 / 51
Slide 9
Slide 9 text
1. ༧උࣝ
࣍ͷ M εςοϓͰɺq(Z) = p(Z|X, θold) Ͱݻఆ͠ɺL(q, θ) Λ θ ʹͭ
͍ͯ࠷େԽ͢Δɻ
L(q, θ) Λ࠷େʹ͢ΔύϥϝʔλΛ θnew ͱ͢Δͱɺ͢Ͱʹ L(q, θ) ͕࠷
େʹୡ͍ͯ͠ͳ͍ݶΓ (θnew ̸= θold ͷͱ͖)ɺL(q, θ) ૿Ճ͠ɺ͞Β
ʹ KL μΠόʔδΣϯεҎԼͷΑ͏ʹਖ਼ʹͳΔɻ
KL(q∥p) = −
∑
Z
p(Z|X, θold) ln
{
p(Z|X, θnew)
p(Z|X, θold)
}
> 0 (1.6)
9 / 51
Slide 10
Slide 10 text
1. ༧උࣝ
Αͬͯɺ͜ΕΒͷ E εςοϓͱ M εςοϓҎԼͷਤͷΑ͏ʹର
ln p(X|θ) Λ૿Ճͤ͞Δɻ
10 / 51
Slide 11
Slide 11 text
1. ༧උࣝ
·ͨɺq(Z) = p(Z|X, θold) ͷͱ͖ͷ L(q, θ)
L(q, θ) =
∑
Z
p(Z|X, θold) ln
{
p(X, Z|θ)
p(Z|X, θold)
}
=
∑
Z
p(Z|X, θold) ln p(X, Z|θ)
−
∑
Z
p(Z|X, θold) ln p(Z|X, θold)
=Q(θ, θold) + const.
(1.7)
͜͜ͰɺQ(θ, θold) ҎԼͷΑ͏ʹఆٛͨ͠ɻ
Q(θ, θold) =
∑
Z
p(Z|X, θold) ln p(X, Z|θ) (1.8)
͜͜Ͱͷ const. ͱɺθ ʹґଘ͠ͳ͍߲Ͱ͋Δɻ
͜ΕΑΓɺM εςοϓͷ L(q, θ) ͷ࠷େԽ Q(θ, θold) ͷ࠷େԽͱՁ
Ͱ͋Δɻ
11 / 51
Slide 12
Slide 12 text
2. ۙࣅਪ๏
EM ΞϧΰϦζϜͰɺ؍ଌมΛ X ͱ͠જࡏมΛ Z ͱ͠ύϥϝʔ
λΛ θ ͢ΔͱɺE εςοϓͱͯ͠ࣄޙ p(Z|X, θ) ΛٻΊɺM εςο
ϓͱͯ͠શର ln p(X, Z|θ) ͷࣄޙ ln p(X, Z|θold) ʹΑΔظ
Q(θ, θold) Λ࠷େʹ͢Δύϥϝʔλ θ ΛٻΊͨɻ
ࠓޙͷٞʹΑΓɺEM ΞϧΰϦζϜͷߟ͑ํΛϕΠζਪఆʹԠ༻Ͱ
͖ɺ͜ͷ࣌ࣄޙ͕ٻΊΔ͜ͱ͕Ͱ͖ͳ͍͕࣌͋Δɻ
ͦͷΑ͏ͳ߹ʹۙࣅతʹظΛٻΊΔํ๏ͱͯ͠ɺ͜͜Ͱมਪ
๏Λಋೖ͢Δɻ(2-1)
ͦͷมਪ๏Λࠞ߹ΨεϞσϧʹదԠ͢Δɻ(2-2)
·ͨɺͷղͱผͷਪ๏ͱͯ͠ɺہॴతมਪ๏Λಋೖ͠
(2-3)ɺͦͷਪ๏ΛϩδεςΟοΫճؼʹద༻͢Δ (2-4)ɻ
12 / 51
Slide 13
Slide 13 text
2-1. มਪ
มਪ๏ม๏ͱ͍͏ֶతख๏Λىݯʹ࣋ͭɻ
ม๏ͰɺҎԼͷΑ͏ͳ൚ؔ (ؔͷܗ͕ܾ·Ε͕ܾ·Δྔ)
ʹ͓͍ͯɺՄೳͳͯ͢ͷؔͷதͰ൚ؔΛ࠷େ͘͠࠷খʹ͢Δ
Α͏ͳؔΛ୳͢ํ๏Ͱ͋Δɻ( D ࢀর)
H[p] = −
∫
p(x) ln p(x) dx (2.1)
(2.1) ΤϯτϩϐʔͰ͋Γɺ͜Εʹม๏Λద༻͢Δͱɺ
p(x) = const. ͕ H[p] Λ࠷େʹ͢ΔͰ͋Δࣄ͕Θ͔Γɺ͔ͨ͠ʹΤ
ϯτϩϐʔͷײͱҰக͢Δɻ
ม๏ࣗମɺ൚ؔΛ࠷େ͘͠࠷খʹ͢ΔΑ͏ͳؔΛՄೳͳ͢
ͯͷؔͷத͔Β୳͢ख๏ͳͷͰɺۙࣅతͳख๏Ͱͳ͍ɻ
มਪ๏Ͱɺ࠷దԽΛߦ͏ؔͷΫϥεΛ੍ݶ͠ɺۙࣅతʹ൚ؔ
Λ࠷େ͘͠࠷খʹ͢ΔΑ͏ͳؔΛٻΊΔɻ
13 / 51
Slide 14
Slide 14 text
2-1. มਪ
ͦΕͰɺมਪ๏ͷԠ༻ํ๏Λݟ͍ͯ͘ɻ
·ͣɺͯ͢ͷύϥϝʔλʹࣄલ͕༩͑ΒΕ͍ͯΔશͳϕΠζϞ
σϧ͕͋Δͱ͢Δɻ
Ϟσϧʹύϥϝʔλͷଞʹજࡏม͕͋ΔՄೳੑ͕͋Γɺύϥϝʔλ
ͱજࡏมΛ·ͱΊͯ Z ͱ͔͖ɺ͞Βʹ؍ଌมͷू߹Λ X ͱ͢Δɻ
·ͨɺ؍ଌมͱજࡏมͷಉ࣌ p(X, Z) ༩͑ΒΕ͍ͯΔͱ
͢Δɻ
͔͠͠ɺࣄޙ
p(Z|X) =
p(X, Z)
∫
p(X, Z) dZ
(2.2)
જࡏมͷ࣍ݩ͕ߴ͗͢ΔͳͲͷཧ༝ͰੵෆՄೳͳͨΊɺٻΊΒΕ
ͳ͍ͱ͢Δɻ
14 / 51
Slide 15
Slide 15 text
2-1. มਪ
͜͜Ͱɺपล p(X)(ϞσϧΤϏσϯε) ͷରҎԼͷΑ͏ʹͳΔ
͜ͱΛ༻͍Δɻ
ln p(X) = L[q] + KL(q∥p) (2.3)
͜͜Ͱɺ
L[q] =
∫
q(Z) ln
{
p(X, Z)
q(Z)
}
dZ (2.4)
KL(q∥p) = −
∫
q(Z) ln
{
p(Z|X)
q(Z)
}
dZ (2.5)
Ͱ͋Δɻ
KL μΠόʔδΣϯεͷੑ࣭ΑΓɺ࠷খʹͳΔͷ q(Z) ͕ࣄޙ
p(Z|X) ʹҰக͢Δͱ͖Ͱ͋Δɻ
ͦΕ L[q] ͕࠷େʹͳΔͱ͖Ͱ͋ΓɺͦͷΑ͏ͳ q(Z) ࣄޙ
Ͱ͋Δɻ
15 / 51
Slide 16
Slide 16 text
2-1. มਪ
ͨͩ͠ԾఆʹΑΓɺࣄޙ p(Z|X) ܭࢉͰ͖ͳ͍ͷͰɺΘΓʹ
q(Z) ͷऔΓ͏Δؔʹ੍ݶΛ༩͑ɺͦͷ੍ݶͷதͰ L[q] Λ࠷େʹ͢Δ
Α͏ͳ q(Z) ΛٻΊΔࣄΛߟ͑Δɻ(มਪ๏)
·੍ͣݶͷྫͱͯ͠ɺͷղΛߟ͑Δɻ
16 / 51
Slide 17
Slide 17 text
2-1-1. ͷղ
͜͜Ͱɺq(Z) ͕ҎԼͷΑ͏ͳܗΛऔ͍ͬͯΔͷʹ੍ݶ͢Δɻ
q(Z) =
M
∏
i=1
qi
(Zi
) (2.6)
͜͜ͰɺZ = (z1
, z2
, · · · , zN
)T Λഉͳ (ॏෳ͠ͳ͍) άϧʔϓ Zi
ʹΘ
͚ͨɻ
͜͜Ͱɺqi
(Zi
) ͷܗʹ੍ݶͳ͍ɻ
(2.6) Λ (2.4) ͷӈลʹೖͯ͠ɺL[q] Λ֤ qi
(Zi
) Ͱॱ൪ʹ࠷దԽ͢Δ
͜ͱΛߟ͑Δɻ
ͦͷͨΊɺL[q] ͷ qj
(Zj
) ʹґଘ͢Δ߲ͷΈΛऔΓग़͢ɻ
17 / 51
Slide 18
Slide 18 text
2-1-1. ͷղ
L[q] ͷ qj
(Zj
) ʹґଘ͢Δ߲ҎԼͷΑ͏ʹͳΔɻ
L[q] =
∫
q(Z) ln
{
p(X, Z)
q(Z)
}
dZ
=
∫ (
ln p(X, Z) −
M
∑
l=1
ln ql(Zl)
)
M
∏
i=1
qi(Zi) dZi
=
∫
ln p(X, Z)
M
∏
i=1
qi(Zi) dZi −
M
∑
l=1
∫
ln ql(Zl)
M
∏
i=1
qi(Zi) dZi
=
∫
qj (Zj )
{ ∫
ln p(X, Z)
∏
i̸=j
qi(Zi) dZi
}
dZj −
M
∑
l=1
∫
ql(Zl) ln ql(Zl) dZl
=
∫
qj (Zj )
{ ∫
ln p(X, Z)
∏
i̸=j
qi(Zi) dZi
}
dZj −
∫
qj (Zj ) ln qj (Zj ) dZj + const.
=
∫
qj (Zj ) ln p(X, Zj ) dZj −
∫
qj (Zj ) ln qj (Zj ) dZj + const.
(2.7)
18 / 51
Slide 19
Slide 19 text
2-1-1. ͷղ
͜͜Ͱɺ
ln p(X, Zj
) = Ei̸=j
[ln p(X, Z)] + const. (2.8)
Ͱ͋Γɺ
Ei̸=j
[ln p(X, Z)] =
∫
ln p(X, Z)
∏
i̸=j
qi
(Zi
) dZi
(2.9)
ͱఆٛͨ͠ɻ
19 / 51
Slide 20
Slide 20 text
2-1-1. ͷղ
࣍ʹɺ(2.7) ͷ L[q] Λ qj
(Zj
) ʹ͍ͭͯ࠷େԽ͢Δ͜ͱΛߟ͑Δɻ
͜Ε؆୯Ͱɺ(2.7) KL μΠόʔδΣϯεΛ༻͍ͯ
L[q] =
∫
qj
(Zj
) ln
(
p(X, Zj
)
qj
(Zj
)
)
dZj
+ const.
= − KL(qj
∥p) + const.
(2.10)
ͱॻ͚ΔͷͰɺKL μΠόʔδΣϯεͷੑ࣭Λ༻͍ΔͱɺL[q] Λ࠷େʹ
͢Δ qj
(Zj
)
ln q⋆
j
(Zj
) = ln p(X, Zj
) = Ei̸=j
[ln p(X, Z)] + const. (2.11)
ͱͳΔɻ
͜͜Ͱɺ(2.11) ͷ࠶ӈล qi
(Zi
) (i ̸= j) Ͱͷظʹґଘ͍ͯ͠Δͷ
Ͱɺղ q⋆(Z) ΛٻΊΔʹɺͯ͢ͷҼࢠ qj
(Zj
) ΛదʹॳظԽ͠ɺ
(2.11) Λ༻͍ͯ͋ΔҼࢠ qj
(Zj
) ΛଞͷҼࢠͷݱࡏͷͰߋ৽͍ͯ͘͠
͜ͱͰٻΊΔɻ(มϕΠζ๏)
20 / 51
Slide 21
Slide 21 text
2-2. ྫɿมࠞ߹Ψε
Ҏ্Ͱհͨ͠มϕΠζ๏Λࠞ߹ΨεϞσϧʹద༻ͯ͠ΈΔɻ
ࠞ߹ΨεϞσϧ༗άϥϑΛ༻͍ΔͱҎԼͷΑ͏ʹදͤΔɻ
͜͜Ͱɺθ = {π, µ, Λ} ύϥϝʔλͰ͋ΓɺX = (x1
, · · · , xN
)T ؍
ଌมͰ Z = (z1
, · · · , zN
)T ͦΕʹରԠ͢ΔજࡏมͰ͋Δɻ
21 / 51
Slide 22
Slide 22 text
2-2. ྫɿมࠞ߹Ψε
͜ͷάϥϑΑΓɺ֤֬มͷ͖݅ಠཱੑΛՃຯ͢Δͱɺಉ࣌
p(X, Z, π, µ, Λ) ҎԼͷΑ͏ʹղͰ͖Δɻ(PRML 8 ষΛࢀর)
p(X, Z, π, µ, Λ) = p(X|Z, µ, Λ)p(Z|π)p(π)p(µ|Λ)p(Λ) (2.12)
͜ΕΒͷӈลͷΛಛఆͷʹԾఆ͍ͯ͘͠ɻ
·ͣɺ p(Z|π) ΛҎԼͷΑ͏ʹԾఆ͢Δɻ
p(Z|π) =
N
∏
n=1
K
∏
k=1
πznk
k
(2.13)
͜͜Ͱɺࣄલ p(π) ҎԼͷΑ͏ʹσΟϦΫϨΛ༻͍Δͱɺ͜
ͷڞࣄલͱͳ͓ͬͯΓɺࣄޙσΟϦΫϨͱ
ͳΔɻ
p(π) = Dir(π|α0
) = C(α0
)
K
∏
k=1
πα0+1
k
(2.14)
͜͜ͰɺC(α0
) σΟϦΫϨͷਖ਼نԽఆ ((B.23) Ͱఆٛ͞Εͯ
͍Δ)
22 / 51
Slide 23
Slide 23 text
2-2. ྫɿมࠞ߹Ψε
࣍ʹɺ p(X|Z, µ, Λ) ΛҎԼͷΑ͏ʹΨεͰԾఆ͢Δɻ
p(X|Z, µ, Λ) =
N
∏
n=1
K
∏
k=1
N(xn
|µk
, Λ−1
k
)znk (2.15)
͜͜ͰɺΨε N(x|µ, Σ) ҎԼͰఆٛ͞ΕΔɻ
N(x|µ, Σ) =
1
(2π)D/2
1
|Σ|1/2
exp
{
−
1
2
(x − µ)TΣ−1(x − µ)
}
(2.16)
͞Βʹ (2.15) Λड͚ͯɺ p(µ, Λ) = p(µ|Λ)p(Λ) ڞࣄલͰ
͋ΔΨε-Ογϟʔτࣄલ (PRML 2.3.6 ͷ (2.157) ࢀর) ͱԾఆ
͢Δɻ
p(µ, Λ) =
K
∏
k=1
N(µk
|m0
, (βΛk
)−1) W(Λk
|W0
, ν0
) =
K
∏
k=1
p(µk
, Λk
) (2.17)
͜͜ͰɺW(Λ|W, ν) ΟγϟʔτͰ͋Δɻ(PRML 2.3.6 ͷ
(2.155), (2.156) ࢀর)
23 / 51
Slide 24
Slide 24 text
2-2. ྫɿมࠞ߹Ψε
͜Ε͕ࠞ߹ΨεͰ͋Δ͜ͱΛ͔֬ΊΔʹɺ(2.12) ͷ྆ลΛ
p(π, µ, Λ) ͰׂΓɺજࡏม Z ͰΛͱΔ͜ͱͰ
p(X|π, µ, Λ) =
∑
Z
p(X, Z|π, µ, Λ)
=
N
∏
n=1
{ K
∑
k=1
πk
N(xn
|µk
, Λ−1
k
)
} (2.18)
ͱͳΓɺ͔֬ʹ͜ͷϞσϧࠞ߹ΨεϞσϧͰ͋Δɻ
ͦΕͰɺ͜ͷઃఆͷԼͰؔͷ࠷େԽΛۙࣅతʹߦ͏ͨΊʹɺҎ
ԼͷΑ͏ͳͷղ ((2.6) ʹରԠ) Λߦ͏ɻ
q(Z, π, µ, Λ) = q(Z)q(π, µ, Λ) (2.19)
24 / 51
Slide 25
Slide 25 text
2-2. ྫɿมࠞ߹Ψε
ͦΕͰɺ(2.11) Λ༻͍ͯҼࢠ q(Z) ͷߋ৽ࣜҎԼͷΑ͏ʹͳΔɻ
ln q⋆(Z) = Eπ,µ,Λ
[ln p(X, Z, π, µ, Λ)] + const. (2.20)
(2.20) ͷӈลʹ (2.12) ͷӈลΛೖ͠ɺZ ʹґଘ͠ͳ͍߲ const. ʹ
ٵऩͯ͠͠·͏ͱɺ
ln q⋆(Z) = Eπ
[ln p(Z|π)] + Eµ,Λ
[ln p(X|Z, µ, Λ)] + const. (2.21)
ͱͳΔɻ
͜͜Ͱɺ(2.13) Λ༻͍Δͱ
Eπ
[ln p(Z|π)] = Eπ
[
N
∑
n=1
K
∑
k=1
znk
ln πk
]
=
N
∑
n=1
K
∑
k=1
znk
E
[
ln πk
]
(2.22)
ͱͳΔɻ
25 / 51
Slide 26
Slide 26 text
2-2. ྫɿมࠞ߹Ψε
·ͨɺ(2.15) Λ༻͍Δͱ
Eµ,Λ
[ln p(X|Z, µ, Λ)]
=
N
∑
n=1
K
∑
k=1
znk
Eµ,Λ
[
ln N(xn
|µk
, Λ−1
k
)
]
=
N
∑
n=1
K
∑
k=1
znk
(
1
2
E
[
ln |Λk
|
]
−
D
2
ln (2π)
−
1
2
Eµk,Λk
[
(xn
− µk
)TΛk
(xn
− µk
)
]
)
(2.23)
ͱͳΔɻ
͜͜Ͱɺ࠷ޙͷΠίʔϧͰΨεͷ͋ΒΘͳࣜ (2.16) Λ༻͍ͨɻ
26 / 51
Slide 27
Slide 27 text
2-2. ྫɿมࠞ߹Ψε
(2.22) ͱ (2.23) Λ༻͍Δͱɺ(2.21)
ln q⋆(Z) =
N
∑
n=1
K
∑
k=1
znk
ln ρnk
+ const. (2.24)
ͱॻ͚Δɻ
͜͜Ͱɺ
ln ρnk
=E
[
ln πk
]
+
1
2
E
[
ln |Λk
|
]
−
D
2
ln (2π)
−
1
2
Eµk,Λk
[
(xn
− µk
)TΛk
(xn
− µk
)
] (2.25)
ͱఆٛͨ͠ɻ
27 / 51
Slide 28
Slide 28 text
2-2. ྫɿมࠞ߹Ψε
Ҏ্ΑΓɺ q⋆(Z)
q⋆(Z) ∝
N
∏
n=1
K
∏
k=1
ρznk
nk
(2.26)
ͱͳΔɻ
͜͜Ͱɺ(2.25) ʹ µ Λ ͷظؚ͕·ΕΔͷͰɺ q⋆(Z) ղ
(2.19) ͷଞͷҼࢠ q(π, µ, Λ) ͷܗʹґଘ͢Δ͜ͱʹҙɻ
ن֨Խ݅ΑΓɺq⋆(Z) ҎԼͷΑ͏ʹͳΔɻ(PRML ԋश 10.12)
q⋆(Z) =
N
∏
n=1
K
∏
k=1
rznk
nk
=
N
∏
n=1
q⋆(zn
) (2.27)
͜͜Ͱ rnk
ҎԼͷΑ͏ʹఆٛ͞ΕΔɻ
rnk
=
ρnk
K
∑
j=1
ρnj
(2.28)
28 / 51
Slide 29
Slide 29 text
2-2. ྫɿมࠞ߹Ψε
࣍ʹɺ(2.19) ͷҼࢠ q(π, µ, Λ) ͷߋ৽ࣜʹ͍ͭͯߟ͑Δɻ
q(Z) ͷ࣌ͱಉ͡Α͏ʹɺߋ৽ࣜͷҰൠతͳࣜ (2.11) Λ༻͍Δͱɺ
ln q⋆(π, µ, Λ) = EZ
[ln p(X, Z, π, µ, Λ)] + const. (2.29)
ͱͳΓɺ(2.29) ͷӈลʹ (2.12) ͷӈลΛೖ͢Δͱɺ
ln q⋆(π, µ, Λ) =EZ
[ln p(X|Z, µ, Λ)p(Z|π)p(π)p(µ|Λ)p(Λ)] + const.
= ln p(π) + EZ
[ln p(Z|π)]
+ ln p(µ, Λ) + EZ
[ln p(X|Z, µ, Λ)] + const.
(2.30)
ͱͳΔɻ
29 / 51
Slide 30
Slide 30 text
2-2. ྫɿมࠞ߹Ψε
͜͜Ͱɺ(2.13) ͱ (2.14) ͱ (2.15) ͱ (2.17) Λ͏ͱɺ(2.30) ҎԼͷΑ
͏ʹͳΔɻ
ln q⋆(π, µ, Λ) =
K
∑
k=1
{
(α0
+ 1) ln πk
+ ln πk
N
∑
n=1
E[znk
]
+ ln p(µk
, Λk
) +
N
∑
n=1
E[znk
] ln N(xn
|µk
, Λ−1
k
)
}
+ const.
(2.31)
͜ͷมܗͰΘ͔ͬͨ͜ͱɺ q⋆(π, µ, Λ) ҎԼͷΑ͏ʹղͰ͖
Δ͜ͱͰ͋Δɻ
q⋆(π, µ, Λ) =
K
∏
k=1
q⋆(πk
)q⋆(µk
, Λk
) (2.32)
30 / 51
Slide 31
Slide 31 text
2-2. ྫɿมࠞ߹Ψε
·ͨɺ(2.31) ͷӈลͷதʹݱΕ͍ͯΔظ E[znk
] જࡏม znk
ͷ
q⋆(Z) ͷԼͰͷظͳͷͰɺ(2.27) Λ༻͍Δͱ
E[znk
] =
∑
Z
znk
q⋆(Z)
=
∑
z1
· · ·
∑
zN
znk
N
∏
n′=1
q⋆(zn′
)
=
[ ∑
z1
q⋆(z1
)
]
=1
· · ·
[ ∑
zn
znk
q⋆(zn
)
]
· · ·
[ ∑
zN
q⋆(zN
)
]
=1
=
∑
zn
znk
K
∏
k′=1
rznk′
nk′
= rnk
(2.33)
ͱͳΔɻ
࠷ޙͷΠίʔϧɺऔΓ͏Δͯ͢ͷ zn
ͷͷதͰɺznk
= 1 ͱͳΔ
߲͔͠Βͳ͍͜ͱΛ༻͍ͨɻ
31 / 51
Slide 32
Slide 32 text
2-2. ྫɿมࠞ߹Ψε
͜ΕΑΓ·ͣɺ(2.32) ͷҼࢠ q⋆(π) =
K
∏
k=1
q⋆(πk
) ͷରɺ(2.31) ΑΓ
ln q⋆(π) =
K
∑
k=1
ln πk
{
(α0
+ 1) +
N
∑
n=1
rnk
}
+ const. (2.34)
ͱͳΔͷͰɺq⋆(π) ҎԼͷΑ͏ʹσΟϦΫϨʹͳΔɻ
q⋆(π) = Dir(π|α) = C(α)
K
∏
k=1
παk+1
k
(2.35)
͜͜Ͱɺ
αk
= α0
+ Nk
(2.36)
ͱఆٛ͠ɺNk
ҎԼͰఆٛ͞ΕΔɻ
Nk
=
N
∑
n=1
rnk
(2.37)
32 / 51
Slide 33
Slide 33 text
2-2. ྫɿมࠞ߹Ψε
࣍ʹɺ(2.31) ΑΓ (2.32) ͷҼࢠ q⋆(µk
, Λk
) ͷରҎԼͷΑ͏ʹΨ
ε-ΟγϟʔτʹͳΔɻ(PRML ԋश 10.13 ࢀর)
q⋆(µk
, Λk
) = N(µk
|mk
, (βk
Λk
)−1) W(Λk
|Wk
, νk
) (2.38)
͜͜ͰɺҎԼͷΑ͏ʹఆٛͨ͠ɻ
βk
= β0
+ Nk
(2.39)
mk
=
1
βk
(β0
m0
+ Nk
xk
) (2.40)
W−1
k
= W−1
0
+ Nk
Sk
+
β0
Nk
β0
+ Nk
(xk
− m0
)(xk
− m0
)T (2.41)
νk
= ν0
+ Nk
(2.42)
xk
=
1
Nk
N
∑
n=1
rnk
xn
(2.43)
Sk
=
1
Nk
N
∑
n=1
rnk
(xn
− xk
)(xn
− xk
)T (2.44)
33 / 51
Slide 34
Slide 34 text
2-2. ྫɿมࠞ߹Ψε
·ͨɺྔ Nk
, xk
, Sk
ΛٻΊΔʹɺ(2.37), (2.43), (2.44) ΑΓɺrnk
Λ
ٻΊΔඞཁ͕͋Δɻ
(2.28) ͱ (2.25) ΑΓɺҎԼͷظΛٻΊΕΑ͍͜ͱ͕Θ͔Δɻ
(PRML ԋश 10.14)
Eµk,Λk
[
(xn
−µk
)TΛk
(xn
− µk
)
]
=Dβ−1
k
+ νk
(xn
− mk
)TWk
(xn
− mk
) (2.45)
E
[
ln |Λk
|
]
=
D
∑
i=1
ψ
(
νk
+ 1 − i
2
)
+ D ln 2 + ln |Wk
| (2.46)
E
[
ln πk
]
=ψ(αk
) − ψ(ˆ
α) (2.47)
͜͜Ͱɺψ(·) PRML ͷ B Ͱఆٛ͞Ε͍ͯΔσΟΨϯϚؔͰ͋
Γɺˆ
α =
∑
k
αk
Ͱ͋Δɻ
34 / 51
Slide 35
Slide 35 text
2-2. ྫɿมࠞ߹Ψε
Ҏ্Λ·ͱΊΔͱɺࠞ߹ΨεϞσϧͷมਪ๏ɺҎԼͷΑ͏ʹ
EM ΞϧΰϦζϜͷ E εςοϓͱ M εςοϓʹࣅͨม E εςοϓͱ
ม M εςοϓͷ෮Ͱ͋Δɻ
1. (ม E εςοϓ) ύϥϝʔλͷݱࡏͷࣄޙΛ༻͍ͯɺظ
(2.45), (2.46), (2.47) Λܭࢉ͠ɺෛ୲ rnk
ΛٻΊΔɻ
2. (ม M εςοϓ) ٻΊͨෛ୲Λ༻͍ͯɺࣄޙ q⋆(π) ͱ
q⋆(µk
, Λk
) ΛͦΕͧΕ (2.35) ͱ (2.38) ͔Βܭࢉ͢Δɻ
35 / 51
Slide 36
Slide 36 text
2-3. ہॴతมਪ๏
͜͜·Ͱ͖ٞͯͨ͠ਪ๏ (ͷղ) Λ·ͱΊΔɻ
ಉ࣌ p(X, Z) ͕༩͑ΒΕ͍ͯΔ࣌ɺ
p(Z|X) =
p(X, Z)
∫
p(X, Z) dZ
(2.48)
্ͷΑ͏ʹࣄޙ p(Z|X) ʹͯ͠ٻΊ͍͕ͨɺͷ Z(ύϥϝʔλ
ͱજࡏม) ͷੵ͕Ͱ͖ͳ͍ͱ͢Δɻ
ҰํͰɺࣄޙ p(Z|X) ൚ؔ L[q]
L[q] =
∫
q(Z) ln
{
p(X, Z)
q(Z)
}
dZ (2.49)
Λ࠷େʹ͢Δ q(Z) Ͱ͋Δɻ
36 / 51
Slide 37
Slide 37 text
2-3. ہॴతมਪ๏
ͦ͜Ͱɺq(Z) ΛҎԼͷΑ͏ʹ
q(Z) =
M
∏
i=1
qi
(Zi
) (2.50)
ղ͠ɺ(2.49) ʹೖ͢Δͱ൚ؔ L[q] ҎԼͷΑ͏ʹͳΔɻ
L[qi
] =
∫ M
∏
i=1
qi
(Zi
) ln
{
p(X, Z)
∏
M
i=1
qi
(Zi
)
}
dZ (2.51)
ͦͯ͠ɺL[qi
] ʹ͢Δͷू߹ {q⋆
i
(Zi
)} ΛٻΊͯɺҎԼͷ q⋆(Z)
q⋆(Z) =
M
∏
i=1
q⋆
i
(Zi
) (2.52)
ΛࣄޙΛۙࣅ͢Δͱ͢ΔͷͰ͋ͬͨɻ
37 / 51
Slide 38
Slide 38 text
2-3. ہॴతมਪ๏
ͭ·Γɺ͜ͷͷղࣄޙ p(Z|X) Λͯ͢ͷ֬มͷશ
ͳࣄޙͷۙࣅ q⋆(Z) ΛٻΊΔͱ͍͏ҙຯͰɺେҬతมਪ๏ͷҰ
छͰ͋Δɻ
࣍ʹհ͢Δਪ๏ہॴతมਪ๏Ͱ͋Δɻ
۩ମతʹɺಉ࣌ p(X, Z) ͷԼքͱͳΔؔΛ h(X, Z, ξ) ͱ͢Δɻ
ͭ·ΓɺҎԼͷΑ͏ͳؔ h(X, Z, ξ) Ͱ͋Δɻ
p(X, Z) ≥ h(X, Z, ξ) (2.53)
͜ͷ h(X, Z, ξ) Z ͷੵ͕Ͱ͖Δ͘Β͍γϯϓϧͳؔͰ͋Δɻ
͋ͱͰ۩ମྫͰઆ໌͢Δ͕ɺh(X, Z, ξ) ξ ʹґଘ͢ΔܗͰॻ͚ͯɺ
(2.53) ʹ͓͍ͯɺ͋Δ Z⋆ ͰΠίʔϧཱ͕͢Δͱ͖ͷ ξ Λ ξ⋆ ͱ͢Δɻ
38 / 51
Slide 39
Slide 39 text
2-3. ہॴతมਪ๏
(2.48) ΑΓɺࣄޙ p(Z|X) Λؔ h(X, Z, ξ⋆) Λ༻͍ͯ
p(Z|X) ∼
p(X, Z)
∫
h(X, Z, ξ⋆) dZ
(2.54)
ͱۙࣅ͢Δํ๏Ͱ͋Δɻ
ξ = ξ⋆ ͷͱ͖ɺෆࣜ (2.53) ɺ͋Δ Z⋆ ͰࣜʹͳΔ͕ɺଞͷ Z
ͰҰൠతʹࣜʹͳΒͳ͍ɻ
ͭ·ΓɺݶΒΕͨ (ہॴతͳ)Z ͰҰக͢ΔΑ͏ͳؔ h(X, Z, ξ⋆) Λ༻
͍ͯࣄޙΛۙࣅ͢ΔͷͰɺہॴతมਪ๏Ͱ͋Δɻ
Ͱɺؔ h(X, Z, ξ⋆) ͷٻΊํΛઆ໌͢Δɻ
39 / 51
Slide 40
Slide 40 text
2-3. ہॴతมਪ๏
·ͣɺҎԼͷը૾ͷΑ͏ͳತؔ f(x) Λߟ͑Δɻ
·ͨɺݪΛ௨Δઢ ηx Λߟ͑Δɻ(ը૾ͷ λ ϛε)
͜ͷઢ ηx ؔ f(x) ͷԼքʹͳ͍ͬͯΔ͕ɺ͖͕ η ͷ࠷ྑͷ
Լք (f(x) ͱͷ͕ࠩ࠷খ͍͞) Ͱͳ͍ɻ
40 / 51
Slide 41
Slide 41 text
2-3. ہॴతมਪ๏
࠷ྑͷԼքҎԼͷΑ͏ʹɺf(x) ͷઢͰ͋Δɻ
͜ͷઢͷยΛ −g(η) ͱ͢Δͱɺઢͷํఔࣜ ηx − g(η) Ͱ͋Δɻ
͜ͷย −g(η) ҎԼͷΑ͏ʹͯ͠ٻΊΔ͜ͱ͕Ͱ͖Δɻ
−g(η) = min
x
{f(x) − ηx}
→ g(η) = max
x
{ηx − f(x)} (2.55)
ηx − f(x) Λ࠷େʹ͢Δ x ͷ x ࠲ඪͰ͋Δɻ
41 / 51
Slide 42
Slide 42 text
2-3. ہॴతมਪ๏
࣍ʹɺf(x) Λ g(η) Ͱදݱ͢Δ͜ͱΛߟ͑Δɻ
f(x) ͷತؔੑʹΑΓɺ͋Δ x ʹ͓͍ͯɺ
max
η
{ηx − g(η)} (2.56)
f(x) ͷ x ʹ͓͚Δͷ y ࠲ඪͰ͋Γɺf(x) Ͱ͋Δɻ
ͭ·Γɺ
f(x) = max
η
{ηx − g(η)} (2.57)
Ͱ͋Δɻ
42 / 51
Slide 43
Slide 43 text
2-3. ہॴతมਪ๏
(2.55) ͱ (2.56) ΑΓɺؔ f(x) ͱԼքͱͳΔઢͷย −g(η) ʹҎ
ԼͷΑ͏ʹର͕ؔ͋Δ͜ͱ͕Θ͔Δɻ
f(x) = max
η
{ηx − g(η)} (2.58)
g(η) = max
x
{ηx − f(x)} (2.59)
ͦΕͰɺ͜ͷςΫχοΫΛ༻͍ͯɺϩδεςΟοΫγάϞΠυ σ(x)
σ(x) =
1
1 + e−x
(2.60)
ͷԼքΛٻΊΔɻ
43 / 51
Slide 44
Slide 44 text
2-3. ہॴతมਪ๏
ͨͩ͠ɺϩδεςΟοΫγάϞΠυ͕ತؔͰͳ͍ͷͰɺϩδεςΟο
ΫγάϞΠυͷରΛҎԼͷΑ͏ʹมܗ͢Δɻ
ln σ(x) = − ln (1 + e−x) = − ln {e−x/2(ex/2 + e−x/2)}
=
x
2
− ln (ex/2 + e−x/2)
(2.61)
͜͜Ͱɺؔ f(x) = − ln (ex/2 + e−x/2) x2 ͷತؔͰ͋Δɻ
(PRML ԋश 10.31)
͜Ε͔Βɺؔ f(x) = − ln (ex/2 + e−x/2) ͷԼքΛͱΊΔɻ
·ͣɺx2 ͷತؔͰ͋ΔͷͰɺ(2.59) ΑΓؔ g(η)
g(η) = max
x2
{ηx2 − f(x)} (2.62)
ͱͳΔɻ
44 / 51
Slide 45
Slide 45 text
2-3. ہॴతมਪ๏
͜͜Ͱɺηx2 − f(x) Λ x2 Ͱඍ͢Δͱ
d
dx2
[
ηx2 − f(x)
]
= η −
dx
dx2
·
d
dx
f(x) (2.63)
ͱͳΓɺ
dx
dx2
=
1
2x
(2.64)
d
dx
f(x) = −
d
dx
[
ln (ex/2 + e−x/2)
]
= −
1
2
tanh
x
2
(2.65)
ΑΓɺ
d
dx2
[
ηx2 − f(x)
]
= η +
1
4x
tanh
x
2
(2.66)
ͱͳΔɻ
45 / 51
Slide 46
Slide 46 text
2-3. ہॴతมਪ๏
(2.66) ͷӈล͕ 0 ͱͳΔ x Λ ξ ͱ͢Δͱɺξ ͱ η
η = −
1
4ξ
tanh
ξ
2
= −
1
2ξ
[
σ(ξ) −
1
2
]
= −λ(ξ) (2.67)
ͱ͍͏ؔʹ͋Δ͜ͱ͕Θ͔Δɻ
(2.62) ΑΓɺg(η)
g(η) =
[
ηx2 − f(x)
]
x=ξ,η=−λ(ξ)
= −λ(ξ)ξ2 − f(ξ)
= −λ(ξ)ξ2 + ln (eξ/2 + e−ξ/2)
(2.68)
ͱͳΔɻ
46 / 51
Slide 47
Slide 47 text
2-3. ہॴతมਪ๏
͜ΕΑΓɺ(2.58) ͱ (2.61) ͱ (2.68) ΑΓ
ln σ(x) =
x
2
+ max
η
{ηx2 − g(η)}
≥
x
2
− λ(ξ)x2 + λ(ξ)ξ2 − ln (eξ/2 + e−ξ/2)
=
x
2
− λ(ξ)x2 + λ(ξ)ξ2 −
ξ
2
+ ln σ(ξ)
=
1
2
(x − ξ) − λ(ξ)(x2 − ξ2) + ln σ(ξ)
(2.69)
ͱͳΓɺରͷ୯ௐ૿ՃੑΑΓɺσ(x) ͷԼք
σ(x) ≥ σ(ξ) exp
{
1
2
(x − ξ) − λ(ξ)(x2 − ξ2)
}
(2.70)
ͱͳΔɻ
47 / 51
Slide 48
Slide 48 text
2-4. มϩδεςΟοΫճؼ
ͦΕͰɺ(2.70) Λ༻͍ͯϩδεςΟοΫճؼͷมਪΛߦ͏ɻ
·ͣɺϩδεςΟοΫճؼͷؔ p(t|x, w)
p(t|x, w) =σ(a)t{1 − σ(a)}1−t
=
(
1
1 + e−a
)t
(
1 −
1
1 + e−a
)1−t
=
1
(1 + e−a)t
e−a(1−t)
(1 + e−a)1−t
=eat
e−a
1 + e−a
= eat
1
1 + e−(−a)
= eatσ(−a)
(2.71)
ͱͳΔɻ
͜͜Ͱɺa = wTϕ Ͱ͋Δɻ
·ͨɺύϥϝʔλ w ͷࣄલΛҎԼͷΑ͏ʹΨεͰԾఆ͢Δɻ
p(w) = N(w|m0
, S0
) (2.72)
͜͜Ͱɺm0
, S0
ϋΠύʔύϥϝʔλɻ
48 / 51
Slide 49
Slide 49 text
2-4. มϩδεςΟοΫճؼ
ೖྗσʔλͷू߹ X = {x1
, x2
, · · · , xN
} ͱͦΕͧΕʹରԠ͢Δඪม
ͷू߹ t = {t1
, t2
, · · · , tN
} Λ༻ҙ͢Δɻ
ڭࢣσʔλҰͭҰ͕ͭ (2.71) ͔Βಠཱʹੜ͞Ε͍ͯΔͱ͢Δͱɺ
ؔ p(t|X, w) ҎԼͷΑ͏ʹͳΔɻ
p(t|X, w) =
N
∏
n=1
p(tn
|xn
, w) (2.73)
ࠓճٻΊ͍ͨͷɺࣄޙ p(w|t, X) Ͱ͋ΓɺҎԼͷࣜͰදͤΔɻ
p(w|t, X) =
p(w)p(t|X, w)
∫
p(w)p(t|X, w) dw
(2.74)
͜ͷ (2.74) ͷӈลͷ͕ؔϩδεςΟοΫγάϞΠυͷੵ
ʹͳ͍ͬͯΔͷͰɺੵͰ͖ͳ͍ɻ
49 / 51
Slide 50
Slide 50 text
2-4. มϩδεςΟοΫճؼ
ͦ͜Ͱɺෆࣜ (2.70) Λ͏ͱҎԼͷΑ͏ʹͳΔɻ
p(w)p(t|X, w)
=p(w)
N
∏
n=1
eantn σ(−an
)
≥p(w)
N
∏
n=1
eantn σ(ξn
) exp {(−an
− ξn
)/2 − λ(ξn
)(a2
n
− ξ2
n
)}
=p(w)
N
∏
n=1
σ(ξn
) exp {an
tn
− (an
+ ξn
)/2 − λ(ξn
)(a2
n
− ξ2
n
)}
(2.75)
͜͜Ͱɺan
= wTϕn
Ͱ͋Δɻ
50 / 51
Slide 51
Slide 51 text
2-4. มϩδεςΟοΫճؼ
͜͜Ͱɺ(2.75) ͷ྆ลͷରΛऔΔͱ
ln {p(w)p(t|X, w)} ≥ ln p(w) +
N
∑
n=1
[
ln σ(ξn
) + wTϕn
tn
− (wTϕn
+ ξn
)/2 − λ(ξn
)([wTϕn
]2 − ξ2
n
)
]
(2.76)
ͱͳΓɺ(2.76) ͷӈล w ͷ 2 ࣍ࣜʹͳ͍ͬͯΔͷͰɺp(w)p(t|X, w)
ͷԼքΨεͱͳΔɻ
ln {p(w)p(t|X, w)} ≥ q(w) = N(w|mN
, SN
) (2.77)
͜͜ͰɺmN
, SN
ҎԼͰఆٛ͞ΕΔɻ
mN
= SN
(
S−1
0
m0
+
N
∑
n=1
(tn
− 1/2)ϕn
)
(2.78)
S−1
N
= S−1
0
+ 2
N
∑
n=1
λ(ξn
)ϕn
ϕT
n
(2.79)
ΨεʹۙࣅͰ͖ͨͷͰɺ(2.74) ͷੵ͕ՄೳʹͳΓɺࣄޙ
͕ٻ·Δɻ
51 / 51