Share
Embed
言語処理のための機械学習入門 5.4 条件付確率場
言語処理のための機械学習入門 (自然言語処理シリーズ)
5.4 条件付確率場
http://amzn.to/2f79qd9
yellow-black
April 16, 2020
Transcript
42 slides
-
ݴޠॲཧͷͨΊͷػցֶशೖ 5.4 ݅֬ ߴ ढ़࡞ April 23, 2020 ߴ ढ़࡞
April 23, 2020 1 / 42 -
࣍ 1 ܥྻϥϕϦϯάͷఆࣜԽ 2 ӅΕϚϧίϑϞσϧ (HMM) ͱͷؔ࿈ ੜϞσϧͱࣝผϞσϧ 3 HMM
HMM ͷ 4 ݅֬ (CRF) ఆࣜԽ ༧ଌ (ਪఆ) ֶश (ύϥϝʔλਪఆ) ∑ y∈Y ϕ(x, y)P(y|x) ͷܭࢉ P(yt−1 , yt |x) ͷܭࢉ forward-backword algorithm Zx,w ͷܭࢉ ߴ ढ़࡞ April 23, 2020 2 / 42 -
ܥྻϥϕϦϯάͷఆࣜԽ ࣍ 1 ܥྻϥϕϦϯάͷఆࣜԽ 2 ӅΕϚϧίϑϞσϧ (HMM) ͱͷؔ࿈ ੜϞσϧͱࣝผϞσϧ 3
HMM HMM ͷ 4 ݅֬ (CRF) ఆࣜԽ ༧ଌ (ਪఆ) ֶश (ύϥϝʔλਪఆ) ∑ y∈Y ϕ(x, y)P(y|x) ͷܭࢉ P(yt−1 , yt |x) ͷܭࢉ forward-backword algorithm Zx,w ͷܭࢉ ߴ ढ़࡞ April 23, 2020 3 / 42 -
ܥྻϥϕϦϯάͷఆࣜԽ ܥྻϥϕϦϯάͷఆࣜԽ ༩͑ΒΕͨܥྻΛ x, ग़ྗͷܥྻ (ϥϕϧྻ) Λ y ͱ͢Δɽ ͦͷ্Ͱɺ༩͑ΒΕֶͨशσʔλ
D = {(x1, y1), (x2, y2), · · · , (x|D|, y|D|)} ͔Β x → y ͱ ͳΔΑ͏ͳରԠؔ (ؔ) Λ౷ܭతʹਪఆ͢Δͱͯ͠ଊ͑ΒΕΔɽ ֶशσʔλશͳͷͰͳ͘ɺະͷσʔλʹର͢Δ༧ଌৗʹ֬తͳৼΔ͍Λ͢Δ ͱߟ͑ΒΕΔ͜ͱ͔ΒɺҎԼͷΑ͏ͳ֬Λ༻͍Δͷ͕ଥͰ͋Δɽ P(y|x) ͦͷ্Ͱɺ༧ଌΛ ˆ y = arg max y∈Y P(y|x) ͱ͢Δ͜ͱͰɺग़ྗͷܥྻΛٻΊΔͱͯ͠ఆࣜԽͰ͖Δɽ ͜͜ͰɺY ϥϕϧྻͷީิू߹Ͱ͋Δɽͭ·ΓɺҐஔ t ʹ͓͚Δग़ྗܥྻதͷཁૉ yt ∈ y ͕औΓ͏Δϥϕϧͷू߹Λ Y ͱͨ͠ͱ͖ɺY = YT (T ܥྻͷཁૉͱͨ͠) Ͱ͋Δɽ ߴ ढ़࡞ April 23, 2020 4 / 42 -
ӅΕϚϧίϑϞσϧ (HMM) ͱͷؔ࿈ ࣍ 1 ܥྻϥϕϦϯάͷఆࣜԽ 2 ӅΕϚϧίϑϞσϧ (HMM) ͱͷؔ࿈
ੜϞσϧͱࣝผϞσϧ 3 HMM HMM ͷ 4 ݅֬ (CRF) ఆࣜԽ ༧ଌ (ਪఆ) ֶश (ύϥϝʔλਪఆ) ∑ y∈Y ϕ(x, y)P(y|x) ͷܭࢉ P(yt−1 , yt |x) ͷܭࢉ forward-backword algorithm Zx,w ͷܭࢉ ߴ ढ़࡞ April 23, 2020 5 / 42 -
ӅΕϚϧίϑϞσϧ (HMM) ͱͷؔ࿈ ӅΕϚϧίϑϞσϧ (HMM) ͱͷؔ࿈ CRF ͱ HMM ͱʹܥྻϥϕϦϯάͰ༻͍ΒΕΔख๏Ͱ͋Δɽ
HMM ͱ CRF ͷओͳҧ͍ɺੜϞσϧͱࣝผϞσϧͰ͋Δ͔Ͱ͋Δɽ ߴ ढ़࡞ April 23, 2020 6 / 42 -
ӅΕϚϧίϑϞσϧ (HMM) ͱͷؔ࿈ ੜϞσϧͱࣝผϞσϧ ࣍ 1 ܥྻϥϕϦϯάͷఆࣜԽ 2 ӅΕϚϧίϑϞσϧ (HMM)
ͱͷؔ࿈ ੜϞσϧͱࣝผϞσϧ 3 HMM HMM ͷ 4 ݅֬ (CRF) ఆࣜԽ ༧ଌ (ਪఆ) ֶश (ύϥϝʔλਪఆ) ∑ y∈Y ϕ(x, y)P(y|x) ͷܭࢉ P(yt−1 , yt |x) ͷܭࢉ forward-backword algorithm Zx,w ͷܭࢉ ߴ ढ़࡞ April 23, 2020 7 / 42 -
ӅΕϚϧίϑϞσϧ (HMM) ͱͷؔ࿈ ੜϞσϧͱࣝผϞσϧ ੜϞσϧͱࣝผϞσϧ ੜϞσϧ σʔλͷഎޙʹ͋ΔੜϝΧχζϜΛɺಉ࣌ P(x, y) ͱͯ͠ϞσϧԽ͢Δํ๏Ͱ͋
Δɽࣝผʹ͍ͪΔ߹ɺP(x) = y P(x, y) ͱҎԼͷࣜʹΑΓ P(y|x) ΛٻΊΔɽ P(y|x) = P(x, y) P(x) • ɺࣝผʹ༻͍Δ߹ɺؒతͳͨΊਫ਼͕͘ͳΔՄೳੑ͕͋Δɽ • རͱͯ͠ɺx, y ͷ͕ಘΒΕΔͷͰɺٖࣅσʔλͷੜɺ֎ΕݕͳͲʹ Ԡ༻͢Δ͜ͱ͕Ͱ͖Δɽ • ྫɿHMMɺGANɺVAEɺLDAɺetc. ߴ ढ़࡞ April 23, 2020 8 / 42 -
ӅΕϚϧίϑϞσϧ (HMM) ͱͷؔ࿈ ੜϞσϧͱࣝผϞσϧ ੜϞσϧͱࣝผϞσϧ ࣝผϞσϧ ࣝผʹ༻͍ΒΕΔ P(y|x) ΛతʹϞσϧԽ͢Δɽ •
རͱͯ͠ɺతʹϞσϧԽ͍ͯ͠ΔͨΊɺൺֱతʹਫ਼͕ߴ͘ͳΔՄೳੑ͕͋ Δɽ • ੜϞσϧΑΓɺඞཁͳԾఆͷ͕গͳ͍έʔε͕ଟ͍ɽ • ྫɿCRFɺLogistic RegressionɺSoftmax + DNNɺetc. ߴ ढ़࡞ April 23, 2020 9 / 42 -
HMM ࣍ 1 ܥྻϥϕϦϯάͷఆࣜԽ 2 ӅΕϚϧίϑϞσϧ (HMM) ͱͷؔ࿈ ੜϞσϧͱࣝผϞσϧ 3
HMM HMM ͷ 4 ݅֬ (CRF) ఆࣜԽ ༧ଌ (ਪఆ) ֶश (ύϥϝʔλਪఆ) ∑ y∈Y ϕ(x, y)P(y|x) ͷܭࢉ P(yt−1 , yt |x) ͷܭࢉ forward-backword algorithm Zx,w ͷܭࢉ ߴ ढ़࡞ April 23, 2020 10 / 42 -
HMM HMM HMM ੜϞσϧͰ͋Δɽͭ·ΓɺP(x, y) ΛϞσϦϯά͍ͯ͠Δɽਪ (༧ଌ) ҎԼͷ ܗͰߦΘΕΔɽ ˆ
y = arg max y∈Y P(y|x) = arg max y∈Y P(x, y) P(x) ͜͜ͰɺP(x) y ʹؔͳ͍ͷͰɺ ˆ y = arg max y∈Y P(y|x) = arg max y∈Y P(x, y) ͱͳΔ. ߴ ढ़࡞ April 23, 2020 11 / 42 -
HMM HMM ͷ ࣍ 1 ܥྻϥϕϦϯάͷఆࣜԽ 2 ӅΕϚϧίϑϞσϧ (HMM) ͱͷؔ࿈
ੜϞσϧͱࣝผϞσϧ 3 HMM HMM ͷ 4 ݅֬ (CRF) ఆࣜԽ ༧ଌ (ਪఆ) ֶश (ύϥϝʔλਪఆ) ∑ y∈Y ϕ(x, y)P(y|x) ͷܭࢉ P(yt−1 , yt |x) ͷܭࢉ forward-backword algorithm Zx,w ͷܭࢉ ߴ ढ़࡞ April 23, 2020 12 / 42 -
HMM HMM ͷ HMM ͷ HMM ͷͱͯ͠ɺҎԼͷ 2 ͕ͭߟ͑ΒΕΔɽ •
ಛྔ͕ॊೈʹઃܭͰ͖ͳ͍Ͱ͋Δɽྫ͑ɺHMM Ͱɺ୯ޠͷܥྻͷฒͼॱΛߟ ྀ͢Δ͜ͱ͕Ͱ͖ͳ͍͕ڍ͛ΒΕΔɽଞʹɺing ͰऴΘ͍ͬͯΔͷಈ໊ࢺݱ ࡏਐߦܥͷಈࢺͰ͋Δͱ͍ͬͨ͜ͱΛߟྀͰ͖ͳ͍͕ڍ͛ΒΕΔɽ • (ҰൠʹɺࣝผͷͨΊʹ༻͍ΔੜϞσϧͷ) ༧ଌͷͨΊʹ P(y|x) Λਫ਼Α͘ ٻΊΕྑ͍ͣͳͷʹɺP(x, y) Λֶश͍ͯ͠ΔͰ͋Δɽ͢ͳΘͪ P(y|x) · P(x) Λਪఆ͠Α͏ͱ͍ͯ͠ΔͨΊɺP(x) ͷͿΜ͚ͩ༨ͳֶशΛ͍ͯ͠Δ. ͦͷͨΊɺύϥ ϝʔλۭ͕ؒڊେʹͳΓɺֶशσʔλ͕େྔͰ͔ͭۉ࣭ͳͷͰͳ͍ͱ P(x, y) ֶश͕ ͏·͍͔͘ͳ͍Մೳੑ͕͋Δɽ ߴ ढ़࡞ April 23, 2020 13 / 42 -
݅֬ (CRF) ࣍ 1 ܥྻϥϕϦϯάͷఆࣜԽ 2 ӅΕϚϧίϑϞσϧ (HMM) ͱͷؔ࿈ ੜϞσϧͱࣝผϞσϧ
3 HMM HMM ͷ 4 ݅֬ (CRF) ఆࣜԽ ༧ଌ (ਪఆ) ֶश (ύϥϝʔλਪఆ) ∑ y∈Y ϕ(x, y)P(y|x) ͷܭࢉ P(yt−1 , yt |x) ͷܭࢉ forward-backword algorithm Zx,w ͷܭࢉ ߴ ढ़࡞ April 23, 2020 14 / 42 -
݅֬ (CRF) ݅֬ (CRF) ্هͰઆ໌ͨ͠ HMM ʹରͯ͠ɺCRF P(y|x) Λֶश͢ΔΞϧΰϦζϜɽ
ߴ ढ़࡞ April 23, 2020 15 / 42 -
݅֬ (CRF) ఆࣜԽ ࣍ 1 ܥྻϥϕϦϯάͷఆࣜԽ 2 ӅΕϚϧίϑϞσϧ (HMM) ͱͷؔ࿈
ੜϞσϧͱࣝผϞσϧ 3 HMM HMM ͷ 4 ݅֬ (CRF) ఆࣜԽ ༧ଌ (ਪఆ) ֶश (ύϥϝʔλਪఆ) ∑ y∈Y ϕ(x, y)P(y|x) ͷܭࢉ P(yt−1 , yt |x) ͷܭࢉ forward-backword algorithm Zx,w ͷܭࢉ ߴ ढ़࡞ April 23, 2020 16 / 42 -
݅֬ (CRF) ఆࣜԽ ఆࣜԽ CRF ରઢܗϞσϧ 1 ͷҰछͰ͋Δɽ ͦͷͨΊɺσʔλ D
= {(x1, y1), (x2, y2), · · · , (x|D|, y|D|)} ͕༩͑ΒΕͨͱ͖ͷΫϥεͷ֬ ҎԼͷΑ͏ʹϞσϧԽ͞ΕΔɽ P(y|x) = 1 Zx,w exp(w · ϕ(x, y)) ͜͜ͰɺZx,w ਖ਼نԽఆɺw ॏΈɺϕ(x, y) ૉੑؔͱݺΕΔͷͰɺσʔλͱϥ ϕϧ͔ΒಛྔϕΫτϧΛ࡞͢ΔؔͰ͋Δ (͜ΕʹΑΓɺHMM ΑΓॊೈʹಛྔ͕ઃ ܭͰ͖Δ)ɽ Zx,w = y∈Y exp(w · ϕ(x, y)) Ͱ͋Γɺ࣮ P(y|x) ιϑτϚοΫεؔͱݺΕΔͷʹͳ͍ͬͯΔɽ 1ରઢܗϞσϧͱɺ log Λͱͬͨࡍʹɺͦͷύϥϝʔλ͕ઢܗ݁߹Ͱද͞ΕΔͷͷ͜ͱͰ͋Δɽ ߴ ढ़࡞ April 23, 2020 17 / 42 -
݅֬ (CRF) ༧ଌ (ਪఆ) ࣍ 1 ܥྻϥϕϦϯάͷఆࣜԽ 2 ӅΕϚϧίϑϞσϧ (HMM)
ͱͷؔ࿈ ੜϞσϧͱࣝผϞσϧ 3 HMM HMM ͷ 4 ݅֬ (CRF) ఆࣜԽ ༧ଌ (ਪఆ) ֶश (ύϥϝʔλਪఆ) ∑ y∈Y ϕ(x, y)P(y|x) ͷܭࢉ P(yt−1 , yt |x) ͷܭࢉ forward-backword algorithm Zx,w ͷܭࢉ ߴ ढ़࡞ April 23, 2020 18 / 42 -
݅֬ (CRF) ༧ଌ (ਪఆ) ༧ଌ ༧ଌͷࡍʹɺ ˆ y = arg
max y∈Y P(y|x) = arg max y∈Y 1 Zx,w exp(w · ϕ(x, y)) = arg max y∈Y (w · ϕ(x, y)) ͱͳΔɽҰൠʹɺߴʹղ͘ͷ͕ࠔͳͨΊɺCRF ͰɺૉੑؔʹҎԼͷ੍ݶΛஔ͘ɽ Y = YT Ͱ͋ΔͷͰɺྫ͑ɺܥྻ͕ 20 ୯ޠɺࢺ͕ 10 ͩͱ͢Δͱɺ1020 ͱͳΔɽ ߴ ढ़࡞ April 23, 2020 19 / 42 -
݅֬ (CRF) ༧ଌ (ਪఆ) ہॴੑͷ੍ݶ ͜Εɺೖྗܥྻ x ʹ͍ͭͯͷใɺৗʹ༻͍Δ͜ͱ͕Մೳ͕ͩɺग़ྗܥྻ y ͷใ
ྡ͢Δͷʹ੍ݶ͍ͯ͠Δɽಛྔͷઃܭͷࡍʹɺ͜ͷ੍ݶΛຬͨ͢Α͏ʹઃܭ͢Δ͜ͱ ͕ཁ͞ΕΔɽ ϕk(x, y) = t ϕk(x, yt, yt−1) ͜͜Ͱɺk ɺϕ ͷ k ൪ͷཁૉͰ͋Δɽ ϕ ͷ࣍ݩʹ͍ͭͯɺઃܭ࣍ୈͰม͑Δ͜ͱ͕ՄೳɽԾʹ K ͱ͓͘ͱɺK ݸͷಛྔΛ༻ ͍Δ͜ͱ͕ՄೳʹͳΔ (߹Θͤͯɺw ͷ࣍ݩมԽ)ɽ ߴ ढ़࡞ April 23, 2020 20 / 42 -
݅֬ (CRF) ༧ଌ (ਪఆ) Viterbi algorithm Αͬͯɺ ˆ y =
arg max y∈Y (w · ϕ(x, y)) = arg max y∈Y (w · t ϕ(x, yt, yt−1)) = arg max y∈Y ( t w · ϕ(x, yt, yt−1)) ͱͳΔɽ͜͜·Ͱɺམͱ͢ͱ Viterbi algorithm Ͱߴʹղ͘͜ͱ͕ՄೳʹͳΔɽ ߴ ढ़࡞ April 23, 2020 21 / 42 -
݅֬ (CRF) ֶश (ύϥϝʔλਪఆ) ࣍ 1 ܥྻϥϕϦϯάͷఆࣜԽ 2 ӅΕϚϧίϑϞσϧ (HMM)
ͱͷؔ࿈ ੜϞσϧͱࣝผϞσϧ 3 HMM HMM ͷ 4 ݅֬ (CRF) ఆࣜԽ ༧ଌ (ਪఆ) ֶश (ύϥϝʔλਪఆ) ∑ y∈Y ϕ(x, y)P(y|x) ͷܭࢉ P(yt−1 , yt |x) ͷܭࢉ forward-backword algorithm Zx,w ͷܭࢉ ߴ ढ़࡞ April 23, 2020 22 / 42 -
݅֬ (CRF) ֶश (ύϥϝʔλਪఆ) తؔ CRF ͰɺP(y|x) ͕ϞσϧԽ͞Ε͍ͯΔͨΊɺֶशσʔλ D ʹରͯ͠ɺ͜ΕΛ࠷దԽ
͢Δɽͦ͜Ͱɺతؔͱͯ͠ɺҎԼͷΑ͏ͳ͖݅Λߟ͑Δɽ (xi,yi)∈D P(yi|xi; w) ͜Ε͕࠷େ͖͘ͳΔΑ͏ͳύϥϝʔλΛٻΊΔͨΊɺҎԼͷղ͚ྑ͍ɽ ˆ w = arg max w (xi,yi)∈D P(yi|xi; w) ߴ ढ़࡞ April 23, 2020 23 / 42 -
݅֬ (CRF) ֶश (ύϥϝʔλਪఆ) ର L(w) = (xi,yi)∈D log P(yi|xi;
w) = (xi,yi)∈D log 1 Zxi,w exp(w · ϕ(xi, yi)) = (xi,yi)∈D (w · ϕ(xi, yi) − log Zxi,w ) ߴ ढ़࡞ April 23, 2020 24 / 42 -
݅֬ (CRF) ֶश (ύϥϝʔλਪఆ) ύϥϝʔλͷภඍ ∂L(w) ∂w = ∑ (xi,yi)∈D
( ∂ ∂w w · ϕ(xi, yi) − ∂ ∂w (log Zxi,w ) ) = ∑ (xi,yi)∈D ( ϕ(xi, yi) − 1 Zxi,w ∂ ∂w Zxi,w ) = ∑ (xi,yi)∈D ϕ(xi, yi) − 1 Zxi,w ∑ y∈Y ∂ ∂w exp(w · ϕ(xi, y)) = ∑ (xi,yi)∈D ϕ(xi, yi) − 1 Zxi,w ∑ y∈Y ϕ(xi, y)exp(w · ϕ(xi, y)) = ∑ (xi,yi)∈D ϕ(xi, yi) − ∑ y∈Y ϕ(xi, y) exp(w · ϕ(xi, y)) Zxi,w = ∑ (xi,yi)∈D ϕ(xi, yi) − ∑ y∈Y ϕ(xi, y)P(y|xi) ߴ ढ़࡞ April 23, 2020 25 / 42 -
݅֬ (CRF) ֶश (ύϥϝʔλਪఆ) ࠷ٸޯ๏ ্هͷࣜɺHMM ͷ߹ͱҟͳΓɺղੳతʹղ͘͜ͱࠔͳͷͰɺ࠷ٸޯ๏Λ༻͍Δɽ (L(w) Ԝ
ͳͷͰɺ࠷దղอূ͞Ε͍ͯΔɽ͜Εɺlog exp ͷܗʹͳ͍ͬͯΔ͜ͱʹ ΑΔɽ) wnew = wold + ϵ · ∂L(wold) ∂wold ͦͷࡍɺ y∈Y ϕ(xi, y)P(y|xi) ͯ͢ͷՄೳͳϥϕϧܥྻ (Y = YT ) Λܭࢉ͢Δඞཁ͕ ͋ΔɽͦͷͨΊɺҎԼͰߴʹܭࢉ͢Δํ๏Λߟ͑Δɽ ߴ ढ़࡞ April 23, 2020 26 / 42 -
݅֬ (CRF) ֶश (ύϥϝʔλਪఆ) ࣍ 1 ܥྻϥϕϦϯάͷఆࣜԽ 2 ӅΕϚϧίϑϞσϧ (HMM)
ͱͷؔ࿈ ੜϞσϧͱࣝผϞσϧ 3 HMM HMM ͷ 4 ݅֬ (CRF) ఆࣜԽ ༧ଌ (ਪఆ) ֶश (ύϥϝʔλਪఆ) ∑ y∈Y ϕ(x, y)P(y|x) ͷܭࢉ P(yt−1 , yt |x) ͷܭࢉ forward-backword algorithm Zx,w ͷܭࢉ ߴ ढ़࡞ April 23, 2020 27 / 42 -
݅֬ (CRF) ֶश (ύϥϝʔλਪఆ) y∈Y ϕ(x, y)P(y|x) ͷܭࢉ y∈Y ϕ(x,
y)P(y|x) = y∈Y P(y|x) t ϕ(x, yt−1 , yt ) ہॴੑͷ੍ݶΑΓ = y1y2···yT ∈Y×Y×···Y P(y1 , y2 , . . . , yT |x) t ϕ(x, yt−1 , yt ) = y1∈Y y2∈Y · · · yT ∈Y P(y1 , y2 , . . . , yT |x) t ϕ(x, yt−1 , yt ) ߴ ढ़࡞ April 23, 2020 28 / 42 -
݅֬ (CRF) ֶश (ύϥϝʔλਪఆ) = y1∈Y y2∈Y · · ·
yT ∈Y (P(y1 , y2 , . . . , yT |x)ϕ(x, y0 , y1 )) + y1∈Y y2∈Y · · · yT ∈Y (P(y1 , y2 , . . . , yT |x)ϕ(x, y1 , y2 )) + · · · + y1∈Y y2∈Y · · · yT ∈Y (P(y1 , y2 , . . . , yT |x)ϕ(x, yT −1 , yT )) = y1∈Y ϕ(x, y0 , y1 ) y2∈Y · · · yT ∈Y (P(y1 , y2 , . . . , yT |x)) + y1∈Y y2∈Y ϕ(x, y1 , y2 ) y3∈Y · · · yT ∈Y (P(y1 , y2 , . . . , yT |x)) + · · · + y1∈Y y2∈Y · · · yT −1∈Y yT ∈Y (ϕ(x, yT −1 , yT )P(y1 , y2 , . . . , yT |x)) ߴ ढ़࡞ April 23, 2020 29 / 42 -
݅֬ (CRF) ֶश (ύϥϝʔλਪఆ) = y1∈Y ϕ(x, y0, y1)P(y1|x) +
y1∈Y y2∈Y ϕ(x, y1, y2)P(y1, y2|x) + · · · + yT −1∈Y yT ∈Y ϕ(x, yT−1, yT )P(yT−1, yT |x) = t yt−1∈Y yt∈Y ϕ(x, yt−1, yt)P(yt−1, yt|x) ্هʹ͓͍ͯɺP(yt−1, yt|x) ͕ܭࢉͰ͖Εྑ͍ͷͰɺ͔͜͜Βܭࢉ͍ͯ͘͠ɽ (y0, yT+1 ͷ߹ɺͦΕͧΕ Y = {BOS}, Y = {EOS} Ͱ͋Δɽ) ߴ ढ़࡞ April 23, 2020 30 / 42 -
݅֬ (CRF) ֶश (ύϥϝʔλਪఆ) ࣍ 1 ܥྻϥϕϦϯάͷఆࣜԽ 2 ӅΕϚϧίϑϞσϧ (HMM)
ͱͷؔ࿈ ੜϞσϧͱࣝผϞσϧ 3 HMM HMM ͷ 4 ݅֬ (CRF) ఆࣜԽ ༧ଌ (ਪఆ) ֶश (ύϥϝʔλਪఆ) ∑ y∈Y ϕ(x, y)P(y|x) ͷܭࢉ P(yt−1 , yt |x) ͷܭࢉ forward-backword algorithm Zx,w ͷܭࢉ ߴ ढ़࡞ April 23, 2020 31 / 42 -
݅֬ (CRF) ֶश (ύϥϝʔλਪఆ) ه๏ͷಋೖ ೖྗܥྻͷ͞Λ T ͱ͢Δ (T ܥྻ͝ͱʹҟͳ͍ͬͯͯྑ͍)ɽ
֤ϥϕϧྻ y ʹ͓͍ͯɺy0 = BOS, yT+1 = EOS ͱͳ͍ͬͯΔͱ͢Δɽ هड़ͷ؆ུԽͷͨΊɺҎԼͷΑ͏ͳه๏Λಋೖ͢Δɽ y0:T −2 = y0y1y2···yT −2∈Y×Y×Y×···Y ·ͨɺψt(yt−1, yt) = exp(w · ϕ(x, yt−1, yt)) ͱ͓͘ɽ͕ͨͬͯ͠ɺ P(y|x) = t ψt(yt−1, yt) Zx,w ࠓճͷܭࢉͷࡍɺw, x Ұఆͱߟ͑ΒΕΔͷͰɺҾʹೖΕ͍ͯͳ͍ɽ ߴ ढ़࡞ April 23, 2020 32 / 42 -
݅֬ (CRF) ֶश (ύϥϝʔλਪఆ) P(yt−1 , yt |x) ͷܭࢉ ্ه͔Βɺ
P(yt−1 , yt |x) = y0:t−2 yt+1:T +1 P(y|x) = y0:t−2 yt+1:T +1 t′ ψt′ (yt′−1 , yt′ ) = 1 Zx,w y0:t−2 yt+1:T +1 t′ ψt′ (yt′−1 , yt′ ) = ψt (yt−1 , yt ) Zx,w y0:t−2 yt+1:T +1 t̸=t′ ψt′ (yt′−1 , yt′ ) ߴ ढ़࡞ April 23, 2020 33 / 42 -
݅֬ (CRF) ֶश (ύϥϝʔλਪఆ) ∑ y0:t−2 ∑ yt+1:T +1 ∏
t̸=t′ ψt′ (yt′−1 , yt′ ) = ∑ y0:t−2 ∑ yt+1:T +1 (ψ1(y0, y1) × ψ2(y1, y2) × · · · × ψt−1(yt−2, yt−1) × ψt+1(yt, yt+1) × · · · × ψT +1(yT , yT +1)) = ∑ y0:t−2 ψ1(y0, y1) × ψ2(y1, y2) × · · · × ψt−1(yt−2, yt−1) ∑ yt+1:T +1 ψt+1(yt, yt+1) × · · · × ψT +1(yT , yT +1) = ∑ y0:t−2 ψ1(y0, y1) × ψ2(y1, y2) × · · · × ψt−1(yt−2, yt−1) ∑ yt+1:T +1 T +1 ∏ t′=t+1 ψt′ (yt′ , yt′−1 ) = ∑ y0:t−2 t−1 ∏ t′=1 ψt′ (yt′ , yt′−1 ) ∑ yt+1:T +1 T +1 ∏ t′=t+1 ψt′ (yt′ , yt′−1 ) = ∑ y0:t−2 t−1 ∏ t′=1 ψt′ (yt′ , yt′−1 ) ∑ yt+1:T +1 T +1 ∏ t′=t+1 ψt′ (yt′ , yt′−1 ) ߴ ढ़࡞ April 23, 2020 34 / 42 -
݅֬ (CRF) ֶश (ύϥϝʔλਪఆ) Αͬͯɺ α(yt, t) = y0:t−1 t
t′=1 ψt′ (yt′ , yt′−1), β(yt, t) = yt+1:T +1 T+1 t′=t+1 ψt′ (yt′ , yt′−1) ͱ͓͘ͱ (ͨͩ͠ɺα(y0, 0) = 1, β(yT+1, T + 1) = 1) P(yt−1, yt|x) = 1 Zx,w ψt(yt−1, yt)α(yt−1, t − 1)β(yt, t) ͱදͤΔɽ ߴ ढ़࡞ April 23, 2020 35 / 42 -
݅֬ (CRF) ֶश (ύϥϝʔλਪఆ) α(yt , t) ͷมܗ α(yt ,
t) = y0:t−1 t t′=1 ψt′ (yt′ , yt′−1 ) = y0:t−1 ψ1 (y0 , y1 ) × ψ2 (y1 , y2 ) × · · · × ψt (yt−1 , yt ) = yt−1 y0:t−2 ψ1 (y0 , y1 ) × ψ2 (y1 , y2 ) × · · · × ψt (yt−1 , yt ) = yt−1 ψt (yt−1 , yt ) y0:t−2 ψ1 (y0 , y1 ) × ψ2 (y1 , y2 ) × · · · × ψt (yt−3 , yt−2 ) × ψt (yt−2 , yt−1 ) = yt−1 ψt (yt−1 , yt ) y0:t−2 t−1 t′=1 ψt′ (yt′ , yt′−1 ) = yt−1 ψt (yt−1 , yt )α(yt−1 , t − 1) ߴ ढ़࡞ April 23, 2020 36 / 42 -
݅֬ (CRF) ֶश (ύϥϝʔλਪఆ) β(yt , t) ͷมܗ ಉ༷ʹͯ͠ɺ β(yt,
t) = y+1 ψt+1(yt, yt+1)β(yt+1, t + 1) ߴ ढ़࡞ April 23, 2020 37 / 42 -
݅֬ (CRF) ֶश (ύϥϝʔλਪఆ) ࣍ 1 ܥྻϥϕϦϯάͷఆࣜԽ 2 ӅΕϚϧίϑϞσϧ (HMM)
ͱͷؔ࿈ ੜϞσϧͱࣝผϞσϧ 3 HMM HMM ͷ 4 ݅֬ (CRF) ఆࣜԽ ༧ଌ (ਪఆ) ֶश (ύϥϝʔλਪఆ) ∑ y∈Y ϕ(x, y)P(y|x) ͷܭࢉ P(yt−1 , yt |x) ͷܭࢉ forward-backword algorithm Zx,w ͷܭࢉ ߴ ढ़࡞ April 23, 2020 38 / 42 -
݅֬ (CRF) ֶश (ύϥϝʔλਪఆ) 1: function Forward(x) 2: α(BOS, 0)
= 1 3: for t = 1, . . . , T do 4: for y ∈ y1, . . . , ym do 5: if t = 1 then 6: α(y, t) = α(BOS, t − 1) ∗ ψt(y, BOS) 7: else 8: α(y, t) = 0 9: for y′ ∈ y1, . . . , ym do 10: α(y, t) = α(y, t) + α(y′, t − 1) ∗ ψt(y, y′) ߴ ढ़࡞ April 23, 2020 39 / 42 -
݅֬ (CRF) ֶश (ύϥϝʔλਪఆ) 1: function Backward(x) 2: β(EOS, T
+ 1) = 1 3: for t = T, . . . , 1 do 4: for y ∈ y1, . . . , ym do 5: if t = T then 6: β(y, t) = β(EOS, T + 1) ∗ ψt+1(y, EOS) 7: else 8: β(y, t) = 0 9: for y′ ∈ y1, . . . , ym do 10: β(y, t) = β(y, t) + β(y′, t + 1) ∗ ψt+1(y, y′) ߴ ढ़࡞ April 23, 2020 40 / 42 -
݅֬ (CRF) ֶश (ύϥϝʔλਪఆ) ࣍ 1 ܥྻϥϕϦϯάͷఆࣜԽ 2 ӅΕϚϧίϑϞσϧ (HMM)
ͱͷؔ࿈ ੜϞσϧͱࣝผϞσϧ 3 HMM HMM ͷ 4 ݅֬ (CRF) ఆࣜԽ ༧ଌ (ਪఆ) ֶश (ύϥϝʔλਪఆ) ∑ y∈Y ϕ(x, y)P(y|x) ͷܭࢉ P(yt−1 , yt |x) ͷܭࢉ forward-backword algorithm Zx,w ͷܭࢉ ߴ ढ़࡞ April 23, 2020 41 / 42 -
݅֬ (CRF) ֶश (ύϥϝʔλਪఆ) Zx,w ͷܭࢉ Zx,w = y∈Y exp(w
· ϕ(x, y)) = y∈Y exp(w · t ϕ(x, yt, yt−1)) = y∈Y t exp(w · ϕ(x, yt, yt−1)) = y∈Y t ψt(yt, yt−1) = yT y0:T −1 t ψt(yt, yt−1) = yT α(yT , T) ߴ ढ़࡞ April 23, 2020 42 / 42