Share
Embed
Sutton "Reinforcement Learning" 2nd Edition Ch13: Policy Gradient Methods
"Reinforcement Learning" by Richard S. Sutton 2nd edition
Chapter 13
Policy Gradient Methods
Kosuke Miyoshi
October 02, 2019
Transcript
24 slides
-
1PMJDZ(SBEJFOU.FUIPET w ͜Ε·Ͱͷষ w "DUJPOWBMVFΛֶश͠"DUJPOWBMVFʹج͍ͮͯ "DUJPOΛબ BDUJPOWBMVFNFUIPET w ࠓճͷষ
w QPMJDZΛQBSBNFUFSͰදݱ QBSBNFUFSJ[FEQPMJDZ w "DUJPOબʹWBMVFGVODUJPOΛར༻͠ͳ͍ QPMJDZͷֶशࣗମʹ͏߹͋Δ w QPMJDZΛදݱ͢ΔQBSBNFUFSʹରͯ͠HSBEJFOUBTDFOU -
1BUBNFUFSJ[FEQPMJDZ θt+1 = θt + α ̂ ∇J (θt) π(a|s,
θ) = Pr {At = a|St = s, θt = θ} ֬తͳ1PMJDZΛE`࣍ݩͷύϥϝʔλВͰද͢ ϙϦγʔΠͷྑ͠ѱ͠Λஅ͢ΔͨΊͷ 1FSGPSNBODFNFBTVSF+ В εΧϥʔ Λಋೖɻ +͕࠷େԽͷର +ͷВʹର͢Δޯͷ֬తͳਪఆͷظ Вͱಉ͘͡E`࣍ݩ -
1FSGPSNBODF.FBTVSF + ϙϦγʔΠͷྑ͠ѱ͠ͷஅج४ w &QJTPEJDDBTF w +ΠʹԊͬͨ࣌ͷ։࢝4UBUF 4 ͷঢ়ଶՁ w
$POUJOVJOHDBTF w +BWFSBHFSFXBSESBUF -
1PMJDZ"QQSPYJNBUJPOBOE JUT"EWBOUBHFT w "DUJPOWBMVFGVODUJPOͷܽ w ֬తͳϙϦγʔΛࣗવʹදݱ͢Δํ๏͕ແ͍ w ΧʔυήʔϜͷ༷ͳෆશใήʔϜͰ࠷దϙϦγʔ͕֬తʹͳΔ߹͕ଟ͍ FYϙʔΧʔͰͷϒϥϑ
w 1BSBNFUFSJ[FEQPMJDZͷॴ w ֬తͳϙϦγʔΛදݱͰ͖Δ w ࿈ଓ"DUJPOʹରԠͰ͖Δ w 1PMJDZͷมߋ͕εϜʔζ w 1PMJDZͷۙࣅදݱ͕WBMVFGVODJUPOΛ͏߹ΑΓ1PMJDZදݱͷํ͕ γϯϓϧʹͳΔΑ͏ͳͷ߹༗ޮ ςτϦεͳͲ w 1PMJDZʹରͯ͠ࣄલใΛΈࠐΈ͍͢ -
4PGUNBYJOBDUJPO QSFGFSFODFT π(a|s, θ) ≐ eh(s,a,θ) ∑ b eh(s,b,θ) h(s,
a, θ) = θ⊤x(s, a) GFBUVSFWFDUPS ύϥϝʔλВΛઢܗͰදݱ͢Δ߹ -
1PMJDZ(SBEJFOU5IFPSFN J(θ) ≐ vπθ (s0) ∇J(θ) ∝ ∑ s μ(s)∑
a qπ (s, a)∇π(a|s, θ) &QJTPEJDͳ߹Ͱͷ1FSGPSNBODFNFBTVSF ΠʹԊͬͨͱ͖ʹঢ়ଶTΛͱΔ֬ 1FSGPSNBODFNFBTVSF ɾBDUJPOબ ɾ๚ΕΔTͷසЖ T ʹґଘ͠ɺͦΕΒϙϦγʔͷύϥϝʔλВͷӨڹΛड͚Δɻ ͔͠͠1FSGPSNBODF.FBTVSF+ͷВʹର͢Δޯ㲆+ Tͷʹର͢ΔඍΛؚ·ͳ͍ܗͰهड़͞ΕΔ -
3&*/'03$& .POUF$BMSP1PMJDZ(SBEJFOU ∇J(θ) ∝ ∑ s μ(s)∑ a qπ (s,
a)∇π(a|s, θ) = π [∑ a qπ (St , a)∇π (a|St , θ) ] θt+1 ≐ θt + α∑ a ̂ q (St , a, w)∇π (a|St , θ) 1PMJDZHSBEJFOUUIFPSFN ͜͜ͰશBDUJPOʹର͢Δͷܗʹͳ͍ͬͯΔ ͜ΕΛBDUJPOͰͷ.POUF$BSMPαϯϓϦϯάͰஔ͖͑Δ͜ͱΛߟ͑Δ -
3&*/'03$& .POUF$BMSP1PMJDZ(SBEJFOU ∇J(θ) = π [∑ a π (a|St ,
θ) qπ (St , a) ∇π (a|St , θ) π (a|St , θ) ] = π [ qπ (St , At) ∇π (At |St , θ) π (At |St , θ) ] = π [ Gt ∇π (At |St , θ) π (At |St , θ) ] , θt+1 ≐ θt + αGt ∇π (At |St , θt) π (At |St , θt) ∇ln x = ∇x x = θt + αGt ∇ln π (At |St , θt) Λར༻ R 4 " Λใु(ʹ͓͖͔͑ શBDUJPOBͰͷΛɺ࣮ࡍʹऔͬͨ"DUJPO"Uʹ ͓͖͔͑ αϯϓϦϯά Вͷߋ৽ࣜ -
4IPSUDPSSJEPS w 4͔Β։࢝ (ʹ౸ୡ͢Δͱ5FSNJOBUF w ঢ়ଶۙࣅΛ͢Δͱશ4UBUF͕ಉ͡ঢ়ଶͱͯ͠ݟ͑Δ w TUBUFͷ࣌ɺӈBDUJPOͰࠨɺࠨBDUJPOͰӈҠಈ ͦΕҎ֎ӈBDUJPOӈɺࠨBDUJPOࠨ w
TUFQಈͨ͘ͼʹSFXBSE w ࠷దͳQPMJDZ֬తQPMJDZ ֬Ͱӈ -
3&*/'03$&XJUI#BTFMJOF ∇J(θ) ∝ ∑ s μ(s)∑ a (qπ (s, a)
− b(s))∇π(a|s, θ) ∑ a b(s)∇π(a|s, θ) = b(s)∇∑ a π(a|s, θ) = b(s)∇1 = 0 θt+1 ≐ θt + α (Gt − b (St)) ∇π (At |St , θt) π (At |St , θt) CBTFMJOFBʹґଘ͠ͳ͍ Вͱؔͳ͍ ͷΛબɺ ԼهͷʹͳΓޯਪఆͷظʹӨڹ͠ͳ͍ CBTFMJOFΛಋೖͯ͠ɺ+ͷޯ㲆+ͷਪఆͷWBSJBODFΛԼ͛Δ͜ͱΛߟ͑Δ CBTFMJOFTUBUFʹґଘ͢Δ͖ ྫશBDUJPOͷWBMVF͕ߴ͍߹CBTFMJOFߴ͍ํ͕֤BDUJPOͷࠩผԽ͕Ͱ͖Δ CBTMJOFͷࣗવͳબঢ়ଶՁ7 Вͷߋ৽ࣜ -
"DUPS$SJUJD.FUIPET θt+1 ≐ θt + α (Gt:t+1 − ̂ v
(St , w)) ∇π (At |St , θt) π (At |St , θt) = θt + α (Rt+1 + γ ̂ v (St+1 , w) − ̂ v (St , w)) ∇π (At |St , θt) π (At |St , θt) = θt + αδt ∇π (At |St , θt) π (At |St , θt) 7ΛCBTFMJOFͱͯ͠ར༻ͨ͠3&*/'03$&ɺ#PPUTUSBQQJOH͍ͯ͠ͳ͍ #PPUTUSBQQJOHΛར༻ͯ͠CJBTΛೖΕͨํ͕WBSJBODF͕ݮֶͬͯशૣ͘ͳΔ &QJTPEFऴྃ·ͰͷใुΛ TUFQͷCPPUTUSBQQJOHͰஔ͖͑ -
$PJOUJOVJOH1SPCMFNT J(θ) ≐ r(π) ≐ lim h→∞ 1 h h
∑ t=1 [Rt |S0 , A0:t−1 ∼ π] = lim t→∞ [Rt |S0 , A0:t−1 ∼ π] = ∑ s μ(s)∑ a π(a|s)∑ s′,r p (s′, r|s, a) r Gt ≐ Rt+1 − r(π) + Rt+2 − r(π) + Rt+3 − r(π) + ⋯ ࿈ଓΤϐιʔυͷ߹ɺQFSGPSNBODFNFBTVSF ΠʹԊͬͨ࣌ͷTUFQ͋ͨΓͷSFXBSEฏۉ vπ (s) ≐ π [Gt |St = s] qπ (s, a) ≐ π [Gt |St = s, At = a] ∇J(θ) = ∑ s μ(s)∑ a qπ (s, a)∇π(a|s, θ) ͱఆٛ͢Δͱ&QJTPEJDͳ߹ͱಉ༷ͷ1PMJDZHSBEJFOUUIFPMFN͕ಋग़͞ΕΔ -
$POUJOVPVT"DUJPOT π(a|s, θ) ≐ 1 σ(s, θ) 2π exp (
− (a − μ(s, θ))2 2σ(s, θ)2 ) μ(s, θ) ≐ θ⊤ μ xμ (s) σ(s, θ) ≐ exp (θ⊤ σ xσ (s)) ࿈ଓ"DUJPOΛߟ͑Δ θ = [θμ , θσ] ⊤ -
4VNNBSZ w 1PMJDZ(SBEJFOUͰߦಈՁؔਪఆΛར༻ͤͣʹQBSBNFUSJ[F͞Εͨ1PMJDZΛֶश w 1BSBNFUFSJ[F͞Εͨ1PMJDZͷར w ֬తͳදݱ୳ࡧܾఆతͳؒΛۙతʹදݱՄೳ w 1PMJDZHSBEJFOUUIFPSFN w
ঢ়ଶͷඍΛؚ·ͳ͍ܗͰ1FSGPSNBODF.FBTVSF͕1PMJDZύϥϝʔλʹ Ͳ͏ӨڹΛड͚Δ͔͕هड़Ͱ͖Δ w 3&*/03$& w .POUF$BSMP w ঢ়ଶՁΛ#BTFMJOFͱͯ͠ಋೖ͠ɺCJBTΛ૿ͣ͞ʹWBSJBODFΛԼ͛Δ w "DUPS$SJUJD w ঢ়ଶՁΛCPPUTUSBQQJOHʹར༻ͯ͠5%ֶश.$ΑΓWBSJBODF w ঢ়ଶՁʹΑΓ1PMJDZͷBDUJPOબʹରͯ͠DSFEJUBTTJHO DSJUJTJ[F -
1PMJDZ(SBEJFOU5IFPMFNͷূ໌ &QJTPEJDDBTF Pr(s → x, k, π) TUBUFT͔ΒLTUFQޙʹTUBUFYʹͳΔ֬ ∇vπ (s)
= ∇ [∑ a π(a|s)qπ (s, a) ] = ∑ a [∇π(a|s)qπ (s, a) + π(a|s)∇qπ (s, a)] = ∑ a ∇π(a|s)qπ (s, a) + π(a|s)∇∑ s′,r p (s′, r|s, a) (r + vπ (s′)) = ∑ a [ ∇π(a|s)qπ (s, a) + π(a|s)∑ s′ p (s′|s, a)∇vπ (s′) ] = ∑ a [ ∇π(a|s)qπ (s, a) + π(a|s)∑ s′ p (s′|s, a) ∑ a′ [ ∇π (a′|s′) qπ (s′, a′) + π (a′|s′)∑ s′′ p (s′′|s′, a′)∇vπ (s′′) ] = ∑ x∈ ∞ ∑ k=0 Pr(s → x, k, π)∑ a ∇π(a|x)qπ (x, a) ੵͷඍ RΛղ SͱQ T` ScT B Вʹґଘ͠ͳ͍ T`ʹؔͯ͠࠶ؼతʹల։ -
∇J(θ) = ∇vπ (s0) = ∑ s ( ∞ ∑
k=0 Pr (s0 → s, k, π) )∑ a ∇π(a|s)qπ (s, a) = ∑ s η(s)∑ a ∇π(a|s)qπ (s, a) = ∑ s′ η (s′)∑ s η(s) ∑ s′ η (s′) ∑ a ∇π(a|s)qπ (s, a) = ∑ s′ η (s′)∑ s μ(s)∑ a ∇π(a|s)qπ (s, a) ∝ ∑ s μ(s)∑ a ∇π(a|s)qπ (s, a) 1FSGPSNBODFNFBTVSFΛٻΊΔͨΊʹTʹTΛೖΕΔ T͕ग़ͯ͘Δճ η(s) : T͕ग़ͯ͘Δ֬ μ(s) : ূ໌ऴΘΓ