Share
Embed
Sutton "Reinforcement Learning" 2nd Edition Ch6: TD-learning
Kosuke Miyoshi
May 09, 2019
Transcript
32 slides
-
࣍ w 1PMJDZ&WBMVBUJPO w Кʹର͢Δ7ͷ1SFEJDUJPO w 5%ͱ.$ %1ͷڞ௨ͱҧ͍ w $POUSPM"MHPSJUINT
w 0O1PMJDZ4BSTB w 0⒎1PMJDZ2MFBSOJOH w &YQFDUFE4BSTB w .BYJNJ[BUJPO#JBT w %PVCMF-FBSOJOH -
5FNQPSBM%J⒎FSFODF -FBSOJOH w %ZOBNJD1SPHSBNNJOH w #PPUTUSBQQJOH 7 4U
ͷਪఆΛͬͯ7 4U ͷਪఆΛߋ৽ w ڥͷ.PEFM Λ͏ w .POUF$BSMP w ใुͷ4BNQMJOH 5%-FBSOJOH P(St+1 , Rt+1 |St , At ) -
%ZOBNJD1SPHSBNNJOH T T T T T T T T T
T T T T V(S t ) ← E π R t+1 + γV(S t+1 ) [ ] S t = X a ⇡(a|St) X s0,r p(s0, r|St, a)[r + V (s0)] r a s0 http://incompleteideas.net/609%20dropbox/slides%20(pdf%20and%20keynote)/11-12-TD.pdf -
.POUF$BSMP T T T T T T T T T
T T T T T T T T T T T V(S t ) ←V(S t )+α G t −V(S t ) [ ] St http://incompleteideas.net/609%20dropbox/slides%20(pdf%20and%20keynote)/11-12-TD.pdf -
5%MFBSOJOH T T T T T T T T T
T T T T T T T T T T T V(S t ) ←V(S t )+α R t+1 + γV(S t+1 )−V(S t ) [ ] S t R t+1 S t+1 http://incompleteideas.net/609%20dropbox/slides%20(pdf%20and%20keynote)/11-12-TD.pdf -
V (St) ← V (St) + α [Gt − V
(St)] V (St) ← V (St) + α [Rt+1 + γV (St+1) − V (St)] .POUF$BSMP $POTUBOUЋ.$ 5% &QJTPEFऴྃ࣌ʹΘ͔Δ ใु ใुͷਪఆ 4UFQ࣮ߦޙʹ͙͢ग़ͤΔ 5%ޡࠩ -
vπ (s) ≐ π [Gt |St = s] = π
[Rt+1 + γGt+1 |St = s] = π [Rt+1 + γvπ (St+1)|St = s] .$͜ΕͷਪఆΛλʔήοτʹ 5%͜ΕͷਪఆΛλʔήοτʹ -
5%ͷ༏Ґੑ w ڥͷ.PEFM ͕ෆཁ w &QJTPEFऴྃ·Ͱͨͳͯ͘ߋ৽͕Մೳ w ಛఆͷ݅ԼͰ 5%.$ऩଋ͕อূ͞Ε͍ͯΔ
w Ұൠతʹ.$ΑΓऩଋ͕ૣ͍ P(St+1 , Rt+1 |St , At ) -
3BOEPN8BML w $͔Βελʔτ w ֤ঢ়ଶͰࠨӈʹͷ֬ͰҠಈ .BSLPW3FXBSE1SPDFTT w &ͰӈʹҠಈͨ͠Β3FXBSEͰऴྃ w
"ͰࠨʹҠಈͨ͠Β3FXBSEͰऴྃ w Ѝͱ͢Δͱ"d&ͷਅͷঢ়ଶՁ7 1 6 , 2 6 , 3 6 , 4 6 , 5 6 -
#BUDI5SBJOJOH w ͜ΕΒͷ6QEBUFΛຖ4UFQߦ͏ͷͰͳ͘ɺ૿ՃΛ 4UBUFຖʹஷΊ͓͍ͯͯɺશ&QJTPEFΛ·ͱΊͯ7 4U ߋ৽ w 7 4U ͷมԽ͕ͳ͘ͳΔ·Ͱ܁Γฦ͢
w .$5%#BUDI6QEBUFͰܾఆతʹऩଋ͢Δ͕ ऩଋ͢Δ.$ͱ5%ͰҟͳΔ V (St) ← V (St) + α [Gt − V (St)] .POUF$BSMP 5% V (St) ← V (St) + α [Rt+1 + γV (St+1) − V (St)] -
:PVBSFUIFQSFEJDUPS Ѝͱͨ࣌͠ɺ7 " 7 # ʁ A, 0, B, 0
B, 1 B, 1 B, 1 B, 1 B, 1 B, 1 B, 0 ҎԼͷΤϐιʔυͷ݁Ռ͕͋ͬͨͱ͢Δ -
A, 0, B, 0 B, 1 B, 1 B, 1
B, 1 B, 1 B, 1 B, 0 ݟํ ɾ#ճग़͖ͯͯճ3FXBSE ɾ"ճग़͖ͯͯશͯใु V(B) = 3 4 V(A) = 0 ݟํ ɾ#ճग़͖ͯͯճ3FXBSE ɾ"ͷ֬Ͱ#ʹભҠͯ͠3FXBSE Αͬͯ7 " 7 # V(B) = 3 4 V(A) = 3 4 #BUDI.$ ͷ݁Ռ #BUDI5% ͷ݁Ռ -
w 5SBJOJOH%BUBͷ.FBO4RVBSF&SSPSΛ࠷খԽ͢Δͷ͕ 7 " w ͜Ε͕#BUDI.$ͷग़͢ղ w ͷ4FRVFOUJBMJUZΛߟྀ͢Δͱ7 "
w σʔλΛੜ͢ΔϚϧίϑϞσϧͷ࠷ਪఆ w ਪఆͨ͠Ϟσϧ͕ਖ਼͚͠Ε7ͷਪఆਖ਼͍͕͠ ࢉग़Ͱ͖Δ w DFSUBJOUZFRVJWBMFODFFTUJNBUFͱݺΕΔ w ͜Ε͕#BUDI5%ͷग़͢ղ -
4BSTB 0OQPMJDZ5%$POUSPM w ࣮ࡍʹ࣍4UBUFͰऔͬͨ"DUJPO "U ͷ2Λར༻ͯ͠2 4U "U Λߋ৽͢Δ
w #FIBWJPS1PMJDZКͷRКͷਪఆΛ͍ͯ͠ΔͷͰ0OQPMJDZ Q (St , At) ← Q (St , At) + α [Rt+1 + γQ (St+1 , At+1) − Q (St , At)] -
2-FBSOJOH 0⒎QPMJDZ5%$POUSPM w ࣮ࡍͷ"DUJPO "U Λൃߦ͢Δ#FIBWJPS1PMJDZЏHSFFEZͳͲͷ ୳ࡧΛؚΜͩ1PMJDZ w 2ͷਪఆΛ͢Δ5BSHFU1PMJDZ(SFFEZͳ1PMJDZ Q
(St , At) ← Q (St , At) + α [Rt+1 + γ max a Q (St+1 , a) − Q (St , At)] -
&YQFDUFE4BSTB w 2MFBSJOHͰ࠷େΛͱ͍ͬͯͨͱ͜ΖΛɺݱࡏͷ1PMJDZͰͱΔ"DUJPOͷ ֬Λར༻ͨ͠ظʹมߋͨ͠ͷ w 4BSTBʹൺͯܭࢉྔ͕૿͑ΔΘΓʹ"U ͷબʹىҼ͢Δ2ਪఆͷ ࢄΛͳ͘͢͜ͱ͕Ͱ͖Δ w 4U
͕༩͑ΒΕͨ࣌ɺ4BSTBʹ͓͚Δ ЏHSFFEZͳͲͷ"DUJPOબʹΑΓ 2͕֬తʹมԽ͢Δํͷظ &YQFDUBUJPO ͷํʹܾఆతʹมԽ͢ΔͨΊɺ &YQFDUFE4BSTBͱݺͿ w Ұൠతʹ4BSTBΑΓྑ͍݁ՌΛग़͢ Q (St , At) ← Q (St , At) + α [Rt+1 + γπ [Q (St+1 , At+1)|St+1] − Q (St , At)] ← Q (St , At) + α [ Rt+1 + γ∑ a π (a|St+1) Q (St+1 , a) − Q (St , At) ] -
&YQFDUFE4BSTB w ্ͷྫͰظΛऔΔК#FIBWJPSͱಉ͡ͷͳͷͰ 0O1PMJDZ͕ͩҟͳΔͷΛར༻ͨ͠0⒎1PMJDZͰߏΘͳ͍ w ͜͜Λ(SFFEZʹม͑ͨͷ͕2MFBSOJOHͱͳΔ w 2MFBSOJOH&YQFDUFE4BSTBͷҰྫͱݟΔ͜ͱ͕Ͱ͖Δ Q (St
, At) ← Q (St , At) + α [Rt+1 + γπ [Q (St+1 , At+1)|St+1] − Q (St , At)] ← Q (St , At) + α [ Rt+1 + γ∑ a π (a|St+1) Q (St+1 , a) − Q (St , At) ] -
w "͔Βελʔτ w "͔Βӈಈ͘ͱ w 3FXBSEͰऴྃ w "͔Βࠨಈ͘ͱ w 3FBSEͰ#ભҠ
w #ʹͯଟͷ"DUJPO w Ͳͷ"DUJPOΛબΜͰฏۉࢄͷ 3FXBSEΛಘͯऴྃ w ࠨΛબΜͩ߹ͷظใु w ӈΛબΜͩํ͕ಘͳͷʹ.BYJNJ[BUJPO#JBT ͷӨڹ͕͋ΔͱࠨΛબͼ͘͢ͳͬͯ͠·͏ Џͷ2-FBSOJOH ࠷దͳ߹ͷࠨΛબͿ֬ -
%PVCMF2MFBSOJOH Q1 (St , At) ← Q1 (St , At)
+ α Rt+1 + γQ2 ( St+1 , arg max a Q1 (St+1 , a) ) − Q1 (St , At) Q2 (St , At) ← Q2 (St , At) + α Rt+1 + γQ1 ( St+1 , arg max a Q2 (St+1 , a) ) − Q2 (St , At) ϥϯμϜʹ બ 2ςʔϒϧΛ2ͱ2ͷೋͭ༻ҙ ࠷େͷJOEFY2͔ΒಘΔ͕ 2͔ΒΛऔ͖ͬͯͯར༻͢Δ -
·ͱΊ w 5%-FBSOJOHͷհ w %1ͷ#PPUTUSBQ .$ͷ4BNQMJOHΛऔΓೖΕ͍ͯΔ w ܭࢉ࣌ʹ.$%1Ͱ༷͋ͬͨͳোน͕গͳ͍ w $POUSPM"MHPSJUINT
w 0O1PMJDZ4BSTB w 0⒎1PMJDZ2MFBSOJOH w &YQFDUFE4BSTBͷհ w %PVCMF-FBSOJOHʹΑΔ.BYJNJ[BUJPO#JBTͷճආ