Upgrade to Pro — share decks privately, control downloads, hide ads and more …

bellman方程式の導出.pdf

Avatar for m_nshr m_nshr
February 16, 2019
1
1.2k

 bellman方程式の導出.pdf

Deriving Bellman equations

Avatar for m_nshr

m_nshr

February 16, 2019
Tweet

Featured

See All Featured
The Power of CSS Pseudo Elements
Avatar for Geoffrey Crofte geoffreycrofte
79
6k
Code Reviewing Like a Champion
Avatar for maltzj maltzj
526
40k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
Avatar for Morgane Peng morganepeng
132
19k
Producing Creativity
Avatar for Steve Smith orderedlist
PRO
347
40k
How to train your dragon (web standard)
Avatar for Monica Dinculescu notwaldorf
97
6.3k
Measuring & Analyzing Core Web Vitals
Avatar for Philip Tellis bluesmoon
9
620
Speed Design
Avatar for Sergey Chernyshev sergeychernyshev
32
1.2k
Understanding Cognitive Biases in Performance Measurement
Avatar for Philip Tellis bluesmoon
31
2.7k
Making the Leap to Tech Lead
Avatar for Ryan Cromwell cromwellryan
135
9.6k
Stop Working from a Prison Cell
Avatar for Chris Michel hatefulcrawdad
271
21k
Thoughts on Productivity
Avatar for Jon Yablonski jonyablonski
70
4.9k
Building Adaptive Systems
Avatar for Chris Keathley keathley
44
2.8k

Transcript

  1. #FMMNBOํఔࣜͷಋग़ .BUI$PEJOHڧԽֶशΛޠΖ͏ 

  2. ໨࣍ ͸͡Ίʹ Ձ஋ؔ਺ͷಋೖ Ձ஋ؔ਺ͷఆ͔ٛΒ#FMMNBOํఔࣜΛಋग़ #BDLVQEJBHSBN͔Β#FMMNBOํఔࣜΛΈΔ ࠷దՁ஋ؔ਺ͷ঺հ #BDLVQEJBHSBN͔Β࠷దՁ஋؍਺ΛΈΔ ·ͱΊ

  3. ڧԽֶशͷ໨తͱ͸ 5IBUBMMPGXIBUXFNFBOCZHPBMTBOEQVSQPTFTDBOCF XFMMUIPVHIUPGBTNBYJNJ[BUJPOPGUIFFYQFDUFEWBMVF PGUIFDVNVMBUJWFTVNPGBSFDFJWFETDBMBSTJHOBM DBMMFESFXBSE 4VUUPO#BSUP  3FXBSE)ZQPUIFTJT ྦྷੵใुͷ࠷େԽ

  4. Gt ≐ Rt+1 + γ Rt+2 + γ2 Rt+3 +

    ⋯ = ∞ ∑ k=0 γkRt+1+k = Rt+1 + γ ∞ ∑ k=0 γkRt+1+k+1 = Rt+1 + γGt+1 ˡʹண໨ S0 π A0 → R1 → S1 π A1 → R2 → S2 π ⋯ ྦྷੵใु ऩӹ γ ∈ [0,1]

  5. ϕϧϚϯ࠷దੑͷݪཧ ࠶ؼతߏ଄Λ࣋ͭ໰୊ ࠷దͳํࡦ͸ɺॳظঢ়ଶͱॳظܾఆ͕ͲΜͳ΋ͷͰ͋Εɺͦͷ݁Ռಘ ΒΕΔ࣍ͷঢ়ଶʹؔͯ͠ɺҎ߱ͷܾఆ͕ඞͣ࠷దํࡦʹͳ͍ͬͯΔͱ ͍͏ੑ࣭Λ΋ͭɻ ࢀরɿɹ#FMMNBO ɺ$IBQ*** 1SJODJQMFPG0QUJNBMJUZ ಈతܭը๏ %1

    Ͱղ͚Δ ͔΋͠Εͳ͍

  6. ໨࣍ ͸͡Ίʹ Ձ஋ؔ਺ͷಋೖ Ձ஋ؔ਺ͷఆ͔ٛΒ#FMMNBOํఔࣜΛಋग़ #BDLVQEJBHSBN͔Β#FMMNBOํఔࣜΛΈΔ ࠷దՁ஋ؔ਺ͷ঺հ #BDLVQEJBHSBN͔Β࠷దՁ஋؍਺ΛΈΔ ·ͱΊ

  7. ه߸ͷ४උ ํࡦ ঢ়ଶભҠ֬཰ π(a|s) ≐ (At = a|St = s)

    p(s′|s, a) ≐ (St+1 = s′|St = s, At = a) r(s, a, s′) ≐ [Rt+1 |St = s, At = a, St+1 = s′] ˠঢ়ଶTͰߦಈBΛબ୒͢Δ֬཰ ˠঢ়ଶTͰߦಈBΛબ୒ͯ࣍͠ঢ়ଶT`ʹભҠ͢Δ֬཰ ˠঢ়ଶTͰߦಈBΛબ୒ͯ࣍͠ঢ়ଶT`ʹભҠͨ͠৔߹ͷଈ࣌ใुͷظ଴஋ ؀ڥͷμΠφϛΫε FOWJSPONFOU`TEZOBNJDT ଈ࣌ใु ใुؔ਺

  8. Ձ஋ؔ਺ zঢ়ଶzՁ஋ؔ਺ lঢ়ଶɾߦಈzՁ஋ؔ਺ Vπ(s) ≐ π [Gt |St = s]

    Qπ(s, a) ≐ π [Gt |St = s, At = a] ྆ऀͷؔ܎ Vπ(s) ≐ π [Gt |St = s] = ∑ a π(a|s)π [Gt |St = s, At = a] = ∑ a π(a|s)Qπ(s, a) ˡ݁Ռ

  9. ໨࣍ ͸͡Ίʹ Ձ஋ؔ਺ͷಋೖ Ձ஋ؔ਺ͷఆ͔ٛΒ#FMMNBOํఔࣜΛಋग़ #BDLVQEJBHSBN͔Β#FMMNBOํఔࣜΛΈΔ ࠷దՁ஋ؔ਺ͷ঺հ #BDLVQEJBHSBN͔Β࠷దՁ஋؍਺ΛΈΔ ·ͱΊ

  10. Vπ(s) ≐ π [Gt |St = s] = π [Rt+1

    + γGt+1 |St = s] = ∑ a π(a|s)∑ s′ p(s′|s, a)π [Rt+1 + γGt+1 |St = s, At = a, St+1 = s′] = ∑ a π(a|s)∑ s′ p(s′|s, a)(r(s, a, s′) + γπ [Gt+1 |St = s, At = a, St+1 = s′]) = ∑ a π(a|s)∑ s′ p(s′|s, a)(r(s, a, s′) + γVπ(s′)) ˡ݁Ռ #FMMNBOํఔࣜGPS Vπ(s)

  11. Qπ(s, a) ≐ π [Gt |St = s, At =

    a] = π [Rt+1 + γGt+1 |St = s, At = a] = ∑ s′ p(s′|s, a)π [Rt+1 + γGt+1 |St = s, At = a, St+1 = s′] = ∑ s′ p(s′|s, a)(r(s, a, s′) + γπ [Gt+1 |St = s, At = a, St+1 = s′]) = ∑ s′ p(s′|s, a)(r(s, a, s′) + γVπ(s′)) = ∑ s′ p(s′|s, a)(r(s, a, s′) + γ∑ a′ π(a′|s′)Qπ(s′, a′)) #FMMNBOํఔࣜGPS Qπ(s, a) ˡ݁Ռ ˣ݁ՌΛ୅ೖ ˡ݁Ռ

  12. ໨࣍ ͸͡Ίʹ Ձ஋ؔ਺ͷಋೖ Ձ஋ؔ਺ͷఆ͔ٛΒ#FMMNBOํఔࣜΛಋग़ #BDLVQEJBHSBN͔Β#FMMNBOํఔࣜΛΈΔ ࠷దՁ஋ؔ਺ͷ঺հ #BDLVQEJBHSBN͔Β࠷దՁ஋؍਺ΛΈΔ ·ͱΊ

  13. #BDLVQEJBHSBN ɾঢ়ଶͱߦಈͷܥྻΛਤͰද͢ ɾ˓͸ঢ়ଶɺ˔͸ߦಈ ·ͨ͸ঢ়ଶɾߦಈର Λද͢ ɾϧʔτϊʔυͷՁ஋Λܭࢉ͢Δ࣌ʹ࢖͏ ɾϧʔτϊʔυ Ұ൪্ͷϊʔυ ͷՁ஋͕ ͲΜͳཁૉ͔Β੒Γཱ͍ͬͯΔ͔Λදݱ͠

    ͍ͯΔ

  14. #BDLVQEJBHSBNͰ #FMMNBOํఔࣜΛ֬ೝ Vπ(s) = ∑ a π(a|s)Qπ(s, a) Qπ(s, a)

    = ∑ s′ p(s′|s, a)(r(s, a, s′) + γVπ(s′)) π(a|s) Vπ(s) s a1 a2 Qπ(s, a1 ) Qπ(s, a2 ) Qπ(s, a) p(s′|s, a) (s, a) r(s, a, s′1 ) s′1 s′2 r(s, a, s′2 ) ˠ݁Ռ ࠶ Vπ(s′1 ) Vπ(s′2 ) ˠ݁Ռ ࠶

  15. #BDLVQEJBHSBNͰ #FMMNBOํఔࣜΛ֬ೝ Vπ(s) = ∑ a π(a|s)Qπ(s, a) = ∑

    a π(a|s)∑ s′ p(s′|s, a)(r(s, a, s′) + γVπ(s′)) Qπ(s, a) = ∑ s′ p(s′|s, a)(r(s, a, s′) + γVπ(s′)) = ∑ s′ p(s′|s, a)(r(s, a, s′) + γ∑ a′ π(a′|s′)Qπ(s′, a′)) Vπ(s) Vπ(s′) Qπ(s, a) Qπ(s, a) Qπ(s′, a′) π ˠ݁Ռ ࠶ p p r(s, a, s′) π(a|s) Vπ(s′) r(s, a, s′) ˣ݁ՌΛ୅ೖ ˠ݁Ռ ࠶

  16. ໨࣍ ͸͡Ίʹ Ձ஋ؔ਺ͷಋೖ Ձ஋ؔ਺ͷఆ͔ٛΒ#FMMNBOํఔࣜΛಋग़ #BDLVQEJBHSBN͔Β#FMMNBOํఔࣜΛΈΔ ࠷దՁ஋ؔ਺ͷ঺հ #BDLVQEJBHSBN͔Β࠷దՁ஋؍਺ΛΈΔ ·ͱΊ

  17. ࠷దՁ஋ؔ਺ V*(s) = max π Vπ(s) for any Q*(s, a)

    = max π Qπ(s, a) for any s ∈ s ∈ , a ∈ ɾ͜ͷؔ܎Λຬͨ͢ ͕গͳ͘ͱ΋ͭଘࡏ͢Δ ࠷దํࡦ ɾ͜ͷ ʹΑͬͯɺऩӹͷ࠷େԽ͕ୡ੒͞ΕΔ π π

  18. ໨࣍ ͸͡Ίʹ Ձ஋ؔ਺ͷಋೖ Ձ஋ؔ਺ͷఆ͔ٛΒ#FMMNBOํఔࣜΛಋग़ #BDLVQEJBHSBN͔Β#FMMNBOํఔࣜΛΈΔ ࠷దՁ஋ؔ਺ͷ঺հ #BDLVQEJBHSBN͔Β࠷దՁ஋؍਺ΛΈΔ ·ͱΊ

  19. #BDLVQEJBHSBNͰ #FMMNBO࠷దํఔࣜΛ֬ೝ V*(s) = ∑ a π(a|s) max a Q*(s,

    a) Q*(s, a) = ∑ s′ p(s′|s, a)(r(s, a, s′) + γV*(s′)) V*(s) = max a ∑ s′ p(s′|s, a)(r(s, a, s′) + γV*(s′)) Q*(s, a) = ∑ s′ p(s′|s, a)(r(s, a, s′) + γ max a′ Q*(s′, a′)) max a max a max a max a

  20. ໨࣍ ͸͡Ίʹ Ձ஋ؔ਺ͷಋೖ Ձ஋ؔ਺ͷఆ͔ٛΒ#FMMNBOํఔࣜΛಋग़ #BDLVQEJBHSBN͔Β#FMMNBOํఔࣜΛΈΔ ࠷దՁ஋ؔ਺ͷ঺հ #BDLVQEJBHSBN͔Β࠷దՁ஋؍਺ΛΈΔ ·ͱΊ

  21. ·ͱΊ ɾ΍ͬͨ͜ͱ ˠ#FMMNBOํఔࣜΛಋ͘ ɾ͜ͷํఔ͕ࣜͨͪղ͚ΔέʔεͰ͸࠷దղ͕ಘΒΕΔ ˠ%ZOBNJD1SPHSBNNJOH #FMMNBO࠷దੑͷݪཧ ɾ௨ৗ͸ɺ͍Ζ͍Ζͳཧ༝Ͱ%1ͷ࣮ߦ͸ෆՄೳ ˠαϯϓϦϯάతख๏ͷग़൪ .POUF$BSMP๏ɺ5%๏ͳͲ ɾͨͩ͠ɺଟ͘ͷΞϧΰϦζϜ͸ɺ#FMMNBOํఔࣜͷۙࣅతͳղ๏ͱͯ͠

    ཧղͰ͖Δ ˠ.$ͱ5%ɺ4BSTBͱ2MFBSOJOHͷಈ࡞ͷҧ͍ͳͲʹ͍ͭͯ #FMMNBOํఔࣜͷ؍఺͔Β΋ཧղͰ͖Δ

  22. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠