$30 off During Our Annual Pro Sale. View Details »

bellman方程式の導出.pdf

Avatar for m_nshr m_nshr
February 16, 2019
1
1.3k

 bellman方程式の導出.pdf

Deriving Bellman equations

Avatar for m_nshr

m_nshr

February 16, 2019
Tweet

Featured

See All Featured
Fashionably flexible responsive web design (full day workshop)
Avatar for Andy Clarke malarkey
407
66k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
Avatar for soudai sone soudai
PRO
196
70k
The Art of Delivering Value - GDevCon NA Keynote
Avatar for David Neal reverentgeek
16
1.8k
A Tale of Four Properties
Avatar for Chris Coyier chriscoyier
162
23k
Unsuck your backbone
Avatar for Amy Palamountain ammeep
671
58k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
Avatar for Eileen M. Uchitelle eileencodes
37
2.6k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
Avatar for Dan Newman danielanewman
231
22k
The Illustrated Children's Guide to Kubernetes
Avatar for Chris Short chrisshort
51
51k
Building Flexible Design Systems
Avatar for Yesenia Perez-Cruz yeseniaperezcruz
330
39k
YesSQL, Process and Tooling at Scale
Avatar for Rocio Delgado rocio
174
15k
The Cult of Friendly URLs
Avatar for Andy Hume andyhume
79
6.7k
The Straight Up "How To Draw Better" Workshop
Avatar for Dennis Kardys denniskardys
239
140k

Transcript

  1. #FMMNBOํఔࣜͷಋग़ .BUI$PEJOHڧԽֶशΛޠΖ͏ 

  2. ໨࣍ ͸͡Ίʹ Ձ஋ؔ਺ͷಋೖ Ձ஋ؔ਺ͷఆ͔ٛΒ#FMMNBOํఔࣜΛಋग़ #BDLVQEJBHSBN͔Β#FMMNBOํఔࣜΛΈΔ ࠷దՁ஋ؔ਺ͷ঺հ #BDLVQEJBHSBN͔Β࠷దՁ஋؍਺ΛΈΔ ·ͱΊ

  3. ڧԽֶशͷ໨తͱ͸ 5IBUBMMPGXIBUXFNFBOCZHPBMTBOEQVSQPTFTDBOCF XFMMUIPVHIUPGBTNBYJNJ[BUJPOPGUIFFYQFDUFEWBMVF PGUIFDVNVMBUJWFTVNPGBSFDFJWFETDBMBSTJHOBM DBMMFESFXBSE 4VUUPO#BSUP  3FXBSE)ZQPUIFTJT ྦྷੵใुͷ࠷େԽ

  4. Gt ≐ Rt+1 + γ Rt+2 + γ2 Rt+3 +

    ⋯ = ∞ ∑ k=0 γkRt+1+k = Rt+1 + γ ∞ ∑ k=0 γkRt+1+k+1 = Rt+1 + γGt+1 ˡʹண໨ S0 π A0 → R1 → S1 π A1 → R2 → S2 π ⋯ ྦྷੵใु ऩӹ γ ∈ [0,1]

  5. ϕϧϚϯ࠷దੑͷݪཧ ࠶ؼతߏ଄Λ࣋ͭ໰୊ ࠷దͳํࡦ͸ɺॳظঢ়ଶͱॳظܾఆ͕ͲΜͳ΋ͷͰ͋Εɺͦͷ݁Ռಘ ΒΕΔ࣍ͷঢ়ଶʹؔͯ͠ɺҎ߱ͷܾఆ͕ඞͣ࠷దํࡦʹͳ͍ͬͯΔͱ ͍͏ੑ࣭Λ΋ͭɻ ࢀরɿɹ#FMMNBO ɺ$IBQ*** 1SJODJQMFPG0QUJNBMJUZ ಈతܭը๏ %1

    Ͱղ͚Δ ͔΋͠Εͳ͍

  6. ໨࣍ ͸͡Ίʹ Ձ஋ؔ਺ͷಋೖ Ձ஋ؔ਺ͷఆ͔ٛΒ#FMMNBOํఔࣜΛಋग़ #BDLVQEJBHSBN͔Β#FMMNBOํఔࣜΛΈΔ ࠷దՁ஋ؔ਺ͷ঺հ #BDLVQEJBHSBN͔Β࠷దՁ஋؍਺ΛΈΔ ·ͱΊ

  7. ه߸ͷ४උ ํࡦ ঢ়ଶભҠ֬཰ π(a|s) ≐ (At = a|St = s)

    p(s′|s, a) ≐ (St+1 = s′|St = s, At = a) r(s, a, s′) ≐ [Rt+1 |St = s, At = a, St+1 = s′] ˠঢ়ଶTͰߦಈBΛબ୒͢Δ֬཰ ˠঢ়ଶTͰߦಈBΛબ୒ͯ࣍͠ঢ়ଶT`ʹભҠ͢Δ֬཰ ˠঢ়ଶTͰߦಈBΛબ୒ͯ࣍͠ঢ়ଶT`ʹભҠͨ͠৔߹ͷଈ࣌ใुͷظ଴஋ ؀ڥͷμΠφϛΫε FOWJSPONFOU`TEZOBNJDT ଈ࣌ใु ใुؔ਺

  8. Ձ஋ؔ਺ zঢ়ଶzՁ஋ؔ਺ lঢ়ଶɾߦಈzՁ஋ؔ਺ Vπ(s) ≐ π [Gt |St = s]

    Qπ(s, a) ≐ π [Gt |St = s, At = a] ྆ऀͷؔ܎ Vπ(s) ≐ π [Gt |St = s] = ∑ a π(a|s)π [Gt |St = s, At = a] = ∑ a π(a|s)Qπ(s, a) ˡ݁Ռ

  9. ໨࣍ ͸͡Ίʹ Ձ஋ؔ਺ͷಋೖ Ձ஋ؔ਺ͷఆ͔ٛΒ#FMMNBOํఔࣜΛಋग़ #BDLVQEJBHSBN͔Β#FMMNBOํఔࣜΛΈΔ ࠷దՁ஋ؔ਺ͷ঺հ #BDLVQEJBHSBN͔Β࠷దՁ஋؍਺ΛΈΔ ·ͱΊ

  10. Vπ(s) ≐ π [Gt |St = s] = π [Rt+1

    + γGt+1 |St = s] = ∑ a π(a|s)∑ s′ p(s′|s, a)π [Rt+1 + γGt+1 |St = s, At = a, St+1 = s′] = ∑ a π(a|s)∑ s′ p(s′|s, a)(r(s, a, s′) + γπ [Gt+1 |St = s, At = a, St+1 = s′]) = ∑ a π(a|s)∑ s′ p(s′|s, a)(r(s, a, s′) + γVπ(s′)) ˡ݁Ռ #FMMNBOํఔࣜGPS Vπ(s)

  11. Qπ(s, a) ≐ π [Gt |St = s, At =

    a] = π [Rt+1 + γGt+1 |St = s, At = a] = ∑ s′ p(s′|s, a)π [Rt+1 + γGt+1 |St = s, At = a, St+1 = s′] = ∑ s′ p(s′|s, a)(r(s, a, s′) + γπ [Gt+1 |St = s, At = a, St+1 = s′]) = ∑ s′ p(s′|s, a)(r(s, a, s′) + γVπ(s′)) = ∑ s′ p(s′|s, a)(r(s, a, s′) + γ∑ a′ π(a′|s′)Qπ(s′, a′)) #FMMNBOํఔࣜGPS Qπ(s, a) ˡ݁Ռ ˣ݁ՌΛ୅ೖ ˡ݁Ռ

  12. ໨࣍ ͸͡Ίʹ Ձ஋ؔ਺ͷಋೖ Ձ஋ؔ਺ͷఆ͔ٛΒ#FMMNBOํఔࣜΛಋग़ #BDLVQEJBHSBN͔Β#FMMNBOํఔࣜΛΈΔ ࠷దՁ஋ؔ਺ͷ঺հ #BDLVQEJBHSBN͔Β࠷దՁ஋؍਺ΛΈΔ ·ͱΊ

  13. #BDLVQEJBHSBN ɾঢ়ଶͱߦಈͷܥྻΛਤͰද͢ ɾ˓͸ঢ়ଶɺ˔͸ߦಈ ·ͨ͸ঢ়ଶɾߦಈର Λද͢ ɾϧʔτϊʔυͷՁ஋Λܭࢉ͢Δ࣌ʹ࢖͏ ɾϧʔτϊʔυ Ұ൪্ͷϊʔυ ͷՁ஋͕ ͲΜͳཁૉ͔Β੒Γཱ͍ͬͯΔ͔Λදݱ͠

    ͍ͯΔ

  14. #BDLVQEJBHSBNͰ #FMMNBOํఔࣜΛ֬ೝ Vπ(s) = ∑ a π(a|s)Qπ(s, a) Qπ(s, a)

    = ∑ s′ p(s′|s, a)(r(s, a, s′) + γVπ(s′)) π(a|s) Vπ(s) s a1 a2 Qπ(s, a1 ) Qπ(s, a2 ) Qπ(s, a) p(s′|s, a) (s, a) r(s, a, s′1 ) s′1 s′2 r(s, a, s′2 ) ˠ݁Ռ ࠶ Vπ(s′1 ) Vπ(s′2 ) ˠ݁Ռ ࠶

  15. #BDLVQEJBHSBNͰ #FMMNBOํఔࣜΛ֬ೝ Vπ(s) = ∑ a π(a|s)Qπ(s, a) = ∑

    a π(a|s)∑ s′ p(s′|s, a)(r(s, a, s′) + γVπ(s′)) Qπ(s, a) = ∑ s′ p(s′|s, a)(r(s, a, s′) + γVπ(s′)) = ∑ s′ p(s′|s, a)(r(s, a, s′) + γ∑ a′ π(a′|s′)Qπ(s′, a′)) Vπ(s) Vπ(s′) Qπ(s, a) Qπ(s, a) Qπ(s′, a′) π ˠ݁Ռ ࠶ p p r(s, a, s′) π(a|s) Vπ(s′) r(s, a, s′) ˣ݁ՌΛ୅ೖ ˠ݁Ռ ࠶

  16. ໨࣍ ͸͡Ίʹ Ձ஋ؔ਺ͷಋೖ Ձ஋ؔ਺ͷఆ͔ٛΒ#FMMNBOํఔࣜΛಋग़ #BDLVQEJBHSBN͔Β#FMMNBOํఔࣜΛΈΔ ࠷దՁ஋ؔ਺ͷ঺հ #BDLVQEJBHSBN͔Β࠷దՁ஋؍਺ΛΈΔ ·ͱΊ

  17. ࠷దՁ஋ؔ਺ V*(s) = max π Vπ(s) for any Q*(s, a)

    = max π Qπ(s, a) for any s ∈ s ∈ , a ∈ ɾ͜ͷؔ܎Λຬͨ͢ ͕গͳ͘ͱ΋ͭଘࡏ͢Δ ࠷దํࡦ ɾ͜ͷ ʹΑͬͯɺऩӹͷ࠷େԽ͕ୡ੒͞ΕΔ π π

  18. ໨࣍ ͸͡Ίʹ Ձ஋ؔ਺ͷಋೖ Ձ஋ؔ਺ͷఆ͔ٛΒ#FMMNBOํఔࣜΛಋग़ #BDLVQEJBHSBN͔Β#FMMNBOํఔࣜΛΈΔ ࠷దՁ஋ؔ਺ͷ঺հ #BDLVQEJBHSBN͔Β࠷దՁ஋؍਺ΛΈΔ ·ͱΊ

  19. #BDLVQEJBHSBNͰ #FMMNBO࠷దํఔࣜΛ֬ೝ V*(s) = ∑ a π(a|s) max a Q*(s,

    a) Q*(s, a) = ∑ s′ p(s′|s, a)(r(s, a, s′) + γV*(s′)) V*(s) = max a ∑ s′ p(s′|s, a)(r(s, a, s′) + γV*(s′)) Q*(s, a) = ∑ s′ p(s′|s, a)(r(s, a, s′) + γ max a′ Q*(s′, a′)) max a max a max a max a

  20. ໨࣍ ͸͡Ίʹ Ձ஋ؔ਺ͷಋೖ Ձ஋ؔ਺ͷఆ͔ٛΒ#FMMNBOํఔࣜΛಋग़ #BDLVQEJBHSBN͔Β#FMMNBOํఔࣜΛΈΔ ࠷దՁ஋ؔ਺ͷ঺հ #BDLVQEJBHSBN͔Β࠷దՁ஋؍਺ΛΈΔ ·ͱΊ

  21. ·ͱΊ ɾ΍ͬͨ͜ͱ ˠ#FMMNBOํఔࣜΛಋ͘ ɾ͜ͷํఔ͕ࣜͨͪղ͚ΔέʔεͰ͸࠷దղ͕ಘΒΕΔ ˠ%ZOBNJD1SPHSBNNJOH #FMMNBO࠷దੑͷݪཧ ɾ௨ৗ͸ɺ͍Ζ͍Ζͳཧ༝Ͱ%1ͷ࣮ߦ͸ෆՄೳ ˠαϯϓϦϯάతख๏ͷग़൪ .POUF$BSMP๏ɺ5%๏ͳͲ ɾͨͩ͠ɺଟ͘ͷΞϧΰϦζϜ͸ɺ#FMMNBOํఔࣜͷۙࣅతͳղ๏ͱͯ͠

    ཧղͰ͖Δ ˠ.$ͱ5%ɺ4BSTBͱ2MFBSOJOHͷಈ࡞ͷҧ͍ͳͲʹ͍ͭͯ #FMMNBOํఔࣜͷ؍఺͔Β΋ཧղͰ͖Δ

  22. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠