Upgrade to Pro — share decks privately, control downloads, hide ads and more …

bellman方程式の導出.pdf

Avatar for m_nshr m_nshr
February 16, 2019
1
1.2k

 bellman方程式の導出.pdf

Deriving Bellman equations

Avatar for m_nshr

m_nshr

February 16, 2019
Tweet

Featured

See All Featured
Save Time (by Creating Custom Rails Generators)
Avatar for Garrett Dimon garrettdimon
PRO
32
1.6k
Writing Fast Ruby
Avatar for Erik Berlin sferik
629
62k
Building Applications with DynamoDB
Avatar for Matt Wood mza
96
6.6k
Learning to Love Humans: Emotional Interface Design
Avatar for Aarron Walter aarron
274
40k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
Avatar for Dan Newman danielanewman
229
22k
For a Future-Friendly Web
Avatar for Brad Frost brad_frost
180
9.9k
How STYLIGHT went responsive
Avatar for nonsquared nonsquared
100
5.8k
GraphQLの誤解/rethinking-graphql
Avatar for sonatard sonatard
73
11k
Code Reviewing Like a Champion
Avatar for maltzj maltzj
525
40k
VelocityConf: Rendering Performance Case Studies
Avatar for Addy Osmani addyosmani
332
24k
What’s in a name? Adding method to the madness
Avatar for Product Marketing Alliance productmarketing
PRO
23
3.7k
Thoughts on Productivity
Avatar for Jon Yablonski jonyablonski
70
4.9k

Transcript

  1. #FMMNBOํఔࣜͷಋग़ .BUI$PEJOHڧԽֶशΛޠΖ͏ 

  2. ໨࣍ ͸͡Ίʹ Ձ஋ؔ਺ͷಋೖ Ձ஋ؔ਺ͷఆ͔ٛΒ#FMMNBOํఔࣜΛಋग़ #BDLVQEJBHSBN͔Β#FMMNBOํఔࣜΛΈΔ ࠷దՁ஋ؔ਺ͷ঺հ #BDLVQEJBHSBN͔Β࠷దՁ஋؍਺ΛΈΔ ·ͱΊ

  3. ڧԽֶशͷ໨తͱ͸ 5IBUBMMPGXIBUXFNFBOCZHPBMTBOEQVSQPTFTDBOCF XFMMUIPVHIUPGBTNBYJNJ[BUJPOPGUIFFYQFDUFEWBMVF PGUIFDVNVMBUJWFTVNPGBSFDFJWFETDBMBSTJHOBM DBMMFESFXBSE 4VUUPO#BSUP  3FXBSE)ZQPUIFTJT ྦྷੵใुͷ࠷େԽ

  4. Gt ≐ Rt+1 + γ Rt+2 + γ2 Rt+3 +

    ⋯ = ∞ ∑ k=0 γkRt+1+k = Rt+1 + γ ∞ ∑ k=0 γkRt+1+k+1 = Rt+1 + γGt+1 ˡʹண໨ S0 π A0 → R1 → S1 π A1 → R2 → S2 π ⋯ ྦྷੵใु ऩӹ γ ∈ [0,1]

  5. ϕϧϚϯ࠷దੑͷݪཧ ࠶ؼతߏ଄Λ࣋ͭ໰୊ ࠷దͳํࡦ͸ɺॳظঢ়ଶͱॳظܾఆ͕ͲΜͳ΋ͷͰ͋Εɺͦͷ݁Ռಘ ΒΕΔ࣍ͷঢ়ଶʹؔͯ͠ɺҎ߱ͷܾఆ͕ඞͣ࠷దํࡦʹͳ͍ͬͯΔͱ ͍͏ੑ࣭Λ΋ͭɻ ࢀরɿɹ#FMMNBO ɺ$IBQ*** 1SJODJQMFPG0QUJNBMJUZ ಈతܭը๏ %1

    Ͱղ͚Δ ͔΋͠Εͳ͍

  6. ໨࣍ ͸͡Ίʹ Ձ஋ؔ਺ͷಋೖ Ձ஋ؔ਺ͷఆ͔ٛΒ#FMMNBOํఔࣜΛಋग़ #BDLVQEJBHSBN͔Β#FMMNBOํఔࣜΛΈΔ ࠷దՁ஋ؔ਺ͷ঺հ #BDLVQEJBHSBN͔Β࠷దՁ஋؍਺ΛΈΔ ·ͱΊ

  7. ه߸ͷ४උ ํࡦ ঢ়ଶભҠ֬཰ π(a|s) ≐ (At = a|St = s)

    p(s′|s, a) ≐ (St+1 = s′|St = s, At = a) r(s, a, s′) ≐ [Rt+1 |St = s, At = a, St+1 = s′] ˠঢ়ଶTͰߦಈBΛબ୒͢Δ֬཰ ˠঢ়ଶTͰߦಈBΛબ୒ͯ࣍͠ঢ়ଶT`ʹભҠ͢Δ֬཰ ˠঢ়ଶTͰߦಈBΛબ୒ͯ࣍͠ঢ়ଶT`ʹભҠͨ͠৔߹ͷଈ࣌ใुͷظ଴஋ ؀ڥͷμΠφϛΫε FOWJSPONFOU`TEZOBNJDT ଈ࣌ใु ใुؔ਺

  8. Ձ஋ؔ਺ zঢ়ଶzՁ஋ؔ਺ lঢ়ଶɾߦಈzՁ஋ؔ਺ Vπ(s) ≐ π [Gt |St = s]

    Qπ(s, a) ≐ π [Gt |St = s, At = a] ྆ऀͷؔ܎ Vπ(s) ≐ π [Gt |St = s] = ∑ a π(a|s)π [Gt |St = s, At = a] = ∑ a π(a|s)Qπ(s, a) ˡ݁Ռ

  9. ໨࣍ ͸͡Ίʹ Ձ஋ؔ਺ͷಋೖ Ձ஋ؔ਺ͷఆ͔ٛΒ#FMMNBOํఔࣜΛಋग़ #BDLVQEJBHSBN͔Β#FMMNBOํఔࣜΛΈΔ ࠷దՁ஋ؔ਺ͷ঺հ #BDLVQEJBHSBN͔Β࠷దՁ஋؍਺ΛΈΔ ·ͱΊ

  10. Vπ(s) ≐ π [Gt |St = s] = π [Rt+1

    + γGt+1 |St = s] = ∑ a π(a|s)∑ s′ p(s′|s, a)π [Rt+1 + γGt+1 |St = s, At = a, St+1 = s′] = ∑ a π(a|s)∑ s′ p(s′|s, a)(r(s, a, s′) + γπ [Gt+1 |St = s, At = a, St+1 = s′]) = ∑ a π(a|s)∑ s′ p(s′|s, a)(r(s, a, s′) + γVπ(s′)) ˡ݁Ռ #FMMNBOํఔࣜGPS Vπ(s)

  11. Qπ(s, a) ≐ π [Gt |St = s, At =

    a] = π [Rt+1 + γGt+1 |St = s, At = a] = ∑ s′ p(s′|s, a)π [Rt+1 + γGt+1 |St = s, At = a, St+1 = s′] = ∑ s′ p(s′|s, a)(r(s, a, s′) + γπ [Gt+1 |St = s, At = a, St+1 = s′]) = ∑ s′ p(s′|s, a)(r(s, a, s′) + γVπ(s′)) = ∑ s′ p(s′|s, a)(r(s, a, s′) + γ∑ a′ π(a′|s′)Qπ(s′, a′)) #FMMNBOํఔࣜGPS Qπ(s, a) ˡ݁Ռ ˣ݁ՌΛ୅ೖ ˡ݁Ռ

  12. ໨࣍ ͸͡Ίʹ Ձ஋ؔ਺ͷಋೖ Ձ஋ؔ਺ͷఆ͔ٛΒ#FMMNBOํఔࣜΛಋग़ #BDLVQEJBHSBN͔Β#FMMNBOํఔࣜΛΈΔ ࠷దՁ஋ؔ਺ͷ঺հ #BDLVQEJBHSBN͔Β࠷దՁ஋؍਺ΛΈΔ ·ͱΊ

  13. #BDLVQEJBHSBN ɾঢ়ଶͱߦಈͷܥྻΛਤͰද͢ ɾ˓͸ঢ়ଶɺ˔͸ߦಈ ·ͨ͸ঢ়ଶɾߦಈର Λද͢ ɾϧʔτϊʔυͷՁ஋Λܭࢉ͢Δ࣌ʹ࢖͏ ɾϧʔτϊʔυ Ұ൪্ͷϊʔυ ͷՁ஋͕ ͲΜͳཁૉ͔Β੒Γཱ͍ͬͯΔ͔Λදݱ͠

    ͍ͯΔ

  14. #BDLVQEJBHSBNͰ #FMMNBOํఔࣜΛ֬ೝ Vπ(s) = ∑ a π(a|s)Qπ(s, a) Qπ(s, a)

    = ∑ s′ p(s′|s, a)(r(s, a, s′) + γVπ(s′)) π(a|s) Vπ(s) s a1 a2 Qπ(s, a1 ) Qπ(s, a2 ) Qπ(s, a) p(s′|s, a) (s, a) r(s, a, s′1 ) s′1 s′2 r(s, a, s′2 ) ˠ݁Ռ ࠶ Vπ(s′1 ) Vπ(s′2 ) ˠ݁Ռ ࠶

  15. #BDLVQEJBHSBNͰ #FMMNBOํఔࣜΛ֬ೝ Vπ(s) = ∑ a π(a|s)Qπ(s, a) = ∑

    a π(a|s)∑ s′ p(s′|s, a)(r(s, a, s′) + γVπ(s′)) Qπ(s, a) = ∑ s′ p(s′|s, a)(r(s, a, s′) + γVπ(s′)) = ∑ s′ p(s′|s, a)(r(s, a, s′) + γ∑ a′ π(a′|s′)Qπ(s′, a′)) Vπ(s) Vπ(s′) Qπ(s, a) Qπ(s, a) Qπ(s′, a′) π ˠ݁Ռ ࠶ p p r(s, a, s′) π(a|s) Vπ(s′) r(s, a, s′) ˣ݁ՌΛ୅ೖ ˠ݁Ռ ࠶

  16. ໨࣍ ͸͡Ίʹ Ձ஋ؔ਺ͷಋೖ Ձ஋ؔ਺ͷఆ͔ٛΒ#FMMNBOํఔࣜΛಋग़ #BDLVQEJBHSBN͔Β#FMMNBOํఔࣜΛΈΔ ࠷దՁ஋ؔ਺ͷ঺հ #BDLVQEJBHSBN͔Β࠷దՁ஋؍਺ΛΈΔ ·ͱΊ

  17. ࠷దՁ஋ؔ਺ V*(s) = max π Vπ(s) for any Q*(s, a)

    = max π Qπ(s, a) for any s ∈ s ∈ , a ∈ ɾ͜ͷؔ܎Λຬͨ͢ ͕গͳ͘ͱ΋ͭଘࡏ͢Δ ࠷దํࡦ ɾ͜ͷ ʹΑͬͯɺऩӹͷ࠷େԽ͕ୡ੒͞ΕΔ π π

  18. ໨࣍ ͸͡Ίʹ Ձ஋ؔ਺ͷಋೖ Ձ஋ؔ਺ͷఆ͔ٛΒ#FMMNBOํఔࣜΛಋग़ #BDLVQEJBHSBN͔Β#FMMNBOํఔࣜΛΈΔ ࠷దՁ஋ؔ਺ͷ঺հ #BDLVQEJBHSBN͔Β࠷దՁ஋؍਺ΛΈΔ ·ͱΊ

  19. #BDLVQEJBHSBNͰ #FMMNBO࠷దํఔࣜΛ֬ೝ V*(s) = ∑ a π(a|s) max a Q*(s,

    a) Q*(s, a) = ∑ s′ p(s′|s, a)(r(s, a, s′) + γV*(s′)) V*(s) = max a ∑ s′ p(s′|s, a)(r(s, a, s′) + γV*(s′)) Q*(s, a) = ∑ s′ p(s′|s, a)(r(s, a, s′) + γ max a′ Q*(s′, a′)) max a max a max a max a

  20. ໨࣍ ͸͡Ίʹ Ձ஋ؔ਺ͷಋೖ Ձ஋ؔ਺ͷఆ͔ٛΒ#FMMNBOํఔࣜΛಋग़ #BDLVQEJBHSBN͔Β#FMMNBOํఔࣜΛΈΔ ࠷దՁ஋ؔ਺ͷ঺հ #BDLVQEJBHSBN͔Β࠷దՁ஋؍਺ΛΈΔ ·ͱΊ

  21. ·ͱΊ ɾ΍ͬͨ͜ͱ ˠ#FMMNBOํఔࣜΛಋ͘ ɾ͜ͷํఔ͕ࣜͨͪղ͚ΔέʔεͰ͸࠷దղ͕ಘΒΕΔ ˠ%ZOBNJD1SPHSBNNJOH #FMMNBO࠷దੑͷݪཧ ɾ௨ৗ͸ɺ͍Ζ͍Ζͳཧ༝Ͱ%1ͷ࣮ߦ͸ෆՄೳ ˠαϯϓϦϯάతख๏ͷग़൪ .POUF$BSMP๏ɺ5%๏ͳͲ ɾͨͩ͠ɺଟ͘ͷΞϧΰϦζϜ͸ɺ#FMMNBOํఔࣜͷۙࣅతͳղ๏ͱͯ͠

    ཧղͰ͖Δ ˠ.$ͱ5%ɺ4BSTBͱ2MFBSOJOHͷಈ࡞ͷҧ͍ͳͲʹ͍ͭͯ #FMMNBOํఔࣜͷ؍఺͔Β΋ཧղͰ͖Δ

  22. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠