icebee16
December 03, 2019
290

# 「強化学習」輪読会資料

「強化学習」(森村 哲郎)の輪読会資料

## icebee16

December 03, 2019

## Transcript

1. ڧԽֶश ྠಡձ #02, #03
@icebee__
CHAPTER 2 ϓϥϯχϯά

2. ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
@icebee__
•ۚ༥ܥۈΊͷࣾձਓೋ೥໨
•ڵຯͱͯ͠͸σʔλ෼ੳ΍ػցֶशɼ౷ܭؔ࿈ͷ࿩
•Kaggle Expert

3. CONTENTS
Chapter 2 ϓϥϯχϯά
2.1 ४උ
2.1.1 ໨తؔ਺
2.1.2 ࠷దՁ஋ؔ਺
2.2 ಈతܭը๏
2.2.1 ϕϧϚϯ࡞༻ૉ
2.2.2 ϕϧϚϯ࡞༻ૉͷ୯ௐੑ
2.2.3 ಈతܭը๏ͷ਺ཧ
2.2.4 ࠷దํࡦ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

4. CONTENTS
Chapter 2 ϓϥϯχϯά
2.3 ಈతܭը๏ʹΑΔղ๏
2.3.1 Ձ஋൓෮๏
2.3.2 ํࡦ൓෮๏
2.4 ઢܗܭը๏ʹΑΔղ๏
2.4.1 ओ໰୊ͷಋग़
2.4.2* ૒ର໰୊ (ࠓճ͸औΓѻΘͳ͍)
2.4.3* ࠷దղͱ࠷దํࡦ (ࠓճ͸औΓѻΘͳ͍)
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

5. 2.1 ४උ
▸ ϓϥϯχϯά໰୊
➡ ؀ڥ͕ط஌ͷ৔߹ͷஞ࣍తҙࢥܾఆ໰୊
➡ ؀ڥ͕ະ஌ͷ৔߹ʹ͋ͨΔڧԽֶश໰୊ΛऔΓѻ͏্Ͱͷج
ૅͱͳΔ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

6. 2.1.1 ໨తؔ਺
▸ ҎԼͷ໨తؔ਺Λ༻ҙ͢Δ
▸ ্͔ࣜΒಋ͔ΕΔ ͸ॳظঢ়ଶ ʹґଘ͢ΔՄೳੑ͕ߟ͑ΒΕΔ͕
ޙड़(2.2અ)͢Δ໋୊ΑΓ೚ҙͷॳظঢ়ଶʹ͓͍ͯҎԼ͕੒ཱ͢
Δɽ
▸ ΑͬͯఆৗతͳํࡦͷΈΛߟ͑Δ͜ͱͰ໨తؔ਺Λ࠷େԽग़དྷΔɽ
ͨͩ͠ɼ ͸ୈmཁૉ͕1Ͱଞͷཁૉ͸θϩͷn࣍ݩϕΫτϧ
Ձ஋ؔ਺ͷॏΈ෇͖࿨
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

7. 2.1.2 ࠷దՁ஋ؔ਺
▸ ҎԼͷࣜʹͯ࠷దՁ஋ؔ਺(optimal value function)Λಋೖ
▸ Ձ஋ؔ਺ ͸ Λ༻͍ͯҎԼͷ༷ʹهड़Մೳ
▸ ͜ΕΑΓ ͸Ϛϧίϑํࡦͷे෼ੑʹؔ͢ΔܥΛຬͨ͠ɼ࠷ద
Ձ஋ؔ਺ʹ͍ͭͯҎԼͷ͕ࣜ੒Γཱͭɽ
ॻ੶ࢀߟϖʔδɿp.22
ಉ࣌पล֬཰ɿ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

8. 2.1.2 ࠷దՁ஋ؔ਺
▸ ࠷దՁ஋ؔ਺͸ҎԼͷΑ͏ʹॻ͖ԼͤΔɽ
▸ ͜ΕΑΓϕϧϚϯ࠷దํఔࣜ(Bellman optimality equation)ͱݺ
͹ΕΔҎԼͷ͕ࣜ੒Γཱͭɽ
Ϧλʔϯͷ࠶ؼੑɿ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

9. 2.1.2 ࠷దՁ஋ؔ਺
▸ ࠷దՁ஋ؔ਺͸ҎԼͷΑ͏ʹॻ͖ԼͤΔɽ
▸ ͜ΕΑΓϕϧϚϯ࠷దํఔࣜ(Bellman optimality equation)ͱݺ
͹ΕΔҎԼͷ͕ࣜ੒Γཱͭɽ
Ϧλʔϯͷ࠶ؼੑɿ
ભҠ֬཰
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

10. 2.2 ಈతܭը๏
▸ ಈతܭը๏
➡ ஞ࣍తҙࢥܾఆͷ࠷దԽ໰୊Λղ͘Ξϓϩʔνͷ͜ͱ
➡ ͨͩ͠ɼ۩ମతͳख๏Λࢦ͢΋ͷͰ͸ͳ͘લड़ͷΑ͏ͳΞϓ
ϩʔνͷ૯শ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

11. 2.2.1 ϕϧϚϯ࡞༻ૉ
▸ ಈతܭը๏Ͱ͸ϕϧϚϯ࡞༻ૉͱݺ͹ΕΔҎԼͷॲཧ Λؔ਺
ʹ܁Γฦ͠ద༻͢Δ͜ͱͰՁ஋ؔ਺ɼ࠷దՁ஋ؔ਺ʹ͚ۙͮΔɽ
▸ ·ͣɼ࣍ͷ̎ͭͷ೚ҙͷؔ਺ʹର͢ΔϕϧϚϯ࡞༻ૉΛಋೖ͢
Δɽ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

12. 2.2.1 ϕϧϚϯ࡞༻ૉ
▸ ϕϧϚϯظ଴࡞༻ૉ
▸ ϕϧϚϯ࠷ద࡞༻ૉ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

13. 2.2.1 ϕϧϚϯ࡞༻ૉ
▸ ·ͨɼՃ͑ͯҎԼͷ͍͔ͭ͘ͷه๏Λಋೖ͢Δɽ
▸ ؆ศԽ
▸ ஞ࣍తͳద༻
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

14. 2.2.1 ϕϧϚϯ࡞༻ૉ
▸ ઌΜͯࣔͨ͡͠ϕϧϚϯ࡞༻ૉ͸ͦΕͧΕϕϧϚϯํఔࣜͱҎԼ
ͷؔ܎Λ࣋ͭɽ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

15. 2.2.2 ϕϧϚϯ࡞༻ૉͷ୯ௐੑ
▸ ิ୊2.1ʢ୯ௐੑͷิ୊ʣ
▸ ೚ҙͷؔ਺ͷ૊ʹରͯ͠ϕϧϚϯ࡞༻ૉΛద༻ͯ͠΋ͦͷେখ
ؔ܎͸อͨΕΔɽ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

16. 2.2.2 ϕϧϚϯ࡞༻ૉͷ୯ௐੑ
▸ ҎԼͷදهΛఆٛ͢Δɽ
▸ ͜ͷͱ͖ϕϧϚϯ࡞༻ૉͷఆٛΑΓҎԼ͕ࣔͤΔ
▸ ิ୊2.2
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

17. 2.2.3 ಈతܭը๏ͷ਺ཧ
▸ ໋୊2.3ʢಈతܭը๏ͷऩଋੑʣ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

18. 2.2.3 ಈతܭը๏ͷ਺ཧ
▸ ໋୊2.3ʢಈతܭը๏ͷऩଋੑʣূ໌ϝϞ
▸ a. ~ (2.14)
▸ ઌʹ্͛ͨิ୊Λ͏·͘࢖͏͚ͩɽ༗քͰ͋Δ͜ͱΛར༻ͯ͠
೚ҙͷఆ਺ͰڬΈɼk>∞Ͱఆ਺Λখ͘͢͞Δ͚ͩɽ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

19. 2.2.3 ಈతܭը๏ͷ਺ཧ
▸ ໋୊2.3ʢಈతܭը๏ͷऩଋੑʣূ໌ϝϞ
▸ b. ~ (2.16)
▸ Ձ஋ؔ਺Λํࡦͷk൪໨Λڥʹ෼ׂ͢Δ͚ͩɽ
▸ (2.16) ~ (2.17)
▸ ใुؔ਺ͷఆٛ(1.1)ʹै֤ͬͯεςοϓͷใुΛ্͔Β཈͑Δɽ
▸ (2.16)ͷӈลೋ߲໨ʹ͍ͭͯղ͍ͯ୅ೖɽ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

20. 2.2.3 ಈతܭը๏ͷ਺ཧ
▸ ໋୊2.3ʢಈతܭը๏ͷऩଋੑʣূ໌ϝϞ
▸ (2.17) Լ~ (2.18)
▸ ʢp.49ͷԼࣜE[]಺ೋ߲໨ɼγͱSͷఴࣈ͸ޡ২Ͱ͸ʁt > kʣ
▸ v(s)ͷظ଴஋͸min(v(s))ͱmax(v(s))ͰڬΊΔɽ
▸ ग़དྷͨࣜͱ(2.17)ͷࠩ෼ΛͱΔɽ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

21. 2.2.3 ಈతܭը๏ͷ਺ཧ
▸ ໋୊2.4ʢϕϧϚϯํఔࣜͷղͷҰҙੑʣ
▸ എཧ๏Ͱ؆୯ʹূ໌
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

22. 2.2.3 ಈతܭը๏ͷ਺ཧ
▸ ิ୊2.5ʢϕϧϚϯ࡞༻ૉͷॖখੑʣ
▸ ূ໌͔Βೖͬͨ΄͏͕ཧղ͕ૣ͍ɽ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

23. 2.2.3 ಈతܭը๏ͷ਺ཧ
▸ ิ୊2.5ʢϕϧϚϯ࡞༻ૉͷॖখੑʣূ໌ϝϞ
▸ (2.21), (2.22), (2.23)
▸ ิ୊2.2͋ͨΓΛ༻͍ͯڬΜͰࣔ͢
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

24. 2.2.3 ಈతܭը๏ͷ਺ཧ
▸ ิ୊2.5ʢϕϧϚϯ࡞༻ૉͷॖখੑʣ͔Β
▸ (2.23)ͷv’ʹରͯ͠V*Λ୅ೖɼϕϧϚϯ࠷దԽ࡞༻ૉͷෆಈ఺͕
V*Ͱ͋Δ͜ͱΛ౿·͑Ε͹ҎԼͷ͕ࣜ੒ཱ
▸ ͜Ε͸ϕϧϚϯ࠷దԽ࡞༻ૉΛ೚ҙͷؔ਺ʹదԠ͢Δ͜ͱͰ࠷ద
Ձ஋ؔ਺ͱͦͷؔ਺ͷڑ཭͕γͷൺ཰Ͱࢦ਺ؔ਺తʹখ͘͞ͳͬ
͍ͯ͘͜ͱΛࣔ͢ɽ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

25. 2.2.4 ࠷దํࡦ
▸ ఆٛ2.6ʢ࠷దํࡦʣ
▸ ͋Δঢ়ଶsͷظ଴ϦλʔϯΛ࠷େʹ͢Δ͜ͱ͸Մೳ͕ͩ೚ҙͷঢ়
ଶsʹରͯ͜͠ΕΛݴ͏ͷ͸೉͍͠ɼͱ͍͏͚ͩɽ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

26. 2.2.4 ࠷దํࡦ
▸ ໋୊2.7ʢ࠷దํࡦͷଘࡏੑͱඞཁे෼৚݅ʣ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

27. 2.2.4 ࠷దํࡦ
▸ ໋୊2.7ʢ࠷దํࡦͷଘࡏੑͱඞཁे෼৚݅ʣূ໌ϝϞ
▸ ඞཁ৚݅ɼे෼৚݅
▸ ʹରͯ͠ϕϧϚϯظ଴࡞༻ૉΛߟ͑ͨͱ͖ɼ͜ͷෆಈ఺͸
ิ୊2.4͔ΒҰҙʹ ͕ͩɼ(2.28)ΑΓ Ͱ΋͋ΔɽΑͬͯ
ͦΕΒ͸Ұக͍ͯ͠Δ
▸ ଘࡏ͢Δ͜ͱΛࣔ͢
▸ ਖ਼௚Α͘Θ͔͍ͬͯͳ͍ɽ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

28. 2.2.4 ࠷దํࡦ
▸ ໋୊2.7ʢ࠷దํࡦͷଘࡏੑͱඞཁे෼৚݅ʣ͔Β
▸ ͷఆٛʹ ΛೖΕͯ͋͛Ε͹ҎԼ͕ݴ͑Δ
▸ ॏΈ෇͖Ձ஋ؔ਺ͷఆ͔ٛΒҎԼ
▸ ͢ͳΘͪɼ࠷దํࡦͷֶशͱ͸໨తؔ਺ Λ࠷େʹ
͢Δํࡦͷ୳ࡧ໰୊Ͱ͋Δͱ͍͑Δɽ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

29. 2.2.4 ࠷దํࡦ
▸ લड़ͨࣜ͠ʹ͍ͭͯ ͱ ͱͳ͍ͬͯΔཧ༝ͱͯ͠͸ɼ = 0ͱ
ͳ͍ͬͯΔঢ়ଶɼ͢ͳΘͪભҠ͢ΔՄೳੑͷͳ͍ঢ়ଶؒͷύε·
Ͱ࠷దԽͷର৅ʹ͢Δඞཁ͸ͳ͍͔Βɽ
▸ ໋୊2.7ͷ݁Ռ͸ҎԼͷೋ఺ΑΓॏཁ
▸ ࠷దԽର৅ͷํࡦू߹Λߟ͑Δʹ͋ͨͬͯ୳ࡧൣғ͸ఆৗͳܾ
ఆతํࡦͷू߹ͷΈͰ໰୊ͳ͍͜ͱΛอূͨ͜͠ͱɽ
▸ ֶश͍ͯ͠ΔՁ஋ؔ਺͕࠷దՁ஋ؔ਺ͱҰக͢Δ͜ͱͰ࠷దํ
ࡦ͔൱͔Λ൑ఆͰ͖Δ͜ͱɽ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

30. 2.3 ಈతܭը๏ʹΑΔղ๏
▸ ಈతܭը๏ͷ۩ମతͳ࣮૷ͱͯ͠ҎԼͷ2ͭΛ঺հ
➡ Ձ஋൓෮๏ʢvalue iteration algorithmʣ
➡ ํࡦ൓෮๏ʢpolicy iteration algorithmʣ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

31. 2.3.1 Ձ஋൓෮๏
▸ ΞϧΰϦζϜ2.1
▸ ऩଋ͕อূ͞Ε͍ͯΔͷͰ୯७ͳ܁Γฦ͠ɽ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

32. 2.3.1 Ձ஋൓෮๏
▸ ໋୊2.8ʢಈతܭը๏ͷޡࠩݶքʣ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

33. 2.3.1 Ձ஋൓෮๏
▸ ໋୊2.8ʢಈతܭը๏ͷޡࠩݶքʣ
▸ ূ໌ͷৄࡉ͸ิ଍A.1.1(p.250)
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

34. 2.3.1 Ձ஋൓෮๏
▸ ໋୊2.8ʢಈతܭը๏ͷޡࠩݶքʣ͔Β
▸ ऴྃ৚݅ͷᮢ஋ ͱٻ·ΔՁ஋ؔ਺ɼํࡦͷ࣭ʹ͍ͭͯؔ܎ੑ
Λ֬ೝ
▸ ޡࠩݶքͷࣜ(2.32)ΛҎԼͷΑ͏ʹॻ͖׵͑Δɽ
▸ ऴྃ৚݅ͷᮢ஋ʹ͍ͭͯఆٛΛ༻͍Ε͹ҎԼͷΑ͏ʹมܗՄೳ
▸ ࠷దՁ஋ؔ਺ ͱਪఆՁ஋ؔ਺ ͷ࠷େޡࠩ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

35. 2.3.1 Ձ஋൓෮๏
▸ ໋୊2.8ʢಈతܭը๏ͷޡࠩݶքʣ͔Β
▸ ࣜ(2.34)ΑΓऴྃ൑ఆʹ༻͍Δ ͸ࢦ਺ݮਰ͠ॳظ஋͕༗քͰ
͋ΔͨΊՁ஋൓෮๏͸ඞͣ༗ݶͷճ਺Ͱऴྃ͢Δɽ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

36. 2.3.1 Ձ஋൓෮๏
▸ ໋୊2.8ʢಈతܭը๏ͷޡࠩݶքʣ͔Β
▸ ํࡦ ʹର͢ΔํࡦޡࠩΛҎԼͱఆٛ
▸ Ձ஋൓෮๏Ͱٻ·Δํࡦ ʹରͯ͠ϕϧϚϯ࡞༻ૉͷఆ͔ٛΒ
ҎԼ͕੒Γཱͭɽʢ΄Μͱ͔ʁʣ
▸ ࣜ(2.31)΍(2.36)Λ༻͍ͯม׵͢Δ͜ͱͰҎԼ͕੒Γཱͭɽ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

37. 2.3.1 Ձ஋൓෮๏
▸ ໋୊2.8ʢಈతܭը๏ͷޡࠩݶքʣ͔Β
▸ ษڧձதͰࢀՃऀͷํ͕ࣔͯͩͬͨ͘͠͞(2.40)ͷิ଍
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

38. 2.3.1 Ձ஋൓෮๏
▸ ໋୊2.8ʢಈతܭը๏ͷޡࠩݶքʣ͔Β
▸ ΑͬͯํࡦޡࠩΛᮢ஋ Λ༻͍ͯҎԼͷΑ͏ʹ཈͑Δɽ
▸ ্ࣜதͷೋஈ໨͔Βࡾஈ໨͸(2.32)ͱ໋୊2.8 b.ଆͷ(2.32)ͱࣅ
͔ͨࣜΒߦ͚ͦ͏ɽ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

39. 2.3.1 Ձ஋൓෮๏
▸ ໋୊2.8ʢಈతܭը๏ͷޡࠩݶքʣ͔Β
▸ ·ͨՁ஋൓෮๏ͷऴྃ৚݅ΛҎԼͰஔ͖׵͑Δ͜ͱΛߟ͑Δɽ
▸ ্৚݅͸ҎԼͷΑ͏ʹॻ͖׵͑ΒΕΔɽ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

40. 2.3.1 Ձ஋൓෮๏
▸ ໋୊2.8ʢಈతܭը๏ͷޡࠩݶքʣ͔Β
▸ ํࡦޡࠩΛ࣍ͷΑ͏ʹ཈͑Δɽ
▸ ࣜ(2.45)Ͱߋʹ্͔Β཈͑ͯ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

41. 2.3.1 Ձ஋൓෮๏
▸ ໋୊2.8ʢಈతܭը๏ͷޡࠩݶքʣ͔Β
▸ ํࡦޡࠩͷҎԼͷࣜͷ ͸ ʁ
▸ p.60 (2.47)ࣜ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

42. 2.3.1 Ձ஋൓෮๏
▸ ໋୊2.8ʢಈతܭը๏ͷޡࠩݶքʣ͔Β
▸ ϦλʔϯͷׂҾ཰ ͕ҟͳΔ৔߹
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
※ਤதॎͷࠇઢ͸൓෮ऴྃճ਺N

43. 2.3.1 Ձ஋൓෮๏
▸ ໋୊2.8ʢಈతܭը๏ͷޡࠩݶքʣ͔Β
▸ ऴྃ৚͕݅ҟͳΔ৔߹
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
※ਤதॎͷࠇઢ͸൓෮ऴྃճ਺N

44. 2.3.2 ํࡦ൓෮๏
▸ ΞϧΰϦζϜ2.2
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

45. 2.3.2 ํࡦ൓෮๏
▸ ໋୊2.9ʢํࡦվળͷ୯ௐੑʣ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

46. 2.4 ઢܗܭը๏ʹΑΔղ๏
▸ ϓϥϯχϯά໰୊Λղ͘͜ͱ͕ग़དྷΔɽஞ࣍తҙࢥܾఆ໰୊ͷ
ੑ࣭Λௐ΂Δ্Ͱ΋༗༻ɽ
▸ ͳ͓ɼઢܗܭը๏ʹ͓͚Δجຊతͳࣄ߲͸ิ଍A.3(p.266)Λࢀরɽ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

47. 2.4.1 ओ໰୊ͷಋग़
▸ ໋୊2.10ʢ࠷దՁ஋ؔ਺ͷ্քͱԼքʣ
▸ Կ౓΋ద༻͢Δ͜ͱͰۃݶΛऔΔ͚ͩɽ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2

48. 2.4.1 ओ໰୊ͷಋग़
▸ ໋୊2.10ʢ࠷దՁ஋ؔ਺ͷ্քͱԼքʣ͔Β
▸ ͜ͷ৚݅Λຬͨ͢࠷খͷ Λߟ͑Δ͜ͱͰ࠷దՁ஋ؔ਺ͷ্
քͱͳΔؔ਺Λߟ͑Δ͜ͱ͕ग़དྷΔɽ
▸ ॏΈؔ਺ Λߟ͑ɼҎԼͰઢܗܭը໰୊ʹఆࣜԽ
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2