「強化学習」輪読会資料
by
icebee16
Link
Embed
Share
Beginning
This slide
Copy link URL
Copy link URL
Copy iframe embed code
Copy iframe embed code
Copy javascript embed code
Copy javascript embed code
Share
Tweet
Share
Tweet
Slide 1
Slide 1 text
ڧԽֶश ྠಡձ #02, #03 @icebee__ CHAPTER 2 ϓϥϯχϯά
Slide 2
Slide 2 text
ڧԽֶश ྠಡձ #02, #03 CHAPTER 2 ABOUT ME @icebee__ •ۚ༥ܥۈΊͷࣾձਓೋ •ڵຯͱͯ͠σʔλੳػցֶशɼ౷ܭؔ࿈ͷ •Kaggle Expert
Slide 3
Slide 3 text
CONTENTS Chapter 2 ϓϥϯχϯά 2.1 ४උ 2.1.1 తؔ 2.1.2 ࠷దՁؔ 2.2 ಈతܭը๏ 2.2.1 ϕϧϚϯ࡞༻ૉ 2.2.2 ϕϧϚϯ࡞༻ૉͷ୯ௐੑ 2.2.3 ಈతܭը๏ͷཧ 2.2.4 ࠷దํࡦ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 4
Slide 4 text
CONTENTS Chapter 2 ϓϥϯχϯά 2.3 ಈతܭը๏ʹΑΔղ๏ 2.3.1 Ձ෮๏ 2.3.2 ํࡦ෮๏ 2.4 ઢܗܭը๏ʹΑΔղ๏ 2.4.1 ओͷಋग़ 2.4.2* ର (ࠓճऔΓѻΘͳ͍) 2.4.3* ࠷దղͱ࠷దํࡦ (ࠓճऔΓѻΘͳ͍) ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 5
Slide 5 text
2.1 ४උ ▸ ϓϥϯχϯά ➡ ڥ͕طͷ߹ͷஞ࣍తҙࢥܾఆ ➡ ڥ͕ະͷ߹ʹ͋ͨΔڧԽֶशΛऔΓѻ͏্Ͱͷج ૅͱͳΔ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 6
Slide 6 text
2.1.1 తؔ ▸ ҎԼͷతؔΛ༻ҙ͢Δ ▸ ্͔ࣜΒಋ͔ΕΔ ॳظঢ়ଶ ʹґଘ͢ΔՄೳੑ͕ߟ͑ΒΕΔ͕ ޙड़(2.2અ)͢Δ໋ΑΓҙͷॳظঢ়ଶʹ͓͍ͯҎԼཱ͕͢ Δɽ ▸ ΑͬͯఆৗతͳํࡦͷΈΛߟ͑Δ͜ͱͰతؔΛ࠷େԽग़དྷΔɽ ͨͩ͠ɼ ୈmཁૉ͕1Ͱଞͷཁૉθϩͷn࣍ݩϕΫτϧ ՁؔͷॏΈ͖ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 7
Slide 7 text
2.1.2 ࠷దՁؔ ▸ ҎԼͷࣜʹͯ࠷దՁؔ(optimal value function)Λಋೖ ▸ Ձؔ Λ༻͍ͯҎԼͷ༷ʹهड़Մೳ ▸ ͜ΕΑΓ Ϛϧίϑํࡦͷेੑʹؔ͢ΔܥΛຬͨ͠ɼ࠷ద Ձؔʹ͍ͭͯҎԼͷ͕ࣜΓཱͭɽ ॻ੶ࢀߟϖʔδɿp.22 ಉ࣌पล֬ɿ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 8
Slide 8 text
2.1.2 ࠷దՁؔ ▸ ࠷దՁؔҎԼͷΑ͏ʹॻ͖ԼͤΔɽ ▸ ͜ΕΑΓϕϧϚϯ࠷దํఔࣜ(Bellman optimality equation)ͱݺ ΕΔҎԼͷ͕ࣜΓཱͭɽ Ϧλʔϯͷ࠶ؼੑɿ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 9
Slide 9 text
2.1.2 ࠷దՁؔ ▸ ࠷దՁؔҎԼͷΑ͏ʹॻ͖ԼͤΔɽ ▸ ͜ΕΑΓϕϧϚϯ࠷దํఔࣜ(Bellman optimality equation)ͱݺ ΕΔҎԼͷ͕ࣜΓཱͭɽ Ϧλʔϯͷ࠶ؼੑɿ ભҠ֬ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 10
Slide 10 text
2.2 ಈతܭը๏ ▸ ಈతܭը๏ ➡ ஞ࣍తҙࢥܾఆͷ࠷దԽΛղ͘Ξϓϩʔνͷ͜ͱ ➡ ͨͩ͠ɼ۩ମతͳख๏Λࢦ͢ͷͰͳ͘લड़ͷΑ͏ͳΞϓ ϩʔνͷ૯শ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 11
Slide 11 text
2.2.1 ϕϧϚϯ࡞༻ૉ ▸ ಈతܭը๏ͰϕϧϚϯ࡞༻ૉͱݺΕΔҎԼͷॲཧ Λؔ ʹ܁Γฦ͠ద༻͢Δ͜ͱͰՁؔɼ࠷దՁؔʹ͚ۙͮΔɽ ▸ ·ͣɼ࣍ͷ̎ͭͷҙͷؔʹର͢ΔϕϧϚϯ࡞༻ૉΛಋೖ͢ Δɽ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 12
Slide 12 text
2.2.1 ϕϧϚϯ࡞༻ૉ ▸ ϕϧϚϯظ࡞༻ૉ ▸ ϕϧϚϯ࠷ద࡞༻ૉ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 13
Slide 13 text
2.2.1 ϕϧϚϯ࡞༻ૉ ▸ ·ͨɼՃ͑ͯҎԼͷ͍͔ͭ͘ͷه๏Λಋೖ͢Δɽ ▸ ؆ศԽ ▸ ஞ࣍తͳద༻ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 14
Slide 14 text
2.2.1 ϕϧϚϯ࡞༻ૉ ▸ ઌΜͯࣔͨ͡͠ϕϧϚϯ࡞༻ૉͦΕͧΕϕϧϚϯํఔࣜͱҎԼ ͷؔΛ࣋ͭɽ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 15
Slide 15 text
2.2.2 ϕϧϚϯ࡞༻ૉͷ୯ௐੑ ▸ ิ2.1ʢ୯ௐੑͷิʣ ▸ ҙͷؔͷʹରͯ͠ϕϧϚϯ࡞༻ૉΛద༻ͯͦ͠ͷେখ ؔอͨΕΔɽ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 16
Slide 16 text
2.2.2 ϕϧϚϯ࡞༻ૉͷ୯ௐੑ ▸ ҎԼͷදهΛఆٛ͢Δɽ ▸ ͜ͷͱ͖ϕϧϚϯ࡞༻ૉͷఆٛΑΓҎԼ͕ࣔͤΔ ▸ ิ2.2 ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 17
Slide 17 text
2.2.3 ಈతܭը๏ͷཧ ▸ ໋2.3ʢಈతܭը๏ͷऩଋੑʣ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 18
Slide 18 text
2.2.3 ಈతܭը๏ͷཧ ▸ ໋2.3ʢಈతܭը๏ͷऩଋੑʣূ໌ϝϞ ▸ a. ~ (2.14) ▸ ઌʹ্͛ͨิΛ͏·͘͏͚ͩɽ༗քͰ͋Δ͜ͱΛར༻ͯ͠ ҙͷఆͰڬΈɼk>∞ͰఆΛখ͘͢͞Δ͚ͩɽ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 19
Slide 19 text
2.2.3 ಈతܭը๏ͷཧ ▸ ໋2.3ʢಈతܭը๏ͷऩଋੑʣূ໌ϝϞ ▸ b. ~ (2.16) ▸ ՁؔΛํࡦͷk൪Λڥʹׂ͢Δ͚ͩɽ ▸ (2.16) ~ (2.17) ▸ ใुؔͷఆٛ(1.1)ʹै֤ͬͯεςοϓͷใुΛ্͔Β͑Δɽ ▸ (2.16)ͷӈลೋ߲ʹ͍ͭͯղ͍ͯೖɽ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 20
Slide 20 text
2.2.3 ಈతܭը๏ͷཧ ▸ ໋2.3ʢಈతܭը๏ͷऩଋੑʣূ໌ϝϞ ▸ (2.17) Լ~ (2.18) ▸ ʢp.49ͷԼࣜE[]ೋ߲ɼγͱSͷఴࣈޡ২Ͱʁt > kʣ ▸ v(s)ͷظmin(v(s))ͱmax(v(s))ͰڬΊΔɽ ▸ ग़དྷͨࣜͱ(2.17)ͷࠩΛͱΔɽ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 21
Slide 21 text
2.2.3 ಈతܭը๏ͷཧ ▸ ໋2.4ʢϕϧϚϯํఔࣜͷղͷҰҙੑʣ ▸ എཧ๏Ͱ؆୯ʹূ໌ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 22
Slide 22 text
2.2.3 ಈతܭը๏ͷཧ ▸ ิ2.5ʢϕϧϚϯ࡞༻ૉͷॖখੑʣ ▸ ূ໌͔Βೖͬͨ΄͏͕ཧղ͕ૣ͍ɽ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 23
Slide 23 text
2.2.3 ಈతܭը๏ͷཧ ▸ ิ2.5ʢϕϧϚϯ࡞༻ૉͷॖখੑʣূ໌ϝϞ ▸ (2.21), (2.22), (2.23) ▸ ิ2.2͋ͨΓΛ༻͍ͯڬΜͰࣔ͢ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 24
Slide 24 text
2.2.3 ಈతܭը๏ͷཧ ▸ ิ2.5ʢϕϧϚϯ࡞༻ૉͷॖখੑʣ͔Β ▸ (2.23)ͷv’ʹରͯ͠V*ΛೖɼϕϧϚϯ࠷దԽ࡞༻ૉͷෆಈ͕ V*Ͱ͋Δ͜ͱΛ౿·͑ΕҎԼͷཱ͕ࣜ ▸ ͜ΕϕϧϚϯ࠷దԽ࡞༻ૉΛҙͷؔʹదԠ͢Δ͜ͱͰ࠷ద Ձؔͱͦͷؔͷڑ͕γͷൺͰࢦؔతʹখ͘͞ͳͬ ͍ͯ͘͜ͱΛࣔ͢ɽ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 25
Slide 25 text
2.2.4 ࠷దํࡦ ▸ ఆٛ2.6ʢ࠷దํࡦʣ ▸ ͋Δঢ়ଶsͷظϦλʔϯΛ࠷େʹ͢Δ͜ͱՄೳ͕ͩҙͷঢ় ଶsʹରͯ͜͠ΕΛݴ͏ͷ͍͠ɼͱ͍͏͚ͩɽ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 26
Slide 26 text
2.2.4 ࠷దํࡦ ▸ ໋2.7ʢ࠷దํࡦͷଘࡏੑͱඞཁे݅ʣ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 27
Slide 27 text
2.2.4 ࠷దํࡦ ▸ ໋2.7ʢ࠷దํࡦͷଘࡏੑͱඞཁे݅ʣূ໌ϝϞ ▸ ඞཁ݅ɼे݅ ▸ ʹରͯ͠ϕϧϚϯظ࡞༻ૉΛߟ͑ͨͱ͖ɼ͜ͷෆಈ ิ2.4͔ΒҰҙʹ ͕ͩɼ(2.28)ΑΓ Ͱ͋ΔɽΑͬͯ ͦΕΒҰக͍ͯ͠Δ ▸ ଘࡏ͢Δ͜ͱΛࣔ͢ ▸ ਖ਼Α͘Θ͔͍ͬͯͳ͍ɽ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 28
Slide 28 text
2.2.4 ࠷దํࡦ ▸ ໋2.7ʢ࠷దํࡦͷଘࡏੑͱඞཁे݅ʣ͔Β ▸ ͷఆٛʹ ΛೖΕͯ͋͛ΕҎԼ͕ݴ͑Δ ▸ ॏΈ͖Ձؔͷఆ͔ٛΒҎԼ ▸ ͢ͳΘͪɼ࠷దํࡦͷֶशͱతؔ Λ࠷େʹ ͢Δํࡦͷ୳ࡧͰ͋Δͱ͍͑Δɽ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 29
Slide 29 text
2.2.4 ࠷దํࡦ ▸ લड़ͨࣜ͠ʹ͍ͭͯ ͱ ͱͳ͍ͬͯΔཧ༝ͱͯ͠ɼ = 0ͱ ͳ͍ͬͯΔঢ়ଶɼ͢ͳΘͪભҠ͢ΔՄೳੑͷͳ͍ঢ়ଶؒͷύε· Ͱ࠷దԽͷରʹ͢Δඞཁͳ͍͔Βɽ ▸ ໋2.7ͷ݁ՌҎԼͷೋΑΓॏཁ ▸ ࠷దԽରͷํࡦू߹Λߟ͑Δʹ͋ͨͬͯ୳ࡧൣғఆৗͳܾ ఆతํࡦͷू߹ͷΈͰͳ͍͜ͱΛอূͨ͜͠ͱɽ ▸ ֶश͍ͯ͠ΔՁ͕ؔ࠷దՁؔͱҰக͢Δ͜ͱͰ࠷దํ ࡦ͔൱͔ΛఆͰ͖Δ͜ͱɽ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 30
Slide 30 text
2.3 ಈతܭը๏ʹΑΔղ๏ ▸ ಈతܭը๏ͷ۩ମతͳ࣮ͱͯ͠ҎԼͷ2ͭΛհ ➡ Ձ෮๏ʢvalue iteration algorithmʣ ➡ ํࡦ෮๏ʢpolicy iteration algorithmʣ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 31
Slide 31 text
2.3.1 Ձ෮๏ ▸ ΞϧΰϦζϜ2.1 ▸ ऩଋ͕อূ͞Ε͍ͯΔͷͰ୯७ͳ܁Γฦ͠ɽ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 32
Slide 32 text
2.3.1 Ձ෮๏ ▸ ໋2.8ʢಈతܭը๏ͷޡࠩݶքʣ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 33
Slide 33 text
2.3.1 Ձ෮๏ ▸ ໋2.8ʢಈతܭը๏ͷޡࠩݶքʣ ▸ ূ໌ͷৄࡉิA.1.1(p.250) ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 34
Slide 34 text
2.3.1 Ձ෮๏ ▸ ໋2.8ʢಈతܭը๏ͷޡࠩݶքʣ͔Β ▸ ऴྃ݅ͷᮢ ͱٻ·ΔՁؔɼํࡦͷ࣭ʹ͍ͭͯؔੑ Λ֬ೝ ▸ ޡࠩݶքͷࣜ(2.32)ΛҎԼͷΑ͏ʹॻ͖͑Δɽ ▸ ऴྃ݅ͷᮢʹ͍ͭͯఆٛΛ༻͍ΕҎԼͷΑ͏ʹมܗՄೳ ▸ ࠷దՁؔ ͱਪఆՁؔ ͷ࠷େޡࠩ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 35
Slide 35 text
2.3.1 Ձ෮๏ ▸ ໋2.8ʢಈతܭը๏ͷޡࠩݶքʣ͔Β ▸ ࣜ(2.34)ΑΓऴྃఆʹ༻͍Δ ࢦݮਰ͠ॳظ͕༗քͰ ͋ΔͨΊՁ෮๏ඞͣ༗ݶͷճͰऴྃ͢Δɽ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 36
Slide 36 text
2.3.1 Ձ෮๏ ▸ ໋2.8ʢಈతܭը๏ͷޡࠩݶքʣ͔Β ▸ ํࡦ ʹର͢ΔํࡦޡࠩΛҎԼͱఆٛ ▸ Ձ෮๏Ͱٻ·Δํࡦ ʹରͯ͠ϕϧϚϯ࡞༻ૉͷఆ͔ٛΒ ҎԼ͕Γཱͭɽʢ΄Μͱ͔ʁʣ ▸ ࣜ(2.31)(2.36)Λ༻͍ͯม͢Δ͜ͱͰҎԼ͕Γཱͭɽ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 37
Slide 37 text
2.3.1 Ձ෮๏ ▸ ໋2.8ʢಈతܭը๏ͷޡࠩݶքʣ͔Β ▸ ษڧձதͰࢀՃऀͷํ͕ࣔͯͩͬͨ͘͠͞(2.40)ͷิ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 38
Slide 38 text
2.3.1 Ձ෮๏ ▸ ໋2.8ʢಈతܭը๏ͷޡࠩݶքʣ͔Β ▸ ΑͬͯํࡦޡࠩΛᮢ Λ༻͍ͯҎԼͷΑ͏ʹ͑Δɽ ▸ ্ࣜதͷೋஈ͔Βࡾஈ(2.32)ͱ໋2.8 b.ଆͷ(2.32)ͱࣅ ͔ͨࣜΒߦ͚ͦ͏ɽ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 39
Slide 39 text
2.3.1 Ձ෮๏ ▸ ໋2.8ʢಈతܭը๏ͷޡࠩݶքʣ͔Β ▸ ·ͨՁ෮๏ͷऴྃ݅ΛҎԼͰஔ͖͑Δ͜ͱΛߟ͑Δɽ ▸ ্݅ҎԼͷΑ͏ʹॻ͖͑ΒΕΔɽ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 40
Slide 40 text
2.3.1 Ձ෮๏ ▸ ໋2.8ʢಈతܭը๏ͷޡࠩݶքʣ͔Β ▸ ํࡦޡࠩΛ࣍ͷΑ͏ʹ͑Δɽ ▸ ࣜ(2.45)Ͱߋʹ্͔Β͑ͯ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 41
Slide 41 text
2.3.1 Ձ෮๏ ▸ ໋2.8ʢಈతܭը๏ͷޡࠩݶքʣ͔Β ▸ ํࡦޡࠩͷҎԼͷࣜͷ ʁ ▸ p.60 (2.47)ࣜ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 42
Slide 42 text
2.3.1 Ձ෮๏ ▸ ໋2.8ʢಈతܭը๏ͷޡࠩݶքʣ͔Β ▸ ϦλʔϯͷׂҾ ͕ҟͳΔ߹ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2 ※ਤதॎͷࠇઢ෮ऴྃճN
Slide 43
Slide 43 text
2.3.1 Ձ෮๏ ▸ ໋2.8ʢಈతܭը๏ͷޡࠩݶքʣ͔Β ▸ ऴ͕ྃ݅ҟͳΔ߹ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2 ※ਤதॎͷࠇઢ෮ऴྃճN
Slide 44
Slide 44 text
2.3.2 ํࡦ෮๏ ▸ ΞϧΰϦζϜ2.2 ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 45
Slide 45 text
2.3.2 ํࡦ෮๏ ▸ ໋2.9ʢํࡦվળͷ୯ௐੑʣ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 46
Slide 46 text
2.4 ઢܗܭը๏ʹΑΔղ๏ ▸ ϓϥϯχϯάΛղ͘͜ͱ͕ग़དྷΔɽஞ࣍తҙࢥܾఆͷ ੑ࣭ΛௐΔ্Ͱ༗༻ɽ ▸ ͳ͓ɼઢܗܭը๏ʹ͓͚Δجຊతͳࣄ߲ิA.3(p.266)Λࢀরɽ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 47
Slide 47 text
2.4.1 ओͷಋग़ ▸ ໋2.10ʢ࠷దՁؔͷ্քͱԼքʣ ▸ Կద༻͢Δ͜ͱͰۃݶΛऔΔ͚ͩɽ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2
Slide 48
Slide 48 text
2.4.1 ओͷಋग़ ▸ ໋2.10ʢ࠷దՁؔͷ্քͱԼքʣ͔Β ▸ ͜ͷ݅Λຬͨ͢࠷খͷ Λߟ͑Δ͜ͱͰ࠷దՁؔͷ্ քͱͳΔؔΛߟ͑Δ͜ͱ͕ग़དྷΔɽ ▸ ॏΈؔ Λߟ͑ɼҎԼͰઢܗܭըʹఆࣜԽ ڧԽֶश ྠಡձ #02, #03 CHAPTER 2