Share
Embed
ChainerRLとマインクラフトで深層強化学習ハンズオン
DLLAB Engineer Days Day1: Hands-onのChainerRLとマインクラフトで深層強化学習ハンズオン用資料。
keisuke umezawa
October 06, 2019
Transcript
56 slides
-
Chainer ॳֶऀ͚ϋϯζΦϯ chug (Chainer User Group) twitterͷϋογϡλά #chug_jp #dllab Ͱ͓ئ͍͠·͢ɻ
࣭ɺSlack chainer-jp ͷ #chainer-handson-ms Ͱɻ ొ͕·ͩͷํɺhttps://bit.ly/chainer-jp-slack -
͝ڠྗͷ͓ئ͍ !2 • Slack chainer-jpʹ͝ొ͓ئ͍͠·͢ʂ • ࠓճɺ࣭Slack chainer-jp ͷ #chainer-handson-ms
Ͱड͚͚͍ͭͨͱࢥ͍·͢ɻ • ߨٛதɺνϡʔλʔͷํ͕ճ͠·͢ɻ • ొ͕·ͩͷํɺhttps://bit.ly/join-chainer-jp-slack • SNSߘͷͨΊࣸਅࡱӨ͠·͢ʂ • ࣸਅʹ͏ͭΓͨ͘ͳ͍ํ͍Βͬ͠Ό͍·ͨ͠Βݴ͍ͬͯ ͚ͨͩΔͱ͋Γ͕͍ͨͰ͢ɻ -
Slack chainer-jpʹ͝ొ͓ئ͍͠·͢ʂ !3 • ࠓճɺ࣭Slack chainer-jp ͷ #chainer-handson-ms Ͱ ड͚͚͍ͭͨͱࢥ͍·͢ɻ
• ߨٛதɺνϡʔλʔͷํ͕ճ͠·͢ɻ • ొ͕·ͩͷํɺhttps://bit.ly/chainer-jp-slack -
ࣗݾհ !4 • കᖒ ܚհ • ۚ༥Ϛʔέοτ༧ଌϞσϧͷݚڀ։ൃ @AlpacaJapan • Chainer
Evangelist ˏPreferred Networks -
Agenda 1. Chainer/CuPyͷհ 2. ຊͷϋϯζΦϯͷ༰ 1. ChainerRLʹ͍ͭͯ 2. MineRLʹ͍ͭͯ 3.
ChainerRL+MineRLͷڥઃఆ 4. MineRLʹ͍ͭͯͬͱ !5 -
CuPy !9 Chainerʹ͓͚ΔGPUܭࢉΛશͯ୲͢ΔϥΠϒϥϦ͕ಠཱ NumPyޓAPIͰίετʹCPUίʔυΛGPUҠߦ ಛҟղͳͲͷઢܗΞϧΰϦζϜΛGPU࣮ߦ KMeans, Gaussian Mixture ModelͳͲͷExampleͷॆ࣮ import
numpy as np x = np.random.rand(10) W = np.random.rand(10, 5) y = np.dot(x, W) import cupy as cp x = cp.random.rand(10) W = cp.random.rand(10, 5) y = cp.dot(x, W) GPU https://github.com/cupy/cupy -
࠷ۙͷͰ͖͝ͱ(2): ChainerX ެ։ • ࣗಈඍՄೳͳNumPy-likeͳ ndarrayΛC++Ͱ࣮͢Δ͜ͱ ʹΑΓҎԼΛୡ • ߴԽ •
ଟ༷ͳσόΠεʹରԠ • pythonҎ֎ͷݴޠͰdeploy Մೳ !12 -
Agenda 1. Chainer/CuPyͷհ 2. ຊͷϋϯζΦϯͷ༰ 1. ChainerRLʹ͍ͭͯ 2. MineRLʹ͍ͭͯ 3.
ChainerRL+MineRLͷڥઃఆ 4. MineRLʹ͍ͭͯͬͱ !14 -
ࠓճ༻͢ΔΞϧΰϦζϜ • DDQN (Double Deep Q Network) • Rainbow •
PPO (Proximal Policy Optimization) -
w 2018/11/25 ਂڧԽֶशΛνϣοτৄ͘͠ w %FFQ2/FUXPSL %2/ w
ڧԽֶशͰɺঢ়ଶʹର͢Δ֤ߦಈͷධՁΛ2ͱݺ ͼɺͦΕΛ༻͍Δख๏Λ2ֶशͱݺͿ w 2Λදݱ͢ΔؔΛ2ؔͱݺͼɺͦΕΛ%FFQ/FVSBM /FUXPSLͰදݱ͢ΔͷͰ%FFQ2/FUXPSLͱ͍͏ɻ ঢ়ଶ Convolutional Neural Network ߦಈධՁ લਐ ޙୀ ӈճ ࠨճ -
w 2018/11/25 %2/ͷ w &YQFSJFODF3FQMBZ w FYQFSJFODFʹੵͨ݁͠ՌΛϥϯμϜͰֶशσʔλʹ͔ͭ͏ Experience
(ঢ়ଶ , ߦಈ , ใु) , ޙୀ , -0.1 , ӈճ , -0.1 , લਐ , 1.0 , લਐ , -0.1 Replay -
w 2018/11/25 ڧԽֶशͷશମͷྲྀΕ Experience , ޙୀ , -0.1
, ӈճ , -0.1 , લਐ , 1.0 , લਐ , -0.1 Replay Qؔ ڥ ঢ়ଶ ߦಈ ใु -
w 2018/11/25 %PVCMF%2/ • QؔΛߦಈબͱঢ়ଶධՁʹ༻͢Δ͕ɺͦΕΛผʑͷωοτ ϫʔΫͰֶशͤͯ͞DQN • ৄ͍͠εϥΠυ
• https://www.slideshare.net/juneokumura/dqnrainbow -
w 2018/11/25 3BJOCPX • ਂڧԽֶशͷ༷ʑͳख๏͕ఏҊ͞Ε͖͕ͯͨɺͦΕΛΈ߹ ΘͤͨΒͲ͏ͳΔ͔ͱ͍͏จʮRainbow: Combining Improvements
in Deep Reinforcement Learningʯ͕ݩ • ৄ͍͠εϥΠυ • https://www.slideshare.net/juneokumura/dqnrainbow -
Agenda 1. Chainer/CuPyͷհ 2. ຊͷϋϯζΦϯͷ༰ 1. ChainerRLʹ͍ͭͯ 2. MineRLʹ͍ͭͯ 3.
ChainerRL+MineRLͷڥઃఆ 4. MineRLʹ͍ͭͯͬͱ !25 -
MineRL ʹ͍ͭͯ • Microsoft ։ൃͷ Open AI gym ޓ Minecraft
ڥ • “mission ϑΝΠϧ” ʹΑΔλεΫఆٛ • multi-agent ϓϩτίϧαϙʔτ • MalmÖ Λར༻͠ɺಉظɾ҆ఆɾߴԽ • ಛʹಉظॏཁ -- 1step ͣͭ action ͍ͨ͠ʂ • ֤λεΫʹରԠ͢Δσʔληοτఏڙ MalmÖ MineRL -
MineRL ʹ͍ͭͯ • μΠϠϞϯυͷೖख͔ͳΓқͷߴ͍λεΫ • ͍͔ͭ͘ͷதؒλεΫ͕ఏڙ͞ΕΔɿ • MineRLTreechop • MineRLNavigate
• MineRLObtainIronPickaxe • MineRLObtainDiamond -- ͜Ε͕࠷ऴλεΫ Navigate ʹ Extreme ൛†ɺNavigate/Obtain* ʹ Dense ൛‡ఏڙ͞ΕΔ †: “extreme” ͳܗͰ։࢝ ‡: ใु͕ΑΓີʹ༩͑ΒΕΔ “ิॿ” λεΫͨͪ -
MineRL ʹ͍ͭͯ • 64ݸͷʮݪ (log)ʯΛूΊΔ • ݪ Minecraft ͷΩʔϦιʔε •
(=पΓʹ͕ଟ͍) Ͱɺ మͷ佁Λ࣋ͬͨঢ়ଶ͔Βελʔτ • log Λೖख͢Δ͝ͱʹ +1 reward MineRLTreechop -
MineRL ʹ͍ͭͯ • ࢦఆ͞Εͨΰʔϧ͔͏ • Minecraft Ͱ࠷جຊͱͳΔεΩϧ • “compass” Λ؍ଌՄೳʀΰʔϧํΛࢦࣔͯ͘͠ΕΔ
• ΰʔϧ౸ୡͰ +100 reward • “Dense” ൛ͰߋʹΰʔϧʹͱͷڑʹԠͯ͡ຖεςοϓใ ुΛ͑Δ (ԕ͔͟Δͱෛͷใु) MineRLNavigate -
MineRL ʹ͍ͭͯ • μΠϠϞϯυΛೖख͢Δ • Minecraft Ͱ࠷وॏͳΞΠςϜͷͻͱͭ • ଟ͘ͷதؒΞΠςϜ͕ඞཁͱͳΔ •
μΠϠϞϯυೖखͰ +1024 reward • தؒΞΠςϜೖखͰ (খ͞Ίͷ) ใु • σϑΥϧτͰΞΠςϜͷ1ճͷೖख࣌ͷΈใु͕͑Δ “Dense” ൛ͰԿͰ͑Δ MineObtainDiamond -
Agenda 1. Chainer/CuPyͷհ 2. ຊͷϋϯζΦϯͷ༰ 1. ChainerRLʹ͍ͭͯ 2. MineRLʹ͍ͭͯ 3.
ChainerRL+MineRLͷڥઃఆ 4. MineRLʹ͍ͭͯͬͱ !33 -
Agenda 1. Chainer/CuPyͷհ 2. ຊͷϋϯζΦϯͷ༰ 1. ChainerRLʹ͍ͭͯ 2. MineRLʹ͍ͭͯ 3.
ChainerRL+MineRLͷڥઃఆ 4. MineRLʹ͍ͭͯͬͱ !35 -
The MineRL Competition • ڧԽֶशͷݚڀۙେ͖͘ਐา͍ͯ͠ΔҰํͰɺ ཁٻ͞ΕΔαϯϓϧ͕૿Ճ • Ұ෦ͷڊେAIاۀҎ֎ʹΑΔݚڀ͕͍͠ • ࣮ੈքԠ༻͕͍͠
• αϯϓϧޮͷྑ͍ RL γεςϜͷ։ൃΛڝ͏ • ͨͩ͠ɺਓؒͷσϞσʔλ (σʔληοτ) ͕ ར༻Մೳͱ͢Δ ՝ MineRL Competition -
MineRL ʹ͍ͭͯͬͱ • ֤λεΫʹରԠ͢ΔσϞϯετϨʔγϣϯ (σʔληοτ) ͕ఏ ڙ͞ΕΔ • ਓؒʹΑΔϓϨΠσʔλ •
λεΫ͝ͱʹͦΕͧΕ100~250ݸఔͣͭ༩͑ΒΕΔ • ڥͱͷ interaction ճ੍ݶ͞Ε͍ͯΔ͕ɺ σʔληοτ (࣌ؒͷڐ͢ݶΓ) ԿͰͬͯྑ͍ σʔληοτ -
MineRL ʹ͍ͭͯͬͱ • ୯ҰͷΩʔೖྗͰͳ͘ɺҙຯͷ͋Δʮߦಈʯ୯ҐͰఆٛ͞ΕΔ • attack, camera, forward, craft, place,
… • ֤ߦಈಉ࣮࣌ߦՄೳ (gym ͷ Dict space ͱͯ͠ఆٛ) ͕ͩɺ ૬͢Δߦಈ͋Δ͜ͱʹҙ (forward ͱ back ͳͲ) • ิॿλεΫͰ͍͔ͭ͘ͷߦಈ͕ෆՄೳʹͳ͍ͬͯΔ͜ͱ͕͋Δ • ྫ͑ Treechop Ͱ craft place Ͱ͖ͳ͍ (ඞཁͳ ͍) action ʹ͍ͭͯ -
MineRL ʹ͍ͭͯͬͱ • observation: • pov (Ұਓশࢹͷը૾) • 64x64x3 (uint8)
ͷ numpy array • inventory (ॴ࣋ΞΠςϜ) • dirt ͱ͔ log ͱ͔ iron_ore ͱ͔ • equipped_items (උ͍ͯ͠Δಓ۩) • wooden_axe ͱ͔ iron_pickaxe ͱ͔ • compassAngle (ΰʔϧํͷࢦࣔ) • Navigate* ͰͷΈར༻Մೳ • action ಉ༷ɺิॿλεΫ͝ͱʹಘΒΕΔ؍ଌҟͳΔ (pov ඞͣ͋Δ) observation ʹ͍ͭͯ -
MineRL ʹ͍ͭͯͬͱ >>> import gym >>> import minerl # ͪΐͬͱ͔͔࣌ؒΔ
(Minecraft ͷىಈ͕ޙΖͰΔ) >>> env = gym.make('MineRLObtainDiamond-v0') جຊతʹ Gym API ४ڌ = ੈͷதͷ RL πʔϧ͕͍͍ͩͨಈ͘ -
MineRL ʹ͍ͭͯͬͱ >>> obs, info = env.reset() # info Ұॹʹฦͬͯ͘Δɻͪΐͬͱ
Gym API ͱҧ͏ >>> obs {'equipped_items': {'mainhand': {'damage': 0, 'maxDamage': 0, 'type': 0}}, 'inventory': {'coal': 0, 'cobblestone': 0, ... }, 'pov': array([[[ 0, 0, 0], [ 16, 32, 9], ..., [ 75, 91, 118]], dtype=uint8)} >>> info {} -
MineRL ʹ͍ͭͯͬͱ >>> action = env.action_space.sample() # దͳ action Λऔಘ
>>> action OrderedDict([('attack', 0), ('back', 1), ('camera', array([ 39.44639 , -77.577675], dtype=float32)), ('craft', 3), ... ('nearbySmelt', 0), ('place', 3), ('right', 1), ('sneak', 0), ('sprint', 1)]) -
MineRL ʹ͍ͭͯͬͱ >>> obs, reward, done, info = env.step(action) >>>
obs (ུ) >>> reward 0.0 >>> done False >>> info {} -
MineRL tutorial with ChainerRL • PFN ։ൃͷ ChainerRL Λར༻ͨ͠ϕʔεϥΠϯΛఏڙ •
https://github.com/minerllabs/quick_start/tree/master/ chainerrl_baselines • “starter kit” ͱͯ͠ɺ • ڥ (σʔληοτ) ΛͲ͏ಈ͔ͤྑ͍͔ • ࠷ॳʹऔΔ͖φΠʔϒͳΞϓϩʔν ͳͲʹ͍ͭͯͷऔֻ͔ͬΓΛఏڙ ϕʔεϥΠϯ https://github.com/minerllabs/baselines/tree/master/general/chainerrl -
MineRL tutorial with ChainerRL • Treechop/Navigate ΛφΠʔϒʹղ͘ • σʔληοτΛແࢹ •
action/observation space ʹڧ͍ prior ΛԾఆ φΠʔϒͳΞϓϩʔν Treechop Navigate NavigateDense training reward training episode https://github.com/minerllabs/baselines/tree/master/general/chainerrl -
MineRL tutorial with ChainerRL • Treechop ͱ NavigateDense ΫϦΞͰ͖Δ •
Navigate Ͱ͖͍ͯͳ͍ → Obtain* λεΫ͜ΕΑΓ͍͠ʂ φΠʔϒͳΞϓϩʔν Treechop Navigate NavigateDense https://github.com/minerllabs/baselines/tree/master/general/chainerrl -
MineRL tutorial with ChainerRL • action space ʹڧ͍ prior ΛԾఆ͍ͯ͠Δ
→ ͜ͷ··ͰଞͷλεΫʹస༻Ͱ͖ͳ͍͜ͱʹҙ • Treechop • 5࣍ݩͷࢄߦಈʹม i. {'forward': 1, 'jump': 0, 'camera': [0, 0]} ii.{'forward': 0, 'jump': 0, 'camera': [0, 0]} iii.{'forward': 1, 'jump': 1, 'camera': [0, 0]} iv.{'forward': 1, 'jump': 0, 'camera': [0, -10]} v. {'forward': 1, 'jump': 0, 'camera': [0, 10]} attack ৗʹ onɺͦΕҎ֎ৗʹ off action space ʹ͍ͭͯ https://github.com/minerllabs/baselines/tree/master/general/chainerrl -
MineRL tutorial with ChainerRL • Navigate/NavigateDense • 4࣍ݩͷࢄߦಈʹม i. {'jump':
0, 'camera': [0, 0]} ii.{'jump': 1, 'camera': [0, 0]} iii.{'jump': 0, 'camera': [0, -10]} iv.{'jump': 0, 'camera': [0, 10]} forward, sprint, attack ৗʹ onɺͦΕҎ֎ৗʹ off action space ʹ͍ͭͯ https://github.com/minerllabs/baselines/tree/master/general/chainerrl -
MineRL tutorial with ChainerRL • Double Dueling DQN (উखʹུͯ͠ݺΜͰΔ͚ͩ) •
DQN ͷ͍ΖΜͳੜΛ٧ΊࠐΜͩख๏ • Proximal Policy Optimization • ࿈ଓۭؒͰఆٛ͞ΕͨߦಈΛࣗવʹѻ͑Δ͕ɺ ϕʔεϥΠϯͰࢄԽͨ͠ߦಈΛར༻ DDDQN Rainbow PPO https://github.com/minerllabs/baselines/tree/master/general/chainerrl -
MineRL tutorial with ChainerRL • Ҏ্ͷࣄલࣝͷͱɺϋΠύʔύϥϝʔλΛదʹνϡʔχϯ άͯ͠ಘΒΕͨ݁Ռ͕Լਤ (࠶ܝ) Treechop Navigate
NavigateDense training reward training episode https://github.com/minerllabs/baselines/tree/master/general/chainerrl -
MineRL tutorial with ChainerRL action space ͷ੍ (ಛʹ camera Λ্Լʹಈ͔ͤͳ͍)
ͷͨΊɺ ਓ͔ؒΒݟΔͱ࠷దͱݴ͑ͳ͍ͷͷɺ ͦͷൣғͰʮλεΫΛΫϦΞʯͰ͖͍ͯΔʁ MineRLTreechop https://github.com/minerllabs/baselines/tree/master/general/chainerrl -
͞Βʹֶश͢Δʹɾɾɾ • ͍ͭઌɺbaselinesʹ฿ֶशͷΞϧΰϦζϜՃ͞Εͨ • behavoral cloning • GAIL • DQfD
• ৄ͍͠આ໌͜ͷεϥΠυͰ • https://www.slideshare.net/pfi/minerl-competition-tutorial-with- chainerrl-156927429 https://github.com/minerllabs/baselines/tree/master/general/chainerrl -
Chainer User Groupͷհ • Slack chainer-jp • Twitter @chug_jp •
׆ಈ༰ • MeetupɾϋϯζΦϯͷ։࠵ • Chainerʹؔ͢ΔใɾυΩϡϝϯτͷ֦ॆ • Web νϡʔτϦΞϧͷ࡞ • Ұॹʹ׆ಈͰ͖ΔํΛืू͍ͯ͠·͢ʂ Slack #chug-jp-management ʹੋඇ !54