ChainerRLとマインクラフトで深層強化学習ハンズオン

Transcript

1. Chainer ॳֶऀ޲͚ϋϯζΦϯ
 chug (Chainer User Group) twitterͷϋογϡλά͸ #chug_jp #dllab Ͱ͓ئ͍͠·͢ɻ

   ࣭໰͸ɺSlack chainer-jp ͷ #chainer-handson-ms Ͱɻ ొ࿥͕·ͩͷํ͸ɺhttps://bit.ly/chainer-jp-slack

2. ͝ڠྗͷ͓ئ͍ !2 • Slack chainer-jpʹ͝ొ࿥͓ئ͍͠·͢ʂ • ࠓճɺ࣭໰͸Slack chainer-jp ͷ #chainer-handson-ms

   Ͱड͚͚͍ͭͨͱࢥ͍·͢ɻ • ߨٛத΋ɺνϡʔλʔͷํ͕ճ౴͠·͢ɻ • ొ࿥͕·ͩͷํ͸ɺhttps://bit.ly/join-chainer-jp-slack • SNS౤ߘͷͨΊࣸਅࡱӨ͠·͢ʂ • ࣸਅʹ͏ͭΓͨ͘ͳ͍ํ͍Βͬ͠Ό͍·ͨ͠Βݴ͍ͬͯ ͚ͨͩΔͱ͋Γ͕͍ͨͰ͢ɻ

3. Slack chainer-jpʹ͝ొ࿥͓ئ͍͠·͢ʂ !3 • ࠓճɺ࣭໰͸Slack chainer-jp ͷ #chainer-handson-ms Ͱ ड͚͚͍ͭͨͱࢥ͍·͢ɻ

   • ߨٛத΋ɺνϡʔλʔͷํ͕ճ౴͠·͢ɻ • ొ࿥͕·ͩͷํ͸ɺhttps://bit.ly/chainer-jp-slack

4. ࣗݾ঺հ !4 • കᖒ ܚհ • ۚ༥Ϛʔέοτ༧ଌϞσϧͷݚڀ։ൃ @AlpacaJapan • Chainer

   Evangelist ˏPreferred Networks

5. Agenda 1. Chainer/CuPyͷ঺հ 2. ຊ೔ͷϋϯζΦϯͷ಺༰ 1. ChainerRLʹ͍ͭͯ 2. MineRLʹ͍ͭͯ 3.

   ChainerRL+MineRLͷ؀ڥઃఆ 4. MineRLʹ͍ͭͯ΋ͬͱ !5

6. Chainer/CuPyͷ঺հ !6

7. Chainer • Chainerͱ͸ (http://chainer.org/) • Preferred Networks੡Deep LearningϑϨʔϜϫʔΫ !7

8. Chainer !8 • Googleࣾ੡TensorFlowͳͲͷಉྨ

9. CuPy !9 Chainerʹ͓͚ΔGPUܭࢉΛશͯ୲౰͢ΔϥΠϒϥϦ͕ಠཱ NumPyޓ׵APIͰ௿ίετʹCPUίʔυΛGPU΁Ҡߦ ಛҟ஋෼ղͳͲͷઢܗ୅਺ΞϧΰϦζϜΛGPU࣮ߦ KMeans, Gaussian Mixture ModelͳͲͷExampleͷॆ࣮ import

   numpy as np x = np.random.rand(10) W = np.random.rand(10, 5) y = np.dot(x, W) import cupy as cp x = cp.random.rand(10) W = cp.random.rand(10, 5) y = cp.dot(x, W) GPU https://github.com/cupy/cupy

10. ֦େ͢ΔChainerϑΝϛϦʔ !10 Chainer UI Chainer Chemistry ڧԽֶश ը૾ೝࣝ ՄࢹԽ άϥϑߏ଄

    େن໛෼ࢄ Menoh ਪ࿦ಛԽ

11. ࠷ۙͷͰ͖͝ͱ(1): Chainer Tutorials ͷެ։ • Chainer͚ͩͰͳ͘ɺ਺ֶ΍PythonͷجૅʙػցֶशɾσΟʔϓϥʔ χϯάͷجૅɾίʔσΟϯάɾԠ༻·Ͱ෯޿͘ղઆɻ • શষΛColab্Ͱ࣮ߦՄೳͳJupyterϊʔτϒοΫͱͯ͠഑෍த !11

    https://tutorials.chainer.org/ja/

12. ࠷ۙͷͰ͖͝ͱ(2): ChainerX ެ։ • ࣗಈඍ෼ՄೳͳNumPy-likeͳ ndarrayΛC++Ͱ࣮૷͢Δ͜ͱ ʹΑΓҎԼΛୡ੒ • ߴ଎Խ •

    ଟ༷ͳσόΠεʹରԠ • pythonҎ֎ͷݴޠͰdeploy Մೳ !12

13. ຊ೔ͷϋϯζΦϯͷ಺༰ !13

14. Agenda 1. Chainer/CuPyͷ঺հ 2. ຊ೔ͷϋϯζΦϯͷ಺༰ 1. ChainerRLʹ͍ͭͯ 2. MineRLʹ͍ͭͯ 3.

    ChainerRL+MineRLͷ؀ڥઃఆ 4. MineRLʹ͍ͭͯ΋ͬͱ !14

15. ChainerRL Deep reinforcement learning library built on top of Chainer

16. ChainerRL • Chainer ʹΑΔ (ਂ૚) ڧԽֶशϑϨʔϜϫʔΫ • ڧԽֶशͷ༗໊ΞϧΰϦζϜͷ࠶ݱ࣮૷΋ؚ·ΕΔ

17. ChainerRL • ࠶ݱ࣮૷Ͱ͸ɺख๏ͦͷ΋ͷ͸΋ͪΖΜɺ • ؀ڥઃఆͷࡉ͔ͳࠩҟ • ϋΠύʔύϥϝʔλ • ධՁϓϩτίϧͱϝτϦοΫ
 ͳͲΛ஫ҙਂ࣮͘૷͠ɺ࠶ݱੑΛ௥ٻ

18. ࠓճ࢖༻͢ΔΞϧΰϦζϜ • DDQN (Double Deep Q Network) • Rainbow •

    PPO (Proximal Policy Optimization)

19. w 2018/11/25 ਂ૚ڧԽֶशΛνϣοτৄ͘͠
     w %FFQ
    2/FUXPSL
    %2/
    w

    ڧԽֶशͰ͸ɺঢ়ଶʹର͢Δ֤ߦಈͷධՁ஋Λ2஋ͱݺ ͼɺͦΕΛ༻͍Δख๏Λ2ֶशͱݺͿ
    w 2஋Λදݱ͢Δؔ਺Λ2ؔ਺ͱݺͼɺͦΕΛ%FFQ
    /FVSBM
    /FUXPSLͰදݱ͢ΔͷͰ%FFQ
    2/FUXPSLͱ͍͏ɻ ঢ়ଶ Convolutional 
 Neural Network ߦಈධՁ લਐ ޙୀ ӈճ ࠨճ

20. w 2018/11/25 %2/ͷ޻෉
     w &YQFSJFODF
    3FQMBZ
    w FYQFSJFODFʹ஝ੵͨ݁͠ՌΛϥϯμϜͰֶशσʔλʹ͔ͭ͏ Experience


    (ঢ়ଶ , ߦಈ , ใु) , ޙୀ , -0.1 , ӈճ , -0.1 , લਐ , 1.0 , લਐ , -0.1 Replay

21. w 2018/11/25 ڧԽֶशͷશମͷྲྀΕ
     Experience
 , ޙୀ , -0.1

    , ӈճ , -0.1 , લਐ , 1.0 , લਐ , -0.1 Replay Qؔ਺ ؀ڥ ঢ়ଶ ߦಈ ใु

22. w 2018/11/25 %PVCMF
    %2/
     • Qؔ਺Λߦಈબ୒ͱঢ়ଶධՁʹ࢖༻͢Δ͕ɺͦΕΛผʑͷωοτ ϫʔΫͰֶशͤͯ͞DQN • ৄ͍͠εϥΠυ

    • https://www.slideshare.net/juneokumura/dqnrainbow

23. w 2018/11/25 3BJOCPX
     • ਂ૚ڧԽֶशͷ༷ʑͳख๏͕ఏҊ͞Ε͖͕ͯͨɺͦΕΛ૊Έ߹ ΘͤͨΒͲ͏ͳΔ͔ͱ͍͏࿦จʮRainbow: Combining Improvements

    in Deep Reinforcement Learningʯ͕ݩ • ৄ͍͠εϥΠυ • https://www.slideshare.net/juneokumura/dqnrainbow

24. w 2018/11/25 110
     • ঢ়ଶ͔Βߦಈ֬཰Λग़ྗ͢Δؔ਺ΛωοτϫʔΫͰදݱͨ͠ϙ ϦγʔϕʔεͷϞσϧͷ̍ͭ • ߦಈ֬཰ͷ෼෍ֶ͕श࣌ʹେ͖ͳมԽ͕ى͖ͳ͍Α͏ʹɺ໨త

    ؔ਺ʹਖ਼ଇԽ߲ΛೖΕͨϞσϧ

25. Agenda 1. Chainer/CuPyͷ঺հ 2. ຊ೔ͷϋϯζΦϯͷ಺༰ 1. ChainerRLʹ͍ͭͯ 2. MineRLʹ͍ͭͯ 3.

    ChainerRL+MineRLͷ؀ڥઃఆ 4. MineRLʹ͍ͭͯ΋ͬͱ !25

26. MineRL ʹ͍ͭͯ • Microsoft ։ൃͷ Open AI gym ޓ׵ Minecraft

    ؀ڥ • “mission ϑΝΠϧ” ʹΑΔλεΫఆٛ • multi-agent ϓϩτίϧ΋αϙʔτ • MalmÖ Λར༻͠ɺಉظɾ҆ఆɾߴ଎Խ • ಛʹಉظ͸ॏཁ -- 1step ͣͭ action ͍ͨ͠ʂ • ֤λεΫʹରԠ͢Δσʔληοτ΋ఏڙ MalmÖ MineRL

27. MineRL ʹ͍ͭͯ • μΠϠϞϯυͷೖख͸͔ͳΓ೉қ౓ͷߴ͍λεΫ • ͍͔ͭ͘ͷதؒλεΫ͕ఏڙ͞ΕΔɿ • MineRLTreechop • MineRLNavigate

    • MineRLObtainIronPickaxe • MineRLObtainDiamond -- ͜Ε͕࠷ऴλεΫ 
 Navigate ʹ͸ Extreme ൛†ɺNavigate/Obtain* ʹ͸ Dense ൛‡΋ఏڙ͞ΕΔ †: “extreme” ͳ஍ܗͰ։࢝ ‡: ใु͕ΑΓີʹ༩͑ΒΕΔ “ิॿ” λεΫͨͪ

28. MineRL ʹ͍ͭͯ • 64ݸͷʮݪ໦ (log)ʯΛूΊΔ • ݪ໦͸ Minecraft ͷΩʔϦιʔε •

    ৿ (=पΓʹ໦͕ଟ͍) Ͱɺ
 మͷ佁Λ࣋ͬͨঢ়ଶ͔Βελʔτ • log Λೖख͢Δ͝ͱʹ +1 reward MineRLTreechop

29. MineRL ʹ͍ͭͯ • ࢦఆ͞Εͨΰʔϧ஍఺΁޲͔͏ • Minecraft Ͱ࠷΋جຊͱͳΔεΩϧ • “compass” Λ؍ଌՄೳʀΰʔϧํ޲Λࢦࣔͯ͘͠ΕΔ

    • ΰʔϧ౸ୡͰ +100 reward • “Dense” ൛Ͱ͸ߋʹΰʔϧʹͱͷڑ཭ʹԠͯ͡ຖεςοϓใ ुΛ໯͑Δ (ԕ͔͟Δͱෛͷใु) MineRLNavigate

30. MineRL ʹ͍ͭͯ • μΠϠϞϯυΛೖख͢Δ • Minecraft Ͱ࠷΋وॏͳΞΠςϜͷͻͱͭ • ଟ͘ͷதؒΞΠςϜ͕ඞཁͱͳΔ •

    μΠϠϞϯυೖखͰ +1024 reward • தؒΞΠςϜೖखͰ΋ (খ͞Ίͷ) ใु • σϑΥϧτͰ͸ΞΠςϜͷ1ճ໨ͷೖख࣌ͷΈใु͕໯͑Δ
 “Dense” ൛Ͱ͸Կ౓Ͱ΋໯͑Δ MineObtainDiamond

31. MineRL ʹ͍ͭͯ • μΠϠϞϯυͷೖखʹ͸ଟ͘ͷதؒΞΠςϜ͕ඞཁͱͳΔͨΊɺ ͔ͳΓࠔ೉ͳλεΫ • ໦Λೖख→ؙੴΛೖख→͔·ͲΛΫϥϑτ→మ߭ੴΛೖख→మΠϯ ΰοτΛਫ਼࿉→మϐοέϧΛΫϥϑτ→μΠϠϞϯυΛೖख MineObtainDiamond

32. MineRL ʹ͍ͭͯ • ެࣜυΩϡϝϯτΛ͝ཡ͍ͩ͘͞ ͦͷଞ؀ڥͷৄࡉ

33. Agenda 1. Chainer/CuPyͷ঺հ 2. ຊ೔ͷϋϯζΦϯͷ಺༰ 1. ChainerRLʹ͍ͭͯ 2. MineRLʹ͍ͭͯ 3.

    ChainerRL+MineRLͷ؀ڥઃఆ 4. MineRLʹ͍ͭͯ΋ͬͱ !33

34. ઃఆखॱ https://bit.ly/2Vh8eG6

35. Agenda 1. Chainer/CuPyͷ঺հ 2. ຊ೔ͷϋϯζΦϯͷ಺༰ 1. ChainerRLʹ͍ͭͯ 2. MineRLʹ͍ͭͯ 3.

    ChainerRL+MineRLͷ؀ڥઃఆ 4. MineRLʹ͍ͭͯ΋ͬͱ !35

36. The MineRL Competition • ڧԽֶशͷݚڀ͸ۙ೥େ͖͘ਐา͍ͯ͠ΔҰํͰɺ
 ཁٻ͞ΕΔαϯϓϧ਺͕૿Ճ • Ұ෦ͷڊେAIاۀҎ֎ʹΑΔݚڀ͕೉͍͠ • ࣮ੈքԠ༻͕೉͍͠

    • αϯϓϧޮ཰ͷྑ͍ RL γεςϜͷ։ൃΛڝ͏ • ͨͩ͠ɺਓؒͷσϞσʔλ (σʔληοτ) ͕
 ར༻Մೳͱ͢Δ ՝୊ MineRL Competition

37. MineRL ʹ͍ͭͯ΋ͬͱ • ֤λεΫʹରԠ͢ΔσϞϯετϨʔγϣϯ (σʔληοτ) ͕ఏ ڙ͞ΕΔ • ਓؒʹΑΔϓϨΠσʔλ •

    λεΫ͝ͱʹͦΕͧΕ100~250ݸఔ౓ͣͭ༩͑ΒΕΔ • ؀ڥͱͷ interaction ճ਺͸੍ݶ͞Ε͍ͯΔ͕ɺ
 σʔληοτ͸ (࣌ؒͷڐ͢ݶΓ) Կ౓Ͱ΋࢖ͬͯྑ͍ σʔληοτ

38. MineRL ʹ͍ͭͯ΋ͬͱ • ୯ҰͷΩʔೖྗͰ͸ͳ͘ɺҙຯͷ͋Δʮߦಈʯ୯ҐͰఆٛ͞ΕΔ • attack, camera, forward, craft, place,

    … • ֤ߦಈ͸ಉ࣮࣌ߦՄೳ (gym ͷ Dict space ͱͯ͠ఆٛ) ͕ͩɺ
 ૬൓͢Δߦಈ΋͋Δ͜ͱʹ஫ҙ (forward ͱ back ͳͲ) • ิॿλεΫͰ͸͍͔ͭ͘ͷߦಈ͕ෆՄೳʹͳ͍ͬͯΔ͜ͱ͕͋Δ • ྫ͑͹ Treechop Ͱ͸ craft ΍ place ͸Ͱ͖ͳ͍ (ඞཁͳ ͍) action ʹ͍ͭͯ

39. MineRL ʹ͍ͭͯ΋ͬͱ • observation: • pov (Ұਓশࢹ఺ͷը૾) • 64x64x3 (uint8)

    ͷ numpy array • inventory (ॴ࣋ΞΠςϜ) • dirt ͱ͔ log ͱ͔ iron_ore ͱ͔ • equipped_items (૷උ͍ͯ͠Δಓ۩) • wooden_axe ͱ͔ iron_pickaxe ͱ͔ • compassAngle (ΰʔϧํ޲ͷࢦࣔ) • Navigate* ͰͷΈར༻Մೳ • action ಉ༷ɺิॿλεΫ͝ͱʹಘΒΕΔ؍ଌ͸ҟͳΔ (pov ͸ඞͣ͋Δ) observation ʹ͍ͭͯ

40. MineRL ʹ͍ͭͯ΋ͬͱ >>> import gym >>> import minerl # ͪΐͬͱ͔͔࣌ؒΔ

    (Minecraft ͷىಈ͕ޙΖͰ૸Δ) >>> env = gym.make('MineRLObtainDiamond-v0') جຊతʹ Gym API ४ڌ = ੈͷதͷ RL πʔϧ͕͍͍ͩͨಈ͘

41. MineRL ʹ͍ͭͯ΋ͬͱ >>> obs, info = env.reset() # info ΋Ұॹʹฦͬͯ͘Δɻͪΐͬͱ

    Gym API ͱҧ͏ >>> obs {'equipped_items': {'mainhand': {'damage': 0, 'maxDamage': 0, 'type': 0}}, 'inventory': {'coal': 0, 'cobblestone': 0, ... }, 'pov': array([[[ 0, 0, 0], [ 16, 32, 9], ..., [ 75, 91, 118]], dtype=uint8)} >>> info {}

42. MineRL ʹ͍ͭͯ΋ͬͱ >>> action = env.action_space.sample() # ద౰ͳ action Λऔಘ

    >>> action OrderedDict([('attack', 0), ('back', 1), ('camera', array([ 39.44639 , -77.577675], dtype=float32)), ('craft', 3), ... ('nearbySmelt', 0), ('place', 3), ('right', 1), ('sneak', 0), ('sprint', 1)])

43. MineRL ʹ͍ͭͯ΋ͬͱ >>> obs, reward, done, info = env.step(action) >>>

    obs (ུ) >>> reward 0.0 >>> done False >>> info {}

44. MineRL tutorial with ChainerRL • PFN ։ൃͷ ChainerRL Λར༻ͨ͠ϕʔεϥΠϯΛఏڙ •

    https://github.com/minerllabs/quick_start/tree/master/ chainerrl_baselines • “starter kit” ͱͯ͠ɺ • ؀ڥ (΍σʔληοτ) ΛͲ͏ಈ͔ͤ͹ྑ͍͔ • ࠷ॳʹऔΔ΂͖φΠʔϒͳΞϓϩʔν ͳͲʹ͍ͭͯͷऔֻ͔ͬΓΛఏڙ ϕʔεϥΠϯ https://github.com/minerllabs/baselines/tree/master/general/chainerrl

45. MineRL tutorial with ChainerRL • Treechop/Navigate ΛφΠʔϒʹղ͘ • σʔληοτΛແࢹ •

    action/observation space ʹڧ͍ prior ΛԾఆ φΠʔϒͳΞϓϩʔν Treechop Navigate NavigateDense training reward training episode https://github.com/minerllabs/baselines/tree/master/general/chainerrl

46. MineRL tutorial with ChainerRL • Treechop ͱ NavigateDense ͸ΫϦΞͰ͖Δ •

    Navigate ͸Ͱ͖͍ͯͳ͍
 → Obtain* λεΫ͸͜ΕΑΓ΋೉͍͠ʂ φΠʔϒͳΞϓϩʔν Treechop Navigate NavigateDense https://github.com/minerllabs/baselines/tree/master/general/chainerrl

47. MineRL tutorial with ChainerRL • action space ʹڧ͍ prior ΛԾఆ͍ͯ͠Δ


    → ͜ͷ··Ͱ͸ଞͷλεΫʹ͸స༻Ͱ͖ͳ͍͜ͱʹ஫ҙ • Treechop • 5࣍ݩͷ཭ࢄߦಈʹม׵ i. {'forward': 1, 'jump': 0, 'camera': [0, 0]} ii.{'forward': 0, 'jump': 0, 'camera': [0, 0]} iii.{'forward': 1, 'jump': 1, 'camera': [0, 0]} iv.{'forward': 1, 'jump': 0, 'camera': [0, -10]} v. {'forward': 1, 'jump': 0, 'camera': [0, 10]}
 attack ͸ৗʹ onɺͦΕҎ֎͸ৗʹ off action space ʹ͍ͭͯ https://github.com/minerllabs/baselines/tree/master/general/chainerrl

48. MineRL tutorial with ChainerRL • Navigate/NavigateDense • 4࣍ݩͷ཭ࢄߦಈʹม׵ i. {'jump':

    0, 'camera': [0, 0]} ii.{'jump': 1, 'camera': [0, 0]} iii.{'jump': 0, 'camera': [0, -10]} iv.{'jump': 0, 'camera': [0, 10]} forward, sprint, attack ͸ৗʹ onɺͦΕҎ֎͸ৗʹ off action space ʹ͍ͭͯ https://github.com/minerllabs/baselines/tree/master/general/chainerrl

49. MineRL tutorial with ChainerRL • Double Dueling DQN (উखʹུͯ͠ݺΜͰΔ͚ͩ) •

    DQN ͷ͍ΖΜͳ೿ੜΛ٧ΊࠐΜͩख๏ • Proximal Policy Optimization • ࿈ଓۭؒͰఆٛ͞ΕͨߦಈΛࣗવʹѻ͑Δ͕ɺ
 ϕʔεϥΠϯͰ͸཭ࢄԽͨ͠ߦಈΛར༻ DDDQN Rainbow PPO https://github.com/minerllabs/baselines/tree/master/general/chainerrl

50. MineRL tutorial with ChainerRL • Ҏ্ͷࣄલ஌ࣝͷ΋ͱɺϋΠύʔύϥϝʔλΛద౰ʹνϡʔχϯ άͯ͠ಘΒΕͨ݁Ռ͕Լਤ (࠶ܝ) Treechop Navigate

    NavigateDense training reward training episode https://github.com/minerllabs/baselines/tree/master/general/chainerrl

51. MineRL tutorial with ChainerRL action space ͷ੍໿ (ಛʹ camera Λ্Լʹಈ͔ͤͳ͍)

    ͷͨΊɺ
 ਓ͔ؒΒݟΔͱ࠷దͱ͸ݴ͑ͳ͍΋ͷͷɺ
 ͦͷൣғͰʮλεΫΛΫϦΞʯͰ͖͍ͯΔʁ MineRLTreechop https://github.com/minerllabs/baselines/tree/master/general/chainerrl

52. ͞Βʹֶश͢Δʹ͸ɾɾɾ • ͍ͭઌ೔ɺbaselinesʹ໛฿ֶशͷΞϧΰϦζϜ΋௥Ճ͞Εͨ • behavoral cloning • GAIL • DQfD

    • ৄ͍͠આ໌͸͜ͷεϥΠυͰ • https://www.slideshare.net/pfi/minerl-competition-tutorial-with- chainerrl-156927429 https://github.com/minerllabs/baselines/tree/master/general/chainerrl

53. chug (Chainer User Group)ͷ঺հ !53

54. Chainer User Groupͷ঺հ • Slack chainer-jp • Twitter @chug_jp •

    ׆ಈ಺༰ • MeetupɾϋϯζΦϯͷ։࠵ • Chainerʹؔ͢Δ৘ใɾυΩϡϝϯτͷ֦ॆ • Web νϡʔτϦΞϧͷ࡞੒ • Ұॹʹ׆ಈͰ͖ΔํΛืू͍ͯ͠·͢ʂ
 Slack #chug-jp-management ʹੋඇ !54

55. Ξϯέʔτ https://forms.gle/oNniPXrDtQv8BiVYA

56. None