TensorFlow & DeepMind Lab & UNREAL

TensorFlow & DeepMind Lab & UNREAL

TensorFlowで実装したUNREALアルゴリズムでDeepMind Labの3D迷路を解く

Bf8061e5e8837f89d56092789c03e19e?s=128

Kosuke Miyoshi

April 20, 2017
Tweet

Transcript

  1. 5FOTPS'MPX %FFQNJOE-BC OBSSBUJWFOJHIUTגࣜձࣾ ࡾ޷߁༞ 5FOTPS'MPX6TFS(SPVQ

  2. %FFQ.JOE-BC

  3. 6/3&"- ڧԽֶशͷ"$ΞϧΰϦζϜΛϕʔεʹ&YQFSJFODF 3FQMBZΛ࢖ͬͨิॿλεΫΛ૊Έ߹Θͤͯ%໎࿏Ͱ Yഒͷֶशͷߴ଎ԽΛ࣮ݱ REINFORCEMENT LEARNING WITH UNSUPERVISED AUXILIARY TASKS

    
 Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki et. al (DeepMind, 2016)
  4. ಈ෺ͷເ w ಈ෺͸ເͷதͰܦݧͨ͠ग़དྷࣄΛ࠶ݱ ϦϓϨΠ ͠ ͳ͕Βւഅ৽ൽ࣭΁ͷهԱͷݻఆΛߦ͍ͬͯΔ w ߠఆత൱ఆతͳใुʹؔΘΔग़དྷࣄͷເΛಘʹස ൟʹݟֶͯशΛߦ͍ͬͯΔ w

    FYʮਫҿΈ৔ͰϥΠΦϯΛݟ͔͚ͯةݥͳ໨ʹ͋ͬ ͨʯ w 6/3&"-Ͱ͸͜ΕΛώϯτʹ͍ͯ͠Δ
  5. ڧԽֶश ؀ڥ ΤʔδΣϯτ "DUJPO ⬆ ➡ ⬇ ঢ়ଶ T ใु

    S
  6. 6/3&"-΁ͷྲྀΕ %2/ "$ 6/3&"-   

  7. "$ "TZODISPOPVT"EWBODFE"DUPS$SJUJD w ෳ਺ͷ؀ڥΛඇಉظʹฒྻʹಈֶ͔ͯ͠शΛߴ଎Խ ҆ఆԽͤͨ͞

  8. К 1PMJDZ 7 ֤"DUJPOΛऔΔ֬཰ ݱࡏͷঢ়ଶՁ஋ ⬆ ➡ ⬇ TPGUNBY MJOFBS

    $POW $POW '$ -45. "$ͷωοτϫʔΫߏ଄
  9. ֤-PDBM/FUXPSLͰ͸ɺֶश݁Ռͷޯ഑ EВ ͷΈΛٻΊɺ ΢ΣΠτʹ൓өͤͣ(MPCBMͷ΢ΣΠτ В ʹݸผʹ൓өɻ (MPCBMͷ΢ΣΠτΛ·֤ͨ-PDBMͷ΢ΣΠτʹίϐʔɻ EВ EВ EВ

    EВ В ʜ
  10. 1PMJDZ К 7ͷޯ഑ R= = = w 7͸3ʹ͚ۙͮΔ༷ʹߋ৽ w 37͕ਖ਼ͳΒɺऔͬͨBDUJPO͕ग़Δ֬཰Λ૿΍༷͢ʹߋ৽


    37͕ෛͳΒɺऔͬͨBDUJPO͕ग़Δ֬཰ΛݮΒ༷͢ʹߋ৽ 
 V network: Policy network: ˞্هͷදهͰ7͸(SBEJFOU%FTDFOU 1PMJDZ͸(SBEJFOU"TDFOUθv = θv - α * dθv, θ = θ + α * dθ 1PMJDZ 7
  11. 6/3&"- w "$ʹɺ&YQFSJFODF3FQMBZΛޮՌతʹ࢖ͬͨิ ॿλεΫΛಋೖ͠ɺ͞ΒʹֶशΛߴ଎Խͤ͞Δ w 1JYFM$POUSPM w 3FXBSE1SFEJDUJPO w 7BMVF'VODUJPO3FQMBZ

    6/TVQFSWJTFE3&JOGPSDFNFOU"VYJMJBSZ-FBSOJOH
  12. &YQFSJFODF3FQMBZ w <ঢ়ଶ "DUJPO ใु ࣍ঢ়ଶ>ͷϖΞΛେྔʹอଘ͠ ͯɺ͔ͦ͜ΒαϯϓϦϯάͯ͠ωοτϫʔΫΛֶश w %2/͸ɺ͜Ε͕ͳ͍ͱֶश͕҆ఆ͠ͳ͔ͬͨ w

    "$Ͱ͸࢖͍ͬͯͳ͍
  13. None
  14. 1JYFM$POUSPM w ը໘ͷϐΫηϧ஋ͷมԽྔΛΑΓେ͖͘͢Δ༷ʹ͞ ͍ͤͨ w ը໘ͷϐΫηϧ஋ͷมԽΛٖࣅใुͱ͢Δิॿλε Ϋ

  15. 1JYFM$POUSPM w ը໘ΛYͷϐΫηϧάϦουʹ෼͚ɺάϦουຖʹ2ֶशΛߦ͏ w %VFMJOH/FUXPSLΛ࢖ͬͨ2ֶश ˞1JYFM$POUSPMͰಘΒΕͨ2஋͕BDUJPOͷબ୒ʹ࢖ΘΕΔ༁Ͱ͸ͳ͍ YͷάϦου BDUJPO਺෼ ֤άϦουͷϐΫηϧมԽྔฏۉΛใुͱͨ࣌͠ͷׂҾՃࢉใु߹ܭ2

  16. 3FXBSE1SFEJDUJPO w &YQFSJFODF3FQMBZ͔Β࿈ଓͨ͠ϑϨʔϜऔΓग़ ͠ɺϑϨʔϜ໨ͷใु͕ɺਖ਼͔ෛ͔θϩ͔Λ༧ଌ ͢ΔิॿλεΫ w ༧ଌ͢Δใु͸ɺ
 ʴ ʔPSͷൺ཰͕ʹͳΔ༷ʹαϯϓϦϯά
 ༗ӹͳใुΠϕϯτ͸ϨΞͰ͋ͬͯ΋ɺසൟʹαϯϓϦϯά͞ΕΔ

  17. 3FXBSE1SFEJDUJPO ࣍ͷใु͕ PSPSΛ༧ଌ

  18. 7BMVF'VODUJPO3FQMBZ w "$Ͱ΍͍ͬͯΔɺঢ়ଶՁ஋ 7 ͷਪఆ "DUPS$SJUJDͷ$SJUJDଆ Λɺ&YQFSJFODF3FQMBZ͔ΒαϯϓϦϯάͨ͠ϑϨʔϜͰ࠶౓ ߦ͏ w 3FXBSE1SFEJDUJPOͱҧͬͯɺαϯϓϦϯά͸ಛʹภΒͤͳ͍

  19. ิॿλεΫ͸ɺ௚઀"DUJPOબ୒ʹӨڹ͸༩͑ͳ͍͕ɺϕʔ εͷ"$ͱ$POW૚ɺ-45.૚ͷ8FJHIUΛڞ༗͍ͯ͠Δͷ ͰɺิॿλεΫΛೖΕΔ͜ͱʹΑΓɺͦΕΛղ͘ޮՌతͳ ಛ௃දݱ͕ಘΒΕΔ͜ͱʹΑΓɺؒ઀తʹ"DUJPOબ୒ʹӨ ڹΛ༩͑Δ

  20. ଛࣦؔ਺ #BTF"$ 7BMVF'VODUJPO 3FQMBZ 1JYFM$POUSPM YάϦου෼ 3FXBSE 1SFEJDUJPO

  21. None
  22. "$ͱͷൺֱ %FFQ.JOE-BC؀ڥʹͯฏۉͰYഒͷߴ଎Խ

  23. ΓΜ͝ΛऔΔͱ ఺ ϫʔϓ஍఺ʹ౸ୡ͢Δͱ ఺ΛಘͯϥϯμϜͳ ৔ॴʹϫʔϓ

  24. ࠶ݱݕূಈը IUUQTZPVUVCFY),R#F)* ˞4QFBLFS%FDL಺Ͱද͍ࣔͯ͠Δ৔߹͸ɺ63-ϦϯΫ͕ΫϦοΫͰ͖ͳ͍ͷͰɺQEGΛμ΢ϯϩʔυͯ͠ΫϦοΫ͍ͯͩ͘͠͞

  25. 1JYFM$POUSPM ֤άϦουͷલϑϨʔϜͱͷ
 ϐΫηϧมԽྔ ֤άϦουͷ2஋
 औͬͨ"DUJPOʹର͢Δ2஋

  26. 1PMJDZ К ֤ΞΫγϣϯΛऔΔ֬཰
 લਐ ޙୀ ࠨӈճస ࠨӈεϥΠυ ֶश͕ਐΉͱ΄΅ͷ֬཰Ͱ֤"DUJPOΛબͿΑ͏ʹͳͬͯ͘Δ

  27. 7BMVF'VODUJPO ݱࡏͷঢ়ଶՁ஋
 ϫʔϓ஍఺ ఺ ʹۙͮ͘ʹͭΕ্͕͍ͯͬͯ͘

  28. 3FXBSE1SFEJDUJPO ϓϥεใु͕དྷΔͱ༧ଌ͍ͯ͠Δ

  29. 4PVSDF w IUUQTHJUIVCDPNNJZPTVEBVOSFBM