TensorFlow & DeepMind Lab & UNREAL

by Kosuke Miyoshi

Slide 1

Slide 1 text

5FOTPS'MPX %FFQNJOE-BC OBSSBUJWFOJHIUTגࣜձࣾ ࡾ޷߁༞ 5FOTPS'MPX6TFS(SPVQ

Slide 2

Slide 2 text

%FFQ.JOE-BC

Slide 3

Slide 3 text

6/3&"- ڧԽֶशͷ"$ΞϧΰϦζϜΛϕʔεʹ&YQFSJFODF 3FQMBZΛ࢖ͬͨิॿλεΫΛ૊Έ߹Θͤͯ%໎࿏Ͱ Yഒͷֶशͷߴ଎ԽΛ࣮ݱ REINFORCEMENT LEARNING WITH UNSUPERVISED AUXILIARY TASKS   Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki et. al (DeepMind, 2016)

Slide 4

Slide 4 text

ಈ෺ͷເ w ಈ෺͸ເͷதͰܦݧͨ͠ग़དྷࣄΛ࠶ݱ ϦϓϨΠ ͠ ͳ͕Βւഅ৽ൽ࣭΁ͷهԱͷݻఆΛߦ͍ͬͯΔ w ߠఆత൱ఆతͳใुʹؔΘΔग़དྷࣄͷເΛಘʹස ൟʹݟֶͯशΛߦ͍ͬͯΔ w FYʮਫҿΈ৔ͰϥΠΦϯΛݟ͔͚ͯةݥͳ໨ʹ͋ͬ ͨʯ w 6/3&"-Ͱ͸͜ΕΛώϯτʹ͍ͯ͠Δ

Slide 5

Slide 5 text

ڧԽֶश ؀ڥ ΤʔδΣϯτ "DUJPO ⬆ ➡ ⬇ ঢ়ଶ T ใु S

Slide 6

Slide 6 text

6/3&"-΁ͷྲྀΕ %2/ "$ 6/3&"-

Slide 7

Slide 7 text

"$ "TZODISPOPVT"EWBODFE"DUPS$SJUJD w ෳ਺ͷ؀ڥΛඇಉظʹฒྻʹಈֶ͔ͯ͠शΛߴ଎Խ ҆ఆԽͤͨ͞

Slide 8

Slide 8 text

К 1PMJDZ 7 ֤"DUJPOΛऔΔ֬཰ ݱࡏͷঢ়ଶՁ஋ ⬆ ➡ ⬇ TPGUNBY MJOFBS $POW $POW '$ -45. "$ͷωοτϫʔΫߏ଄

Slide 9

Slide 9 text

֤-PDBM/FUXPSLͰ͸ɺֶश݁Ռͷޯ഑ EВ ͷΈΛٻΊɺ ΢ΣΠτʹ൓өͤͣ(MPCBMͷ΢ΣΠτ В ʹݸผʹ൓өɻ (MPCBMͷ΢ΣΠτΛ·֤ͨ-PDBMͷ΢ΣΠτʹίϐʔɻ EВ EВ EВ EВ В ʜ

Slide 10

Slide 10 text

1PMJDZ К 7ͷޯ഑ R= = = w 7͸3ʹ͚ۙͮΔ༷ʹߋ৽ w 37͕ਖ਼ͳΒɺऔͬͨBDUJPO͕ग़Δ֬཰Λ૿΍༷͢ʹߋ৽  37͕ෛͳΒɺऔͬͨBDUJPO͕ग़Δ֬཰ΛݮΒ༷͢ʹߋ৽   V network: Policy network: ˞্هͷදهͰ7͸(SBEJFOU%FTDFOU 1PMJDZ͸(SBEJFOU"TDFOUθv = θv - α * dθv, θ = θ + α * dθ 1PMJDZ 7

Slide 11

Slide 11 text

6/3&"- w "$ʹɺ&YQFSJFODF3FQMBZΛޮՌతʹ࢖ͬͨิ ॿλεΫΛಋೖ͠ɺ͞ΒʹֶशΛߴ଎Խͤ͞Δ w 1JYFM$POUSPM w 3FXBSE1SFEJDUJPO w 7BMVF'VODUJPO3FQMBZ 6/TVQFSWJTFE3&JOGPSDFNFOU"VYJMJBSZ-FBSOJOH

Slide 12

Slide 12 text

&YQFSJFODF3FQMBZ w <ঢ়ଶ "DUJPO ใु ࣍ঢ়ଶ>ͷϖΞΛେྔʹอଘ͠ ͯɺ͔ͦ͜ΒαϯϓϦϯάͯ͠ωοτϫʔΫΛֶश w %2/͸ɺ͜Ε͕ͳ͍ͱֶश͕҆ఆ͠ͳ͔ͬͨ w "$Ͱ͸࢖͍ͬͯͳ͍

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

1JYFM$POUSPM w ը໘ͷϐΫηϧ஋ͷมԽྔΛΑΓେ͖͘͢Δ༷ʹ͞ ͍ͤͨ w ը໘ͷϐΫηϧ஋ͷมԽΛٖࣅใुͱ͢Δิॿλε Ϋ

Slide 15

Slide 15 text

1JYFM$POUSPM w ը໘ΛYͷϐΫηϧάϦουʹ෼͚ɺάϦουຖʹ2ֶशΛߦ͏ w %VFMJOH/FUXPSLΛ࢖ͬͨ2ֶश ˞1JYFM$POUSPMͰಘΒΕͨ2஋͕BDUJPOͷબ୒ʹ࢖ΘΕΔ༁Ͱ͸ͳ͍ YͷάϦου BDUJPO਺෼ ֤άϦουͷϐΫηϧมԽྔฏۉΛใुͱͨ࣌͠ͷׂҾՃࢉใु߹ܭ2

Slide 16

Slide 16 text

3FXBSE1SFEJDUJPO w &YQFSJFODF3FQMBZ͔Β࿈ଓͨ͠ϑϨʔϜऔΓग़ ͠ɺϑϨʔϜ໨ͷใु͕ɺਖ਼͔ෛ͔θϩ͔Λ༧ଌ ͢ΔิॿλεΫ w ༧ଌ͢Δใु͸ɺ  ʴ ʔPSͷൺ཰͕ʹͳΔ༷ʹαϯϓϦϯά  ༗ӹͳใुΠϕϯτ͸ϨΞͰ͋ͬͯ΋ɺසൟʹαϯϓϦϯά͞ΕΔ

Slide 17

Slide 17 text

3FXBSE1SFEJDUJPO ࣍ͷใु͕PSPSΛ༧ଌ

Slide 18

Slide 18 text

7BMVF'VODUJPO3FQMBZ w "$Ͱ΍͍ͬͯΔɺঢ়ଶՁ஋ 7 ͷਪఆ "DUPS$SJUJDͷ$SJUJDଆ Λɺ&YQFSJFODF3FQMBZ͔ΒαϯϓϦϯάͨ͠ϑϨʔϜͰ࠶౓ ߦ͏ w 3FXBSE1SFEJDUJPOͱҧͬͯɺαϯϓϦϯά͸ಛʹภΒͤͳ͍

Slide 19

Slide 19 text

ิॿλεΫ͸ɺ௚઀"DUJPOબ୒ʹӨڹ͸༩͑ͳ͍͕ɺϕʔ εͷ"$ͱ$POW૚ɺ-45.૚ͷ8FJHIUΛڞ༗͍ͯ͠Δͷ ͰɺิॿλεΫΛೖΕΔ͜ͱʹΑΓɺͦΕΛղ͘ޮՌతͳ ಛ௃දݱ͕ಘΒΕΔ͜ͱʹΑΓɺؒ઀తʹ"DUJPOબ୒ʹӨ ڹΛ༩͑Δ

Slide 20

Slide 20 text

ଛࣦؔ਺ #BTF"$ 7BMVF'VODUJPO 3FQMBZ 1JYFM$POUSPM YάϦου෼ 3FXBSE 1SFEJDUJPO

Slide 21

Slide 21 text