Curiosity-driven Exploration: 好奇心駆動探索

Explanation of one of the reinforcement learning, Curiosity-driven Exploration.
強化学習の一つである好奇心駆動探索の解説。

Yasunari Ota, Hokkaido Univ., Faculty of Engineering, Mechanical Engineering Department
北海道大学工学部機械知能工学科3年 大田 康就

Transcript

1. ޷ح৺ۦಈ୳ࡧ Curiosity-driven Exploration ๺ւಓେֶ޻ֶ෦
   ػց஌ೳ޻ֶՊ೥
   େాɹ߁ब
2. ڧԽֶशͷྲྀΕͱ໰୊఺ ؀ڥʹରͯ͠ߦಈΛͯ͠ใुΛಘΔ ࣮ࡍʹใुΛಘΒΕΔঢ়ଶ͕كʹ ͔͠ى͜Βͳ͍έʔε͕ଟʑ͋Δ ex. ໎࿏಺ͰΰʔϧΛݟͨΒใु ಺ൃతใुͷಋೖ ୳ࡧͷޮ཰Խ
3. ڧԽֶशʹ͓͚Δใु ֎ൃతใु extrinsic reward ؀ڥࣗମʹΑͬͯఏڙ͞ΕΔ ex. ໨త஍఺΁ͷ౸ୡɺࢮ๢etc ಺ൃతใु intrinsic reward ΤʔδΣϯτࣗମʹΑͬͯੜ੒͞ΕΔ ex. ໨৽͠͞ɺ޷ح৺etc
4. ɹ޷ح৺ۦಈ୳ࡧ 2ͭͷผʑͷχϡʔϥϧωοτϫʔΫΛτϨʔχϯά 1.ٯϞσϧ inverse dynamics model 2.ॱϞσϧ forward dynamics model
5. ɹಛ௃ϕΫτϧԽํ๏ͷ࠷దԽ ٯϞσϧ inverse dynamics model st , st+1Λ؍ଌޙCNN౳Ͱಛ௃ϕΫτϧԽ ؀ڥ͔Βঢ়ଶ ؒͷߦಈ st , st+1 at Λਪଌ = ̂ at ࣮ࡍͷߦಈͱਪଌ͞ΕͨߦಈͷෆҰக౓Λද͢ ଛࣦؔ਺ͷ࠷খԽ ϕ(st ), ϕ(st+1 ) ̂ at = g(st , st+1 ; θI ) st , st+1 → at ɹͷ࠷దԽ͕໨త ϕ
6. ɹ޷ح৺ͷಋೖ ॱϞσϧ forward dynamics model st , at ͔Β࣍ͷঢ়ଶ ঢ়ଶͱߦಈ Λਪଌ ࣮ࡍͷঢ়ଶͱਪଌ͞Εͨঢ়ଶͷෆҰக౓Λද͢ ޡࠩؔ਺Λ࠷খԽ ϕ(st+1 ) ̂ ϕ(st+1 ) = f(ϕ(st ), at ; θF ) ಺෦ใु͸࣍ঢ়ଶͷ༧૝͕೉͍͠΄Ͳେ͖͍ ri t = η 2 || ̂ ϕ(st+1 ) − ϕ(st+1 )||2 st , at → st+1 ٯϞσϧͰಘͨ ಛ௃ϕΫτϧ
7. Intrinsic Curiosity Module (ICM)ͷߏ଄ ৞ΈࠐΈ૚: 4૚ ϑΟϧλʔ: ֤ʑ32ݸ ΧʔωϧαΠζ: 3x3 શ݁߹૚: 288+1 → 256 → 288 શ݁߹૚: 288x2 → 256 → 4
8. PPO+CuriosityͱPPOͷൺֱʢPyramids؀ڥʣ
9. ࢀߟจݙ • ݩ࿦จ
 https://pathak22.github.io/noreward-rl/resources/ icml17.pdf • Unity Blog
 https://blogs.unity3d.com/jp/2018/06/26/solving-sparse- reward-tasks-with-curiosity/ • ୈ44ճCVษڧձʮڧԽֶश࿦จಡΈձʯൃදࢿྉ
 https://www.slideshare.net/takmin/curiosity-driven- exploration