Slide 1
Slide 1 text
ح৺ۦಈ୳ࡧ
Curiosity-driven Exploration
ւಓେֶֶ෦
ػցೳֶՊ
େాɹ߁ब
Slide 2
Slide 2 text
ڧԽֶशͷྲྀΕͱ
ڥʹରͯ͠ߦಈΛͯ͠ใुΛಘΔ
࣮ࡍʹใुΛಘΒΕΔঢ়ଶ͕كʹ
͔͠ى͜Βͳ͍έʔε͕ଟʑ͋Δ
ex. ໎࿏ͰΰʔϧΛݟͨΒใु
ൃతใुͷಋೖ
୳ࡧͷޮԽ
Slide 3
Slide 3 text
ڧԽֶशʹ͓͚Δใु
֎ൃతใु extrinsic reward
ڥࣗମʹΑͬͯఏڙ͞ΕΔ
ex. తͷ౸ୡɺࢮetc
ൃతใु intrinsic reward
ΤʔδΣϯτࣗମʹΑͬͯੜ͞ΕΔ
ex. ৽͠͞ɺح৺etc
Slide 4
Slide 4 text
ɹح৺ۦಈ୳ࡧ
2ͭͷผʑͷχϡʔϥϧωοτϫʔΫΛτϨʔχϯά
1.ٯϞσϧ inverse dynamics model
2.ॱϞσϧ forward dynamics model
Slide 5
Slide 5 text
ɹಛϕΫτϧԽํ๏ͷ࠷దԽ
ٯϞσϧ inverse dynamics model
st
, st+1Λ؍ଌޙCNNͰಛϕΫτϧԽ
ڥ͔Βঢ়ଶ
ؒͷߦಈ
st
, st+1
at Λਪଌ = ̂
at
࣮ࡍͷߦಈͱਪଌ͞ΕͨߦಈͷෆҰகΛද͢
ଛࣦؔͷ࠷খԽ
ϕ(st
), ϕ(st+1
)
̂
at
= g(st
, st+1
; θI
)
st
, st+1
→ at
ɹͷ࠷దԽ͕త
ϕ
Slide 6
Slide 6 text
ɹح৺ͷಋೖ
ॱϞσϧ forward dynamics model
st
, at ͔Β࣍ͷঢ়ଶ
ঢ়ଶͱߦಈ Λਪଌ
࣮ࡍͷঢ়ଶͱਪଌ͞Εͨঢ়ଶͷෆҰகΛද͢
ޡࠩؔΛ࠷খԽ
ϕ(st+1
)
̂
ϕ(st+1
) = f(ϕ(st
), at
; θF
)
෦ใु࣍ঢ়ଶͷ༧͕͍͠΄Ͳେ͖͍
ri
t
=
η
2
|| ̂
ϕ(st+1
) − ϕ(st+1
)||2
st
, at
→ st+1
ٯϞσϧͰಘͨ
ಛϕΫτϧ
Slide 7
Slide 7 text
Intrinsic Curiosity Module (ICM)ͷߏ
ΈࠐΈ: 4
ϑΟϧλʔ: ֤ʑ32ݸ
ΧʔωϧαΠζ: 3x3
શ݁߹:
288+1 → 256 → 288
શ݁߹:
288x2 → 256 → 4
Slide 8
Slide 8 text
PPO+CuriosityͱPPOͷൺֱʢPyramidsڥʣ
Slide 9
Slide 9 text
ࢀߟจݙ
• ݩจ
https://pathak22.github.io/noreward-rl/resources/
icml17.pdf
• Unity Blog
https://blogs.unity3d.com/jp/2018/06/26/solving-sparse-
reward-tasks-with-curiosity/
• ୈ44ճCVษڧձʮڧԽֶशจಡΈձʯൃදࢿྉ
https://www.slideshare.net/takmin/curiosity-driven-
exploration