[ICML2021 論文読み会]Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research

Transcript

1. ICML2021 論文読み会 Revisiting Rainbow: Promoting more Insightful and Inclusive Deep

   Reinforcement Learning Research Masato Ninohira, LINE Corporation 2021.08

2. 
   ࿦จબఆͷϓϩηε 
   ࿦จ಺༰ 
   ॴײ Contents

3. 
   ࿦จબఆͷϓϩηε 
   ࿦จ಺༰ 
   ॴײ Contents

4. ࣗ਎ॳͷ࿦จಡΈձ ࿦จબఆϓϩηε • ੎͍ͰࢀՃ • ୲౰࿦จͲ͏͠Αʁ ͱΓ͋͑ͣ(PPHMF͕ग़͍ͯ͠Δ࿦จ୳ͦ͏ • (PPHMFͱ͍͏ڊਓͷݞʹ৐͍ͬͯ͘ελΠϧ •

   (PPHMF
   BU
   *$.-
   Λݟ͚ͭΔ ڧԽֶश
   ϒϩάهࣄͷ͋Δຊ࿦จΛબఆ • ڧԽֶशͳΒଟগಡΊΔ • (PPHMF
   "*
   #MPHʹղઆϒϩάهࣄͷ͋Δຊ࿦จʹબఆ

5. 
   ࿦จબఆͷϓϩηε 
   ࿦จ಺༰ 
   ॴײ Contents

6. Πϯτϩɾഎܠ ڧԽֶशͷख๏ͱͯ͠3BJOCPX͕஌ΒΕ͍ͯΔ • %2/ ʹ౰࣌࿩୊Ͱ͋ͬͨԼهΞϧΰϦζϜΛ௥Ճ •  %PVCMF
   2MFBSOJOH
    1SJPSJUJ[FE
   FYQFSJFODF
   SFQMBZ
   

    %VFMJOH
   OFUXPSL
    .VMUJTUFQ
   MFBSOJOH
    %JTUSJCVUJPOBM
   3-
    /PJTZ
   OFUT • ஌Γ߹͍͕ϩοΫϚϯͷதϘεϥογϡͬͯᄻ͑ͯ·ͨ͠ খن໛ͳݚڀάϧʔϓ͕࠶ݱ PS
   ཱͪ޲͔͏ͷ͸ࠔ೉ • "SDBEF
   -FBSOJOH
   &OWJSPONFOU
   "-&
   ήʔϜΛࢼߦ͢Δͷʹ 1Ͱ (16
   EBZ খن໛؀ڥͰ΋Ձ஋ͷ͋ΔΞϧΰͰ͋Δ͔Λূ໌Ͱ͖Δ͜ͱ͸ࠓޙͷݚڀऀʹͱͬͯॏཁ • *O
   MJHIU
   PG
   UIJT
   XF
   XJTI
   UP
   JOWFTUJHBUF
   UISFF
   RVFTUJPOT • 8PVME
   )FTTFM
   FU
   BM
   
   IBWF
   BSSJWFE
   BU
   UIF
   TBNF
   RVBMJUBUJWF
   DPODMVTJPOT
   IBE
   UIFZ
   SVO
   UIFJS
   FYQFSJNFOUT
   PO
   B
   TFU
   PG
   TNBMMFSTDBMF
   FYQFSJNFOUT
   খ͍͞؀ڥͰ΋ಉ݁͡Ռͩͬͨʁ • %P
   UIF
   SFTVMUT
   PG
   )FTTFM
   FU
   BM
   
   HFOFSBMJ[F
   XFMM
   UP
   OPO"-&
   FOWJSPONFOUT
   PS
   BSF
   UIFJS
   SFTVMUT
   PWFSMZTQFDJGJD UP
   UIF
   DIPTFO
   CFODINBSL
   "SDBEF
   -FBSOJOH
   &OWJSPONFOU
   Ҏ֎Ͱ΋௨༻͢Δʁ • *T
   UIFSF
   TDJFOUJGJD
   WBMVF
   JO
   DPOEVDUJOH
   FNQJSJDBM
   SFTFBSDI
   JO
   SFJOGPSDFNFOU
   MFBSOJOH
   XIFO
   SFTUSJDUJOH
   POFTFMG
   UP TNBMM UP
   NJETDBMF
   FOWJSPONFOUT ڧԽֶशͰ͸খɾதن໛ʹݶఆ࣮ͨ͠ূݚڀ͸ՊֶతʹՁ஋͕͋Δ͔ʁ

7. 3FWJTJUJOH
   3BJOCPX ͭͷΞϧΰΛશ෦࢖Θͳ͍ ؀ڥ • ݹయతͳTNBMMͳ؀ڥ $BSU1PMF
   "DSPCPU
   -VOBS-BOEFS
   BOE
   .PVOUBJO$BS •

   $16Ͱ࣌ؒະຬ • ৮ͬͨ͜ͱ͕͋Δਓʹ͸෼͔Δͱࢥ͏͕ɺຊ౰ʹTNBMM • "-&
   .JO"UBS "TUFSJYɺ#SFBLPVUɺ'SFFXBZɺ4FBRVFTUɺ4QBDF*OWBEFST • ؆қ൛"-& • 1Ͱd࣌ؒ ͜ΕͰ΋ݩͷ"UBSJͷHBNFʹൺ΂ͨΒ୹͍ ݁Ռ • جຊతʹ͸ɺ%2/୯඼ΑΓ͸ɺ͍͔ͭ͘ͷཁૉΛ଍ͨ͠ํ͕ྑ͍ • %JTUSJCVUJPOBM
   3-͸୯඼ͰೖΕͯ΋ྑ͘ͳΒͳ͍έʔε΋͋Δ

8. #FZPOE
   UIF
   3BJOCPX  
   &YBNJOJOH
   OFUXPSL
   BSDIJUFDUVSFT
   BOE
   CBUDI
   TJ[FT • ݕূ಺༰
   ϋΠύϥ ϨΠϠʔ਺ 
   தؒ૚ͷVOJU਺ 
   όοναΠζ • ݁Ռ
   ϨΠϠʔ਺͸d

   ͩͱਫ਼౓Լ͕Δ
   
   VOJU਺͸Ҏ্ 
   όοναΠζ Ҏ্ 
   &YBNJOJOH
   EJTUSJCVUJPO
   QBSBNFUFSJ[BUJPOT • ݕূ಺༰
   %JTUSJCVUJPOBM
   3-ͷؔ࿈ख๏ͱͯ͠ɺҎԼͭͷख๏͕஌ΒΕ͍ͯΔ • %JTUSJCVUJPOBM
   3FJOGPSDFNFOU
   -FBSOJOH
   XJUI
   2VBOUJMF
   3FHSFTTJPO 23%2/ • *NQMJDJU
   RVBOUJMF
   OFUXPSLT
   GPS
   EJTUSJCVUJPOBM
   SFJOGPSDFNFOU
   MFBSOJOH *2/ • ݁Ռ • 23%2/ • ݹయతͳ؀ڥ
   3BJOCPX
   NFUIPE͸جຊతʹྑ͘ͳΔҰํ3BJOCPXʹ23Λ଍͢ͱѱԽ • .JOBUBS
   3BJOCPX
   
   23BJOCPX 
   ը૾௚ೖΕέʔεͰྑ͍ 
   DPOW૚͕͋Δ//ʹ23͕༗ޮ͔΋ • *2/ • ݹయతͳ؀ڥ
   ؀ڥʹΑΓ3BJOCPX
   NFUIPEͷޮՌ͸༷ʑ • .JOBUBS 
   3BJOCPX
   
   *3BJOCPX 23%2/ͱಉ༷ͷ݁Ռ

9. #FZPOE
   UIF
   3BJOCPX  
   .VODIBVTFO
   3FJOGPSDFNFOU
   -FBSOJOH .%2/ • ͭͷಛ௃ 
   UIF
   VTF
   PG
   TUPDIBTUJD
   QPMJDJFT
   
   BVHNFOUJOH
   UIF
   SFXBSE
   XJUI
   UIF
   TDBMFE
   MPH
   r QPMJDZ • ݁Ռ

   • %2/ • ݹయతͳ؀ڥ
   .VODIBVTFO͸ͳ͍ํ͕ྑ͍ • .JOBUBS
   %2/
   
   .%2/͕ͩɺ*2/
   
   .*2/ • 3BJOCPX • ݹయతͳ؀ڥ
   .3BJOCPX
   
   3BJOCPX
   
   .*3BJOCPX • .JOBUBS
   ͳΜͱ΋͍͑ͳ͍͕ɺ "TUFSJY
   #SFBLPVU
   BOE
   4QBDF*OWBEFST Ͱ͸.VODIBVTFO͕ޮ͍ͯͦ͏ 
   3FFWBMVBUJOH
   UIF
   )VCFS
   MPTT • ݕূ಺༰
   ݩ࿦จ
   -PTT
   )VCFSMPTT
   PQUJNJ[FS
   3.41SPQ͕ͩɺ.4&
   BEBNʹͨ͠ΒͲ͏ͳΔ͔ • ݁Ռ • )VCFS
   -PTT
   
   3.41SPQ 
   .4&
   
   "EBN • 3.41SPQݻఆͩͱɺ.4&
   
   )VCFS
   MPTT

10. 
    1VUUJOH
    JU
    BMM
    UPHFUIFS 
    3BJOCPX
    GMBWPST • %2/
    
    3BJOCPX
    
    #FZPOE
    .FUIPE 23BJOCPX
    *3BJOCPX • 3BJOCPX
    
    #FZPOE
    .FUIPEͷதͰͲΕ͕Ұ൪ྑ͍͔͸Ұ֓ʹݴ͑ͳ͍

    
    &OWJSPONFOU
    QSPQFSUJFT
    ֤छ؀ڥʹ͍ͭͯͷߟ࡯ɻ෇࿥ʹৄࡉ͕هࡌ • ݹయతͳ؀ڥ • $BSU1PMF୯७ͳλεΫ͕ނʹɺֶश཰ʹහײͰPQUJNJ[FSͷ҆ఆੑͷݕূʹ޲͍͍ͯΔ OPJTZ
    OFUͱ.4&͕༗ޮ • -VOBS-BOEFS
    %JTUSJCVUJPOBM
    3-ܥͷݕূʹ޲͍͍ͯΔ • "DSPCPU
    .PVOUBJO$BS
    ใु͕গͳ͘ɺ୳ࡧઓུͷݕূʹ޲͍͍ͯΔ • .JO"UBS
    ੜ"UBSJΛ࢖͏ΑΓ΋ɺܰྔతʹ$//
    
    3-ͷݕূΛͰ͖Δ • ݩ"UBSJͩͱɺҰൠతͳը૾ೝࣝతͳ$//෦෼ͷܭࢉίετ͕େ͖͗͢Δ • 4FBRVFTU
    'SFFXBZ
    ෦෼తͳ؍ଌՄೳੑͱใुͷεύʔεੑ͔Βɺ୳ࡧख๏ͷݕূʹ޲͍͍ͯΔ

11. 
    $PODMVTJPO ݶΒΕͨܭࢉ؀ڥͰ༷ʑͳݕূΛߦͬͨ • Կ͔Λൃݟ͢ΔΑΓ΋ݟ௚͢ํ͕͸Δ͔ʹ؆୯͕ͩ͜ͷݚڀͷҙਤ͸தখن໛ͷ؀ڥʹ͓͚Δ࣮ূݚڀͷଥ౰ੑͱҙٛͷשى • ܭࢉྔͷগͳ͍؀ڥͰͷݕূ΋ݚڀͷՁ஋ʹͳΔ େن໛ͳϕϯνϚʔΫΛ൷൑͍ͯ͠ΔΘ͚Ͱͳ͘ɺখن໛ͳϕϯνϚʔΫʹ΋Ձ஋͕͋Δ͜ͱΛओு • େن໛ͳϕϯνϚʔΫΛॏࢹ͠ͳ͍Α͏ٻΊ͍ͯΔͷͰ͸ͳ͍ •

    ݚڀऀʹ͸খن໛ͳ؀ڥΛௐࠪͷوॏͳπʔϧͱͯ͠ߟྀ͢ΔΑ͏ɺࠪಡऀʹ͸খن໛ͳ໰୊ʹয఺Λ౰࣮ͯͨূతͳݚڀΛ ൱ఆ͠ͳ͍Α͏ٻΊ͍ͯΔ͚ͩ • ݚڀͷঢ়گΛΑΓ໌֬ʹ೺Ѳ͢Δ͜ͱ͕Ͱ͖ɺ·ͨɺଟ༷Ͱܙ·Εͳ͍ίϛϡχςΟ͔Βͷ৽نࢀೖऀͷোนΛݮΒ͢͜ͱ͕ Ͱ͖Δͱ৴͍ͯ͡Δ

12. 
    ࿦จબఆͷϓϩηε 
    ࿦จ಺༰ 
    ॴײ Contents

13. ॴײ • 3BJOCPXपΓΛ͋Δఔ౓஌͍ͬͯͨͷͰεϥεϥಡΊͨ • ܭࢉࢿݯ͕ͳ͍ͱݚڀͷελʔτϥΠϯʹཱͯͳ͍͜ͱʹҰੴΛ౤͡Δ͜ͱ͸ڵຯਂ͍