ICML2021 論文読み会 Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research Masato Ninohira, LINE Corporation 2021.08

࿦จબఆͷϓϩηε ࿦จ಺༰ ॴײ Contents

࿦จબఆͷϓϩηε ࿦จ಺༰ ॴײ Contents

ࣗ਎ॳͷ࿦จಡΈձ ࿦จબఆϓϩηε • ੎͍ͰࢀՃ • ୲౰࿦จͲ͏͠Αʁ ͱΓ͋͑ͣ(PPHMF͕ग़͍ͯ͠Δ࿦จ୳ͦ͏ • (PPHMFͱ͍͏ڊਓͷݞʹ৐͍ͬͯ͘ελΠϧ • (PPHMFBU*$.-Λݟ͚ͭΔ ڧԽֶश ϒϩάهࣄͷ͋Δຊ࿦จΛબఆ • ڧԽֶशͳΒଟগಡΊΔ • (PPHMF"*#MPHʹղઆϒϩάهࣄͷ͋Δຊ࿦จʹબఆ

࿦จબఆͷϓϩηε ࿦จ಺༰ ॴײ Contents

Πϯτϩɾഎܠ ڧԽֶशͷख๏ͱͯ͠3BJOCPX͕஌ΒΕ͍ͯΔ • %2/ ʹ౰࣌࿩୊Ͱ͋ͬͨԼهΞϧΰϦζϜΛ௥Ճ • %PVCMF2MFBSOJOH 1SJPSJUJ[FEFYQFSJFODFSFQMBZ %VFMJOHOFUXPSL .VMUJTUFQMFBSOJOH %JTUSJCVUJPOBM3- /PJTZOFUT • ஌Γ߹͍͕ϩοΫϚϯͷதϘεϥογϡͬͯᄻ͑ͯ·ͨ͠ খن໛ͳݚڀάϧʔϓ͕࠶ݱ PSཱͪ޲͔͏ͷ͸ࠔ೉ • "SDBEF-FBSOJOH&OWJSPONFOU "-& ήʔϜΛࢼߦ͢Δͷʹ 1Ͱ (16EBZ খن໛؀ڥͰ΋Ձ஋ͷ͋ΔΞϧΰͰ͋Δ͔Λূ໌Ͱ͖Δ͜ͱ͸ࠓޙͷݚڀऀʹͱͬͯॏཁ • *OMJHIUPGUIJT XFXJTIUPJOWFTUJHBUFUISFFRVFTUJPOT • 8PVME)FTTFMFUBM IBWFBSSJWFEBUUIFTBNFRVBMJUBUJWFDPODMVTJPOT IBEUIFZSVOUIFJSFYQFSJNFOUTPOBTFUPG TNBMMFSTDBMFFYQFSJNFOUT খ͍͞؀ڥͰ΋ಉ݁͡Ռͩͬͨʁ • %PUIFSFTVMUTPG)FTTFMFUBM HFOFSBMJ[FXFMMUPOPO"-&FOWJSPONFOUT PSBSFUIFJSSFTVMUT PWFSMZTQFDJGJD UPUIFDIPTFOCFODINBSL "SDBEF-FBSOJOH&OWJSPONFOUҎ֎Ͱ΋௨༻͢Δʁ • *TUIFSFTDJFOUJGJDWBMVFJODPOEVDUJOHFNQJSJDBMSFTFBSDIJOSFJOGPSDFNFOUMFBSOJOHXIFOSFTUSJDUJOHPOFTFMGUP TNBMM UPNJETDBMFFOWJSPONFOUT ڧԽֶशͰ͸খɾதن໛ʹݶఆ࣮ͨ͠ূݚڀ͸ՊֶతʹՁ஋͕͋Δ͔ʁ

3FWJTJUJOH3BJOCPX ͭͷΞϧΰΛશ෦࢖Θͳ͍ ؀ڥ • ݹయతͳTNBMMͳ؀ڥ $BSU1PMF "DSPCPU -VOBS-BOEFS BOE.PVOUBJO$BS • $16Ͱ࣌ؒະຬ • ৮ͬͨ͜ͱ͕͋Δਓʹ͸෼͔Δͱࢥ͏͕ɺຊ౰ʹTNBMM • "-&.JO"UBS "TUFSJYɺ#SFBLPVUɺ'SFFXBZɺ4FBRVFTUɺ4QBDF*OWBEFST • ؆қ൛"-& • 1Ͱd࣌ؒ ͜ΕͰ΋ݩͷ"UBSJͷHBNFʹൺ΂ͨΒ୹͍ ݁Ռ • جຊతʹ͸ɺ%2/୯඼ΑΓ͸ɺ͍͔ͭ͘ͷཁૉΛ଍ͨ͠ํ͕ྑ͍ • %JTUSJCVUJPOBM3-͸୯඼ͰೖΕͯ΋ྑ͘ͳΒͳ͍έʔε΋͋Δ

#FZPOEUIF3BJOCPX &YBNJOJOHOFUXPSLBSDIJUFDUVSFTBOECBUDITJ[FT • ݕূ಺༰ϋΠύϥ ϨΠϠʔ਺ தؒ૚ͷVOJU਺ όοναΠζ • ݁ՌϨΠϠʔ਺͸d ͩͱਫ਼౓Լ͕Δ VOJU਺͸Ҏ্ όοναΠζ Ҏ্ &YBNJOJOHEJTUSJCVUJPOQBSBNFUFSJ[BUJPOT • ݕূ಺༰%JTUSJCVUJPOBM3-ͷؔ࿈ख๏ͱͯ͠ɺҎԼͭͷख๏͕஌ΒΕ͍ͯΔ • %JTUSJCVUJPOBM3FJOGPSDFNFOU-FBSOJOHXJUI2VBOUJMF3FHSFTTJPO 23%2/ • *NQMJDJURVBOUJMFOFUXPSLTGPSEJTUSJCVUJPOBMSFJOGPSDFNFOUMFBSOJOH *2/ • ݁Ռ • 23%2/ • ݹయతͳ؀ڥ3BJOCPXNFUIPE͸جຊతʹྑ͘ͳΔҰํ3BJOCPXʹ23Λ଍͢ͱѱԽ • .JOBUBS3BJOCPX23BJOCPX ը૾௚ೖΕέʔεͰྑ͍ DPOW૚͕͋Δ//ʹ23͕༗ޮ͔΋ • *2/ • ݹయతͳ؀ڥ؀ڥʹΑΓ3BJOCPXNFUIPEͷޮՌ͸༷ʑ • .JOBUBS 3BJOCPX*3BJOCPX 23%2/ͱಉ༷ͷ݁Ռ

#FZPOEUIF3BJOCPX .VODIBVTFO3FJOGPSDFNFOU-FBSOJOH .%2/ • ͭͷಛ௃ UIFVTFPGTUPDIBTUJDQPMJDJFTBVHNFOUJOHUIFSFXBSEXJUIUIFTDBMFEMPHr QPMJDZ • ݁Ռ • %2/ • ݹయతͳ؀ڥ.VODIBVTFO͸ͳ͍ํ͕ྑ͍ • .JOBUBS%2/.%2/͕ͩɺ*2/.*2/ • 3BJOCPX • ݹయతͳ؀ڥ.3BJOCPX3BJOCPX.*3BJOCPX • .JOBUBSͳΜͱ΋͍͑ͳ͍͕ɺ "TUFSJY #SFBLPVU BOE4QBDF*OWBEFST Ͱ͸.VODIBVTFO͕ޮ͍ͯͦ͏ 3FFWBMVBUJOHUIF)VCFSMPTT • ݕূ಺༰ݩ࿦จ-PTT)VCFSMPTT PQUJNJ[FS3.41SPQ͕ͩɺ.4& BEBNʹͨ͠ΒͲ͏ͳΔ͔ • ݁Ռ • )VCFS-PTT3.41SPQ .4&"EBN • 3.41SPQݻఆͩͱɺ.4&)VCFSMPTT

1VUUJOHJUBMMUPHFUIFS 3BJOCPXGMBWPST • %2/3BJOCPX#FZPOE.FUIPE 23BJOCPX *3BJOCPX • 3BJOCPX#FZPOE.FUIPEͷதͰͲΕ͕Ұ൪ྑ͍͔͸Ұ֓ʹݴ͑ͳ͍ &OWJSPONFOUQSPQFSUJFT ֤छ؀ڥʹ͍ͭͯͷߟ࡯ɻ෇࿥ʹৄࡉ͕هࡌ • ݹయతͳ؀ڥ • $BSU1PMF୯७ͳλεΫ͕ނʹɺֶश཰ʹහײͰPQUJNJ[FSͷ҆ఆੑͷݕূʹ޲͍͍ͯΔ OPJTZOFUͱ.4&͕༗ޮ • -VOBS-BOEFS%JTUSJCVUJPOBM3-ܥͷݕূʹ޲͍͍ͯΔ • "DSPCPU .PVOUBJO$BSใु͕গͳ͘ɺ୳ࡧઓུͷݕূʹ޲͍͍ͯΔ • .JO"UBSੜ"UBSJΛ࢖͏ΑΓ΋ɺܰྔతʹ$//3-ͷݕূΛͰ͖Δ • ݩ"UBSJͩͱɺҰൠతͳը૾ೝࣝతͳ$//෦෼ͷܭࢉίετ͕େ͖͗͢Δ • 4FBRVFTU 'SFFXBZ෦෼తͳ؍ଌՄೳੑͱใुͷεύʔεੑ͔Βɺ୳ࡧख๏ͷݕূʹ޲͍͍ͯΔ

$PODMVTJPO ݶΒΕͨܭࢉ؀ڥͰ༷ʑͳݕূΛߦͬͨ • Կ͔Λൃݟ͢ΔΑΓ΋ݟ௚͢ํ͕͸Δ͔ʹ؆୯͕ͩ͜ͷݚڀͷҙਤ͸தখن໛ͷ؀ڥʹ͓͚Δ࣮ূݚڀͷଥ౰ੑͱҙٛͷשى • ܭࢉྔͷগͳ͍؀ڥͰͷݕূ΋ݚڀͷՁ஋ʹͳΔ େن໛ͳϕϯνϚʔΫΛ൷൑͍ͯ͠ΔΘ͚Ͱͳ͘ɺখن໛ͳϕϯνϚʔΫʹ΋Ձ஋͕͋Δ͜ͱΛओு • େن໛ͳϕϯνϚʔΫΛॏࢹ͠ͳ͍Α͏ٻΊ͍ͯΔͷͰ͸ͳ͍ • ݚڀऀʹ͸খن໛ͳ؀ڥΛௐࠪͷوॏͳπʔϧͱͯ͠ߟྀ͢ΔΑ͏ɺࠪಡऀʹ͸খن໛ͳ໰୊ʹয఺Λ౰࣮ͯͨূతͳݚڀΛ ൱ఆ͠ͳ͍Α͏ٻΊ͍ͯΔ͚ͩ • ݚڀͷঢ়گΛΑΓ໌֬ʹ೺Ѳ͢Δ͜ͱ͕Ͱ͖ɺ·ͨɺଟ༷Ͱܙ·Εͳ͍ίϛϡχςΟ͔Βͷ৽نࢀೖऀͷোนΛݮΒ͢͜ͱ͕ Ͱ͖Δͱ৴͍ͯ͡Δ

࿦จબఆͷϓϩηε ࿦จ಺༰ ॴײ Contents

ॴײ • 3BJOCPXपΓΛ͋Δఔ౓஌͍ͬͯͨͷͰεϥεϥಡΊͨ • ܭࢉࢿݯ͕ͳ͍ͱݚڀͷελʔτϥΠϯʹཱͯͳ͍͜ͱʹҰੴΛ౤͡Δ͜ͱ͸ڵຯਂ͍