Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[ICML2021 論文読み会]Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research

7e06fe88ab42d892ac64d3fb95b1d56e?s=47 ninohira
August 18, 2021
1k

[ICML2021 論文読み会]Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research

7e06fe88ab42d892ac64d3fb95b1d56e?s=128

ninohira

August 18, 2021
Tweet

Transcript

  1. ICML2021 論文読み会 Revisiting Rainbow: Promoting more Insightful and Inclusive Deep

    Reinforcement Learning Research Masato Ninohira, LINE Corporation 2021.08
  2. ࿦จબఆͷϓϩηε ࿦จ಺༰ ॴײ Contents

  3. ࿦จબఆͷϓϩηε ࿦จ಺༰ ॴײ Contents

  4. ࣗ਎ॳͷ࿦จಡΈձ ࿦จબఆϓϩηε • ੎͍ͰࢀՃ • ୲౰࿦จͲ͏͠Αʁ ͱΓ͋͑ͣ(PPHMF͕ग़͍ͯ͠Δ࿦จ୳ͦ͏ • (PPHMFͱ͍͏ڊਓͷݞʹ৐͍ͬͯ͘ελΠϧ •

    (PPHMFBU*$.-Λݟ͚ͭΔ ڧԽֶश ϒϩάهࣄͷ͋Δຊ࿦จΛબఆ • ڧԽֶशͳΒଟগಡΊΔ • (PPHMF"*#MPHʹղઆϒϩάهࣄͷ͋Δຊ࿦จʹબఆ
  5. ࿦จબఆͷϓϩηε ࿦จ಺༰ ॴײ Contents

  6. Πϯτϩɾഎܠ ڧԽֶशͷख๏ͱͯ͠3BJOCPX͕஌ΒΕ͍ͯΔ • %2/ ʹ౰࣌࿩୊Ͱ͋ͬͨԼهΞϧΰϦζϜΛ௥Ճ •  %PVCMF2MFBSOJOH  1SJPSJUJ[FEFYQFSJFODFSFQMBZ

     %VFMJOHOFUXPSL  .VMUJTUFQMFBSOJOH  %JTUSJCVUJPOBM3-  /PJTZOFUT • ஌Γ߹͍͕ϩοΫϚϯͷதϘεϥογϡͬͯᄻ͑ͯ·ͨ͠ খن໛ͳݚڀάϧʔϓ͕࠶ݱ PSཱͪ޲͔͏ͷ͸ࠔ೉ • "SDBEF-FBSOJOH&OWJSPONFOU "-& ήʔϜΛࢼߦ͢Δͷʹ 1Ͱ (16EBZ খن໛؀ڥͰ΋Ձ஋ͷ͋ΔΞϧΰͰ͋Δ͔Λূ໌Ͱ͖Δ͜ͱ͸ࠓޙͷݚڀऀʹͱͬͯॏཁ • *OMJHIUPGUIJT XFXJTIUPJOWFTUJHBUFUISFFRVFTUJPOT • 8PVME)FTTFMFUBM  IBWFBSSJWFEBUUIFTBNFRVBMJUBUJWFDPODMVTJPOT IBEUIFZSVOUIFJSFYQFSJNFOUTPOBTFUPG TNBMMFSTDBMFFYQFSJNFOUT  খ͍͞؀ڥͰ΋ಉ݁͡Ռͩͬͨʁ • %PUIFSFTVMUTPG)FTTFMFUBM  HFOFSBMJ[FXFMMUPOPO"-&FOWJSPONFOUT PSBSFUIFJSSFTVMUT PWFSMZTQFDJGJD UPUIFDIPTFOCFODINBSL  "SDBEF-FBSOJOH&OWJSPONFOUҎ֎Ͱ΋௨༻͢Δʁ • *TUIFSFTDJFOUJGJDWBMVFJODPOEVDUJOHFNQJSJDBMSFTFBSDIJOSFJOGPSDFNFOUMFBSOJOHXIFOSFTUSJDUJOHPOFTFMGUP TNBMM UPNJETDBMFFOWJSPONFOUT ڧԽֶशͰ͸খɾதن໛ʹݶఆ࣮ͨ͠ূݚڀ͸ՊֶతʹՁ஋͕͋Δ͔ʁ
  7. 3FWJTJUJOH3BJOCPX ͭͷΞϧΰΛશ෦࢖Θͳ͍ ؀ڥ • ݹయతͳTNBMMͳ؀ڥ $BSU1PMF "DSPCPU -VOBS-BOEFS BOE.PVOUBJO$BS •

    $16Ͱ࣌ؒະຬ • ৮ͬͨ͜ͱ͕͋Δਓʹ͸෼͔Δͱࢥ͏͕ɺຊ౰ʹTNBMM • "-&.JO"UBS "TUFSJYɺ#SFBLPVUɺ'SFFXBZɺ4FBRVFTUɺ4QBDF*OWBEFST • ؆қ൛"-& • 1Ͱd࣌ؒ ͜ΕͰ΋ݩͷ"UBSJͷHBNFʹൺ΂ͨΒ୹͍ ݁Ռ • جຊతʹ͸ɺ%2/୯඼ΑΓ͸ɺ͍͔ͭ͘ͷཁૉΛ଍ͨ͠ํ͕ྑ͍ • %JTUSJCVUJPOBM3-͸୯඼ͰೖΕͯ΋ྑ͘ͳΒͳ͍έʔε΋͋Δ
  8. #FZPOEUIF3BJOCPX  &YBNJOJOHOFUXPSLBSDIJUFDUVSFTBOECBUDITJ[FT • ݕূ಺༰ϋΠύϥ ϨΠϠʔ਺ தؒ૚ͷVOJU਺ όοναΠζ • ݁ՌϨΠϠʔ਺͸d

    ͩͱਫ਼౓Լ͕Δ VOJU਺͸Ҏ্ όοναΠζ Ҏ্ &YBNJOJOHEJTUSJCVUJPOQBSBNFUFSJ[BUJPOT • ݕূ಺༰%JTUSJCVUJPOBM3-ͷؔ࿈ख๏ͱͯ͠ɺҎԼͭͷख๏͕஌ΒΕ͍ͯΔ • %JTUSJCVUJPOBM3FJOGPSDFNFOU-FBSOJOHXJUI2VBOUJMF3FHSFTTJPO 23%2/ • *NQMJDJURVBOUJMFOFUXPSLTGPSEJTUSJCVUJPOBMSFJOGPSDFNFOUMFBSOJOH *2/ • ݁Ռ • 23%2/ • ݹయతͳ؀ڥ 3BJOCPXNFUIPE͸جຊతʹྑ͘ͳΔҰํ3BJOCPXʹ23Λ଍͢ͱѱԽ • .JOBUBS3BJOCPX23BJOCPX ը૾௚ೖΕέʔεͰྑ͍ DPOW૚͕͋Δ//ʹ23͕༗ޮ͔΋ • *2/ • ݹయతͳ؀ڥ؀ڥʹΑΓ3BJOCPXNFUIPEͷޮՌ͸༷ʑ • .JOBUBS 3BJOCPX*3BJOCPX 23%2/ͱಉ༷ͷ݁Ռ
  9. #FZPOEUIF3BJOCPX  .VODIBVTFO3FJOGPSDFNFOU-FBSOJOH .%2/ • ͭͷಛ௃ UIFVTFPGTUPDIBTUJDQPMJDJFTBVHNFOUJOHUIFSFXBSEXJUIUIFTDBMFEMPHr QPMJDZ • ݁Ռ

    • %2/ • ݹయతͳ؀ڥ.VODIBVTFO͸ͳ͍ํ͕ྑ͍ • .JOBUBS%2/.%2/͕ͩɺ*2/.*2/ • 3BJOCPX • ݹయతͳ؀ڥ.3BJOCPX3BJOCPX.*3BJOCPX • .JOBUBSͳΜͱ΋͍͑ͳ͍͕ɺ "TUFSJY #SFBLPVU BOE4QBDF*OWBEFST Ͱ͸.VODIBVTFO͕ޮ͍ͯͦ͏ 3FFWBMVBUJOHUIF)VCFSMPTT • ݕূ಺༰ݩ࿦จ-PTT)VCFSMPTT PQUJNJ[FS3.41SPQ͕ͩɺ.4& BEBNʹͨ͠ΒͲ͏ͳΔ͔ • ݁Ռ • )VCFS-PTT 3.41SPQ .4& "EBN • 3.41SPQݻఆͩͱɺ.4&)VCFSMPTT
  10. 1VUUJOHJUBMMUPHFUIFS 3BJOCPXGMBWPST • %2/3BJOCPX #FZPOE.FUIPE 23BJOCPX *3BJOCPX • 3BJOCPX #FZPOE.FUIPEͷதͰͲΕ͕Ұ൪ྑ͍͔͸Ұ֓ʹݴ͑ͳ͍

    &OWJSPONFOUQSPQFSUJFT ֤छ؀ڥʹ͍ͭͯͷߟ࡯ɻ෇࿥ʹৄࡉ͕هࡌ • ݹయతͳ؀ڥ • $BSU1PMF୯७ͳλεΫ͕ނʹɺֶश཰ʹහײͰPQUJNJ[FSͷ҆ఆੑͷݕূʹ޲͍͍ͯΔ OPJTZOFUͱ.4&͕༗ޮ • -VOBS-BOEFS%JTUSJCVUJPOBM3-ܥͷݕূʹ޲͍͍ͯΔ • "DSPCPU .PVOUBJO$BSใु͕গͳ͘ɺ୳ࡧઓུͷݕূʹ޲͍͍ͯΔ • .JO"UBSੜ"UBSJΛ࢖͏ΑΓ΋ɺܰྔతʹ$// 3-ͷݕূΛͰ͖Δ • ݩ"UBSJͩͱɺҰൠతͳը૾ೝࣝతͳ$//෦෼ͷܭࢉίετ͕େ͖͗͢Δ • 4FBRVFTU 'SFFXBZ෦෼తͳ؍ଌՄೳੑͱใुͷεύʔεੑ͔Βɺ୳ࡧख๏ͷݕূʹ޲͍͍ͯΔ
  11. $PODMVTJPO ݶΒΕͨܭࢉ؀ڥͰ༷ʑͳݕূΛߦͬͨ • Կ͔Λൃݟ͢ΔΑΓ΋ݟ௚͢ํ͕͸Δ͔ʹ؆୯͕ͩ͜ͷݚڀͷҙਤ͸தখن໛ͷ؀ڥʹ͓͚Δ࣮ূݚڀͷଥ౰ੑͱҙٛͷשى • ܭࢉྔͷগͳ͍؀ڥͰͷݕূ΋ݚڀͷՁ஋ʹͳΔ େن໛ͳϕϯνϚʔΫΛ൷൑͍ͯ͠ΔΘ͚Ͱͳ͘ɺখن໛ͳϕϯνϚʔΫʹ΋Ձ஋͕͋Δ͜ͱΛओு • େن໛ͳϕϯνϚʔΫΛॏࢹ͠ͳ͍Α͏ٻΊ͍ͯΔͷͰ͸ͳ͍ •

    ݚڀऀʹ͸খن໛ͳ؀ڥΛௐࠪͷوॏͳπʔϧͱͯ͠ߟྀ͢ΔΑ͏ɺࠪಡऀʹ͸খن໛ͳ໰୊ʹয఺Λ౰࣮ͯͨূతͳݚڀΛ ൱ఆ͠ͳ͍Α͏ٻΊ͍ͯΔ͚ͩ • ݚڀͷঢ়گΛΑΓ໌֬ʹ೺Ѳ͢Δ͜ͱ͕Ͱ͖ɺ·ͨɺଟ༷Ͱܙ·Εͳ͍ίϛϡχςΟ͔Βͷ৽نࢀೖऀͷোนΛݮΒ͢͜ͱ͕ Ͱ͖Δͱ৴͍ͯ͡Δ
  12. ࿦จબఆͷϓϩηε ࿦จ಺༰ ॴײ Contents

  13. ॴײ • 3BJOCPXपΓΛ͋Δఔ౓஌͍ͬͯͨͷͰεϥεϥಡΊͨ • ܭࢉࢿݯ͕ͳ͍ͱݚڀͷελʔτϥΠϯʹཱͯͳ͍͜ͱʹҰੴΛ౤͡Δ͜ͱ͸ڵຯਂ͍