Distributed prioritized experience replay

Research paper readings in my laboratory

Transcript

1. %JTUSJCVUFE
   QSJPSJUJ[FE
   FYQFSJFODF
   SFQMBZ  കຊ੖໻  
   Horgan, Dan, et al. "Distributed prioritized experience replay." arXiv preprint arXiv:1803.00933 (2018). 
2. ໨࣍  ڧԽֶश  ݚڀഎܠɼݚڀ໨త  ؔ࿈ݚڀ  ఏҊख๏  ධՁ࣮ݧ  ෼ੳ  ·ͱΊͱߟ࡯ 
3. ڧԽֶशͱ͸ Ϟσϧ͕ࣗ෼Ͱ༷ʑʹߦಈ͠ɼྑ͍ใु͕ಘΒΕΔ ߦಈΛֶश͍ͯ͘͠ख๏ ࣮༻ྫ "MQIB(P ғޟͷଧͪํΛֶश 
4. ڧԽֶशͷཁૉ  
    Policy   <ྫ> 
   ಛఆͷғޟͷ൫໘Ͱ࠷΋উͭͱࢥ͏खΛଧͭ 
   উͭ PS
   ෛ͚Δ 
   উͯΔͳΒ͜ͷखΛ࢖͍ɼෛ͚ΔͳΒ࢖Θͳ͍ Λ܁Γฦ͢͜ͱͰɼͲͷ൫໘ͰͲͷखΛଧͯ ͹উͪ΍͍͔͢Λֶश͍ͯ͘͠ ߦಈ ݁Ռ ใुؔ਺ͷߋ৽
5. ໨࣍  ڧԽֶश  ݚڀഎܠɼݚڀ໨త  ؔ࿈ݚڀ  ఏҊख๏  ධՁ࣮ݧ  ෼ੳ  ·ͱΊͱߟ࡯ 
6. ݚڀഎܠ ڧྗͳܭࢉࢿݯΛޮՌతʹར༻ͨ͠Ϟσϧ͕୆಄ n (PSJMB  n "$  n (16
   "EWBOUBHF
   "DUPS
   $SJUJD  ݱঢ়ଟ͘ͷϞσϧ͸୯ҰͷϚγϯΛ૝ఆ ݱࡏͷڧԽֶशख๏ ଟ਺ͷϚγϯΛ༻͍ͨϞσϧͷඞཁੑ 
7. ݚڀ໨త ڧԽֶशख๏"QF9ͷఏҊ n ෼ࢄγεςϜʴ༏ઌॱҐ෇͖ܦݧ࠶ੜ n ࠷৽ͷΞϧΰϦζϜͷ૊Έ߹Θͤ n ࣮ӡ༻্ʹ͓͚Δࡉ͔͍मਖ਼ ఏҊख๏ͷύϥϝʔλͷֶश΁ͷޮՌͷ෼ੳ n ܦݧΛੜ੒͢ΔXPSLFSͷ਺ n ܦݧͷอ࣋਺ 
8. ໨࣍  ڧԽֶश  ݚڀഎܠɼݚڀ໨త  ؔ࿈ݚڀ  ఏҊख๏  ධՁ࣮ݧ  ෼ੳ  ·ͱΊͱߟ࡯ 
9. ؔ࿈ݚڀ ਂ૚ֶशͷޯ഑Λฒྻʹܭࢉ͢Δख๏ ಉظɼඇಉظͰͷߋ৽ํ๏͕ఏҊ /BJSΒ͸͜ΕΒΛڧԽֶशʹద༻ n ޯ഑ͷ෼ࢄඇಉظߋ৽ n ෼ࢄܦݧੜ੒  ෼ࢄ֬཰ޯ഑߱Լ๏ 
   !$ !
   #
   
   ""%&
   !
   #"
   !
   
   !!#
   !
   !% ! !#
   !%   
   
   $&
   
    ୯ҰϚγϯɼϚϧνεϨουͰߴ͍݁Ռ
10. ؔ࿈ݚڀ ֶशͷ଎౓޲্ͨΊʹΑ͘࢖ΘΕ͍ͯΔख๏ n ༏ઌ౓Λ༻͍ͨαϯϓϦϯά͸ภΓ͕ൃੜ n ௿֬཰ͳαϯϓϧͰͷޯ഑มԽΛେ͖͘͢Δ "MBJOΒ͸ڭࢣ͋ΓֶशʹԠ༻ ෼ࢄγεςϜ΁ͷԠ༻ʹ੒ޭ  ෼ࢄԽॏཁ౓αϯϓϦϯά 
    Guillaume Alain, Alex Lamb, Chinnadhurai Sankar, Aaron Courville, and Yoshua Bengio. Variance reduction in sgd by distributed importance sampling. arXiv preprint arXiv:1511.06481, 2015. 
11. ؔ࿈ݚڀ ੜ੒ͨ͠ܦݧΛอଘ͠Կ౓΋ֶशʹ࢖༻͢Δख๏ n ੜ੒ͨ͠ܦݧΛޮ཰తʹ࢖༻Ͱ͖Δ n ݹ͍ํࡦͷܦݧΛ࢒͢͜ͱͰաద߹Λ๦͛Δ 1SJPSJUJ[FE
    &YQFSJFODF
    3FQMBZ n ༗༻ͳܦݧΛΑΓଟ͘࠶ੜ͢Δख๏ n 5%ޡࠩΛ༻͍ͯ༏ઌ౓෇͚  &YQFSJFODF
    3FQMBZ  
    -$%%('"
    $'
     %!$&)*(.$'"
    * ,$.
    " ',+
    + 
    ('
    * $'!(* & ',
    % *'$'"
    )%''$'"
    '
    , #$'"
    #$'
     *'$'"
       (&
    #-%
    (#'
    -'
    (''$+ ',('("%(-
    '
    .$
    $%. *
    *$(*$,$1 
    /) *$ '
    * )%0
    '
    ', *',$('%
    ('! * '
    ('
     *'$'"
     )* + ',,$('+
     
     
12. ໨࣍  ڧԽֶश  ݚڀഎܠɼݚڀ໨త  ؔ࿈ݚڀ  ఏҊख๏  ධՁ࣮ݧ  ෼ੳ  ·ͱΊͱߟ࡯ 
13. ఏҊख๏  "QF9ͷ֓ཁ Learner Network Replay Experiences Actor Network Environment 
      
     
     
       
     ڧԽֶशΛͭͷ໾ׂ΁෼ׂ
14. ఏҊख๏ n ֤ࣗͷߦಈՁ஋OFUXPSLͱFOWJSPONFOUΛॴ࣋ n ํࡦʹج͖ͮߦಈ͠ɼঢ়ଶભҠΛ؍ଌ n ભҠʹ༏ઌ౓Λ෇༩͠ɼ3FQMBZ
    .FNPSZʹૹ৴ n "DUPS͸ߦಈՁ஋OFUXPSLΛֶश͠ͳ͍  "DUPS େྔͷ"DUPS͕ಠཱʹߦಈ͠ɼܦݧΛେྔʹੜ੒
15. ఏҊख๏ "DUPS͔Βૹ৴͞ΕͨܦݧΛอ࣋ n શମͰͭͷ3FQMBZ
    .FNPSZΛ࣋ͭ n อ࣋Ͱ͖Δܦݧͷ্ݶ਺Λઃఆ n ্ݶΛ௒͑ͨ৔߹͸'*'0Ͱ࡟আ  3FQMBZ
    .FNPSZ -FBSOFSֶ͕श͢ΔܦݧΛେྔʹอ࣋
16. ఏҊख๏ n ܦݧΛ༏ઌॱҐʹج͖ͮαϯϓϦϯάɼֶश n ֶशʹ༻͍ͨܦݧ͸༏ઌ౓Λ࠶ܭࢉ n ҰఆִؒͰ"DUPS΁ύϥϝʔλΛૹ৴  -FBSOFS ༗༻ͳܦݧΛ༏ઌతʹֶश
17. ఏҊख๏  "QF9ͷ֓ཁͷ·ͱΊ Learner Network Replay Experiences Actor Network Environment 
      
     
     
       
     ฒྻʹܦݧΛେྔʹੜ੒ େྔͷܦݧΛอ࣋ ใुΛ૿΍͢Α͏ʹֶश
18. ఏҊख๏ (16Λେྔʹཁٻ͠ͳ͍ n -FBSOFS͸(16ΛੵΜͩϚγϯ্Ͱಈ࡞ ͭ n "DUPS͸$16ͷΈͷϚγϯ্Ͱಈ࡞ େྔ ܦݧͷޮ཰తͳར༻ n 3FQMBZ
    NFNPSZ͸શମͰڞ༗ n ܦݧʹ͸༏ઌ౓Λ෇༩  ఏҊख๏ͷಛ௃ ͭͷ"DUPSʹΑΔ༗༻ͳൃݟ͕શମͰڞ༗
19. ఏҊख๏ n ֶशΞϧΰϦζϜ n 2ؔ਺ͷۙࣅث n σʔλͷαϯϓϦϯά  -FBSOFSͷϞσϧ %PVCMF
    %FFQ
    2/FUXPSL
    NVMUJTUFQ
    CPPUTUSBQ
    UBSHFU %VFMJOH
    /FUXPSL 1SJPSJUJ[FE
    &YQFSJFODF
    3FQMBZ
20. ఏҊख๏ n "DUPS͸ݸผʹઃఆ͞Εͨ! − greedy๏ʹै͏ l ֬཰!ͰϥϯμϜʹߦಈ͢Δख๏ l ϥϯμϜʹߦಈ͢Δ͜ͱͰաద߹Λ๦͛Δ l "DUPSຖʹઃఆ͢Δ͜ͱͰଟ༷ੑΛ୲อ n ༏ઌॱҐʹج͖ͮαϯϓϦϯά͢ΔͨΊɼ ॏཁ౓αϯϓϦϯάʹΑͬͯ෼෍ͷภΓΛमਖ਼  ͦͷଞͷࡉ͔͍ઃఆ
21. ໨࣍  ڧԽֶश  ݚڀഎܠɼݚڀ໨త  ؔ࿈ݚڀ  ఏҊख๏  ධՁ࣮ݧ  ෼ੳ  ·ͱΊͱߟ࡯ 
22. ධՁ࣮ݧ n ࣮ݧ͸"UBSJͷήʔϜ FH
    ϒϩοΫ่͠ n "DUPS਺ɿ
    "DUPSʹ$16 n 
    "DUPSͷੜ੒ܦݧ਺ɿ
    '14 n શମੜ੒ܦݧ਺ɿ 
    ,
    '14
    
    
    
    
    
    
    3FQFBU n ޯ഑ͷߋ৽ɿճTFD n ܦݧ͸༰ྔ࡟ݮͷͨΊ1/(Ͱѹॖ͠อଘ  ࣮ݧઃఆ
23. ධՁ࣮ݧ  ֶशऴྃ࣌ͷੑೳൺֱ ֶश࣌ؒ είΞ n ήʔϜͷείΞͷதԝ஋ n ͸ਓؒͷείΞ n ࠷ऴείΞɼֶश࣌ؒڞʹ طଘख๏͔Βେ͖͘վળ 
    
24. ධՁ࣮ݧ  ใुͷ࣌ؒมԽ ֶश࣌ؒ ใु n ͭͷήʔϜʹ͓͚Δ ֫ಘใुͷฏۉ n ଞͷख๏ͱൺֱ͠ɼ ֫ಘใुΛΑΓૣ͘ େ͖͍ͯ͘͠Δ  
          
    
25. ධՁ࣮ݧ  ࣮ݧ݁Ռ - )1( ) ) ) 3) - 1 0 0-2 0 %) - -. %) (2 . % 50 - 0 ) -4 % 50 % 50 - 0 n "QF9͕࠷΋ߴ͍είΞΛه࿥ n ෼ࢄֶशʹΑֶͬͯश࣌ؒ΋େ෯ʹ୹ॖ
26. ໨࣍  ڧԽֶश  ݚڀഎܠɼݚڀ໨త  ؔ࿈ݚڀ  ఏҊख๏  ධՁ࣮ݧ  ෼ੳ  ·ͱΊͱߟ࡯ 
27. ෼ੳ  "DUPS਺ͱใुͷؔ܎ "DUPS਺͕ଟ͍΄ͲɼΑΓྑ͍ใुΛ֫ಘ 
    
28. ෼ੳ  3FQMBZ
    .FNPSZͱใुͷؔ܎ ༰ྔ͕ଟ͍΄Ͳɼൺֱతྑ͍ใुΛ֫ಘ 
    
29. ෼ੳ ΑΓ࠷৽ͷܦݧͷֶश͸είΞʹد༩͢Δ͔ʁ ࠷৽ͷܦݧ͸ɼ࠷৽ͷύϥϝʔλʹجͮ͘ "DUPS͕ૹ৴͢ΔܦݧΛෳ੡ͯ͠ଟΊʹૹ৴ ΑΓ৽͍͠ܦݧ͕ଟΊʹαϯϓϦϯά͞ΕΔ  ࠷৽ͷܦݧ
30. ෼ੳ  ࠷৽ͷܦݧͱใुͷؔ܎  !
       ࠷৽ͷܦݧͷֶशͱ ใु͸݁ͼ͍͍ͭͯͳ͍
31. ෼ੳ n "DUPS਺Λ૿΍͢ͱใु͕૿Ճ l ہॴղ΁ؕΔ͜ͱΛ๦͛Δಇ͖ l େྔͷ୳ࡧͰɼ༗༻ͳܦݧΛ֫ಘ n 3FQMBZ
    .FNPSZΛ૿΍͢͜ͱͰใु͕૿Ճ n ࠷৽ͷܦݧͱใुʹ͸௚઀తͳد༩͸ͳ͍  ෼ੳ݁Ռ·ͱΊ ༗༻ͳܦݧΛΑΓ௕͘อ࣋Ͱ͖ͨ ܦݧͷਫ૿͠͸ଟ༷ੑΛ௿͘͠ɼ ύϑΥʔϚϯεΛԼ͛Δ
32. ໨࣍  ڧԽֶश  ݚڀഎܠɼݚڀ໨త  ؔ࿈ݚڀ  ఏҊख๏  ධՁ࣮ݧ  ෼ੳ  ·ͱΊͱߟ࡯ 
33. ·ͱΊͱߟ࡯ n ෼ࢄʴ༏ઌ౓෇͖ܦݧ࠶ੜͷ'SBNF
    XPSLΛఏҊ n "QF9͸ֶ࣮࣌ؒश଎౓ɼ࠷ऴੑೳʹ͓͍ͯ࠷΋ྑ ͍ੑೳΛࣔͨ͠ n աద߹͸ڧԽֶशʹ͓͚Δେ͖ͳ໰୊Ͱɼࠓճ͸σʔ λΛେྔʹੜ੒͢Δ୯७ͳํ๏͕ޮՌతͰ͋Δ͜ͱΛ ࣔͨ͠ n কདྷతʹ͸σʔλΛޮ཰Α͘࢖͏ํ๏Λ໛ࡧ͢Δ΂͖  ·ͱΊ
34. ·ͱΊͱߟ࡯ "QF9͸ܦݧΛߴ଎ʹେྔʹूΊΔख๏ ෳࡶͳλεΫͰ͸ঢ়ଶ!"͕େྔʹଘࡏ େྔͷܦݧͷੜ੒͕ঢ়ଶ!"Λ޿͘Χόʔֶ͠श͕ਐΜͩ ݱঢ়ɼϥϯμϜ୳ࡧʹΑͬͯະ஌ͷߦಈΛܦݧ ൃੜස౓ͷ௿͍ঢ়ଶ!"Λॏ఺తʹ୳ࡧ͢Δख๏  ߟ࡯
35. 2MFBSOJOHͷ2ؔ਺ͷߋ৽ࣜ ! "# , %# ← ! "# , %# + α(*#+, + - max 12∈4 52 ! "#+, , 67 − ! "# , %# ) "# : ࣌ࠁ;ͷঢ়ଶ %# :࣌ࠁ;ͷߦಈ ! "# , %# ঢ়ଶ"#Ͱߦಈ%#Λͱͬͨ৔߹ͷਪఆใु *#ɿ࣌ࠁ;ʹ͓͚Δใु αɿֶश཰ -ɿׂҾ཰  5%ޡࠩʢ5FNQPSBMMZ
    %JGGFSFODFʣ ෇࿥ 5%ޡࠩ ਪఆใुͱ࣮ࡍͷใुͷࠩ