Soft Actor-Critic 解説

Soft Actor-Critic 解説

Transcript

1. Soft Actor-Critic: Off-Policy Maximum Deep Reinforcement Learning with a Stochastic Actor
        Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine NIPS 2017 Keio Machine Learning Seminar
2. 
    3
3. 
    4 48.: • >,$ &  KJ .:6 • MGLIC
   -0ICmnistD+AF9
   )@F9 48 .: •  .:6 • M$%(
   ?=#&.: 3*.: • /2B 6E1 7H;
   .:6 • M'!<5
   "AIAlphaGoD
4.  
      (1/2) 4 Environment Agent Action Reward State
5.  
      (2/2) !" = $ ∑ &'" ( ) &*" + ,& , .&  
   5 /0 ," , ." = $0,1 +" + )/0 ,"34 , 5 ,"34 60 ," = $0,1 +" + )60(,"34 )         5 ," = argmax> Q0 sA , aA
6. Actor Critic 4 Policy    
   Critic       
   Environment
   Actor Critic 
   
7.    
   7 State  Action
     
    σ  μ
   σ μ
8. 
    8 ! = −1 %& = 0.5 ! = 0 %& = 1 ! = 1 %& = 3.0
9. Soft Actor-Critic 7
10. Soft Actor-Critic 10
11. Maximum Entropy Reinforcement Learning • Soft Actor-Critic 
    Soft Q-Learning 11         
12. Soft Actor-Critic • Maximum Entropy Reinforcement Learning "    12  #   # " # $!
    log $% &' = )* (&' )  
13. Soft Actor-Critic Algorithm 13
14.  
      14      
       
15. half-cheetah 15
16. Another Tasks 16
17. 
     17
18. 
     18 Pendulum-v0 MountainCarContinuous-v0
19. 
    • 0/) DDPGME* :  
    $1,3' • 0-08 
    7 9 =; $14 $1@6<! • 2?+.(%&$1)   05#">   19