Deep Reinforcement Learning

Transcript

1. Deep RL Abecon 20.05.2016
2. RL-type Problems • game of chess, GO, Space Invaders • balancing a unicycle • investing in stock market • running a business • making fast food • life…!
3. Markov Decision Process • S - set of states • A - set of actions (or actions for state) • P(s, s’ | a) - state change • R(s, s’ | a) - reward • ∈ [0, 1] - discount factor
5. Maximize the total discounted reward: The GOAL
6. : discount factor 0 instant gratification … patience 1
7. Value Functions http://cs.stanford.edu/people/karpathy/reinforcejs/gridworld_dp. html
8. SARSA http://cs.stanford.edu/people/karpathy/reinforcejs/gridworld_td.html
9. Q-Learning http://cs.stanford.edu/people/karpathy/reinforcejs/gridworld_td.html
10. Reinforce.js http://cs.stanford.edu/people/karpathy/reinforcejs/ // create an environment object var env = {}; env.getNumStates = function() { return 8; } env.getMaxNumActions = function() { return 4; } // create the DQN agent var spec = { alpha: 0.01 } agent = new RL.DQNAgent(env, spec); setInterval(function(){ // start the learning loop var action = agent.act(s); // s is an array of length 8 agent.learn(reward); }, 0);
11. 2013: Deep RL http://arxiv.org/abs/1312.5602
12. 2014: Google buys DeepMind
13. 2015: AlphaGO
14. Deep Q-Learning 1. Do a feedforward pass for the current state s to get predicted Q-values for all actions. 2. Do a feedforward pass for the next state s’ and calculate maximum overall network outputs max a’ Q(s’, a’). 3. Set Q-value target for action to r + γmax a’ Q(s’, a’) (use the max calculated in step 2). For all other actions, set the Q-value target to the same as originally returned from step 1, making the error 0 for those outputs. 4. Update the weights using backpropagation. http://www.nervanasys.com/demystifying-deep-reinforcement-learning/
15. Deep Q-Learning http://www.nervanasys.com/demystifying-deep-reinforcement-learning/
16. https://www.youtube.com/watch?v=32y3_iyHpBc http://gabrielecirulli.github.io/2048/
17. Asynchronous Gradient Descent http://arxiv.org/abs/1602.01783
18. http://www.rethinkrobotics.com/baxter/