Reinforcement Learning

90-minute presentation introducing reinforcement learning to the CS229 (machine learning) class of Spring 2014, at KAUST.

1. Reinforcement Learning CS 229: Machine Learning Emaad Ahmed Manzoor April 28, 2014
2. Introduction
3. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Reinforcement Learning - Learning by interacting with the environment
4. A Simple Example: k-armed bandit Target: earn money AMAP Task: decide which lever to pull Agent: you State: single state (one slot machine) Action: choose one lever to pull Reward: money given by the machine after one action (immediate reward after a single action) Xiangliang Zhang, KAUST AMCS/CS229: Machine Learning Supervised learning? Need a teacher to tell you which one?
5. Another Example: a machine to play chess Target: win the game Task: decide a sequence of moves Agent: player State: state of the board(environment) Action: decide a legal move Reward: win or lose (when game is over) Xiangliang Zhang, KAUST AMCS/CS229: Machine Learning Is there a supervised learner who can tell you how to move?
6. Target: reach the exit Task: decide a sequence of movements Agent: robot State: position of the robot in the maze Action: choose one of the 4 directions without hitting the walls Reward: length of trajectories (playing time) (when the robot reaches the exit) Xiangliang Zhang, KAUST AMCS/CS229: Machine Learning Another Example: a robot in a maze
7. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Reinforcement Learning - Learning by interacting with the environment - Delayed rewards Credit attribution
8. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Reinforcement Learning - Learning by interacting with the environment - Delayed rewards Credit attribution - Explore and exploit
9. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning VS Supervised Learning - Supervision provides training samples with the right answer Instructive feedback, independent of the action taken Difficult to provide in many cases - Reinforcement provides only rewards based on states and actions Evaluative feedback, dependant on the action taken The agent selects actions over time to maximise its total reward
10. RL vs Supervised Learning Xiangliang Zhang, KAUST AMCS/CS229: Machine Learning Reinforcement Learning Supervised Learning Learning with a critic Learning with a teacher How well we have been doing in the past What’s good or bad Learn to generate an internal value for the intermediate states or actions (how good they are) Learn to minimize general risk on predicting Exploration and Exploitation Over-fitting and Under-fitting
11. Successful examples of RL • Robocup Soccer Teams Stone & Veloso, Reidmiller etal. – World’s best player of simulated soccer, 1999; Runner--up 2000 • Dynamic Channel Assignment Singh & Bertsekas, Nie &Haykin – World's best assigner of radio channels to mobile telephone calls • Elevator Control Crites & Barto – (Probably) world's best down---peak elevator controller • Many Robots – navigation, bi--pedal walking, grasping, switching between skills... • TD---Gammonand Jellyfish Tesauro, Dahl – World's best backgammon player Xiangliang Zhang, KAUST AMCS/CS229: Machine Learning
12. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Elements of Reinforcement Learning - Model of the environment - Policy - Reward function - Value function
13. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Elements of Reinforcement Learning - Model of the environment States State transitions: stochastic or deterministic - Policy - Reward function - Value function
14. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Elements of Reinforcement Learning - Model of the environment - Policy Maps states to actions to be taken in those states Lookup table or function Stochastic or deterministic - Reward function - Value function
15. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Elements of Reinforcement Learning - Model of the environment - Policy - Reward function Rewards are numbers indicating the desirability of a state Provided by the environment May be stochastic or deterministic - Value function
16. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Elements of Reinforcement Learning - Model of the environment - Policy - Reward function - Value function Values are numbers indicating the long-term desirability of a state The total amount of reward the agent can expect to accumulate, starting from that state Estimated from sequences of observed actions and rewards
17. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Mechanics of Reinforcement Learning States S = { S 1 , S 2 , … } Reward Functions R, Q ENVIRONMENT AGENT Policy π Value Fn V Action a ~ π(a|s) New state s' ~ P(S' | a, S) Reward r' ~ R(R' | s), or r' ~ Q(R' | s, a)
18. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Mechanics of Reinforcement Learning Foundations of Machine Learning. Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. t Discrete time step s t State at time t r t Reward when in state s t at time t a t Action taken at time t, leading to new state s t+1 with reward r t+1
19. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Goals - Maximise the expected return (finite, episodic tasks) - Maximise the expected discounted return (infinite, continuous) (short-sighted) (far-sighted)
20. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning - Unified notation for episodic and continuous tasks - Every episode ends in an absorbing state that always produces a reward 0 - Both episodic and continuous tasks can now maximise: Goals
21. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Examples of Reinforcement Learning - Actions: { Forward, Backward, No Thrust } - Reward: -1 for all state transitions, except the goal, where the reward is 0 - Return: -Time to reach the goal The agent does not have enough thrust to simply drive up the hill. It minimizes the time it takes to reach the goal, by learning to use momentum to its advantage Reinforcement Learing: A Tutorial. Mance E. Harmon, Stephanie S. Harmon
22. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Examples of Reinforcement Learning - Chess: What is the reward? - Robot in a maze: What is the reward? - Flappy Bird RL hack: http://sarvagyavaish.github.io/FlappyBirdRL/
23. Bandit Problems
24. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning The Multi-Armed Bandit A B C $1 every 5 rounds $1 every 7 rounds $1 every 3 rounds 1000 rounds available
25. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning The Multi-Armed Bandit A B C $1 every 5 rounds $1 every 7 rounds $1 every 3 rounds 1000 rounds available
26. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning The Multi-Armed Bandit A B C $1 every 5 rounds $1 every 7 rounds $1 every 3 rounds 0 1 0 ESTIM. VALUE 0 1 1
27. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning The Multi-Armed Bandit A B C $1 every 5 rounds $1 every 7 rounds $1 every 3 rounds 0 1 1 0 0 0 ESTIM. VALUE 0.5 0.5 0
28. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning The Multi-Armed Bandit a 1 a 2 a k q*(a 1 ) Actual Value . . . q*(a 2 ) q*(a k ) Estimated Value Q t (a 1 ) Q t (a 2 ) Q t (a k )
29. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning The Multi-Armed Bandit a 1 a 2 a k q*(a 1 ) Actual Value . . . q*(a 2 ) q*(a k ) Estimated Value Q t (a 1 ) Q t (a 2 ) Q t (a k ) Estimate value by averaging rewards Q t (a) = R 1 + R 2 + … R k K a
30. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning The Multi-Armed Bandit a 1 a 2 a k q*(a 1 ) Actual Value . . . q*(a 2 ) q*(a k ) Estimated Value Q t (a 1 ) Q t (a 2 ) Q t (a k ) Estimate value by averaging rewards incrementally Q t+1 (a) = Q t (a) + 1 ( R t - Q t (a) ) t
31. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning The Multi-Armed Bandit a 1 a 2 a k q*(a 1 ) Actual Value . . . q*(a 2 ) q*(a k ) Estimated Value Q t (a 1 ) Q t (a 2 ) Q t (a k ) Action selection policy: greedy Always select the action with the maximum estimated value
32. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning The Multi-Armed Bandit a 1 a 2 a k q*(a 1 ) Actual Value . . . q*(a 2 ) q*(a k ) Estimated Value Q t (a 1 ) Q t (a 2 ) Q t (a k ) Action selection policy: ϵ-greedy With probability ϵ, select an action uniformly at random
33. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning The Multi-Armed Bandit a 1 a 2 a k q*(a 1 ) Actual Value . . . q*(a 2 ) q*(a k ) Estimated Value Q t (a 1 ) Q t (a 2 ) Q t (a k ) Softmax action selection policy With probability ϵ, select an action with probability
34. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning The Contextual Multi-Armed Bandit a 1 a 2 a k q*(a 1 ) . . . q*(a 2 ) q*(a k ) Q t (a 1 ) Q t (a 2 ) Q t (a k ) Policy π(s) = a a 1 a 2 a k q*(a 1 ) . . . q*(a 2 ) q*(a k ) Q t (a 1 ) Q t (a 2 ) Q t (a k ) a 1 a 2 a k q*(a 1 ) . . . q*(a 2 ) q*(a k ) Q t (a 1 ) Q t (a 2 ) Q t (a k ) . . . s 1 s 2 s 3
35. Formalization
36. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Markov Property
37. A tuple ( , where: - S Set of states - A Set of actions - P sa State transition probabilities; a distribution over the state space - Discount factor - R Reward function of a state (or a state-action pair) Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Markov Decision Processes
38. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning MDP Dynamics
39. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning MDP Dynamics
40. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning MDP Dynamics
41. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning MDP Goal Choose actions over time so as to maximise the expected value of the total payoff
42. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Policy and Value Functions Policy
43. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Policy and Value Functions Value function for a policy, starting at state s
44. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Policy and Value Functions Value function for a policy, for taking an action a in state s
45. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Bellman Equations Current state-dependent reward and deterministic policy
46. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Bellman Equations Current state-dependent reward and stochastic policy
47. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Bellman Equations Current and next state-dependent reward and stochastic policy
48. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Optimal Value Function
49. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Bellman Optimality Equations Value of a state under an optimal policy is the expected return for the best action from that state
50. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Optimal Policy for all States This is the greedy policy with respect to V* (requires a one-step search) or Q* (does not require any search)
51. Transition probability matrix Value column matrix Reward column matrix Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Solving Bellman Equations Foundations of Machine Learning. Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar.
52. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Solving Bellman Equations Foundations of Machine Learning. Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar.
53. Planning
54. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Value Iteration
55. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Value Iteration - Has converged when the largest V(s) update is smaller than ϵ - Updates to V(s) may be in a batch or for single states
56. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Policy Iteration
57. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Policy Iteration VS Value Iteration - Policy iteration converges in fewer iterations But solving a large system of equations in each iteration is expensive - In practice, value iteration is used more often
58. Learning
59. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Mean Estimation
60. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Mean Estimation
61. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Model-based VS. Model-free Learning Reinforcement learning: The Good, The Bad and The Ugly. Peter Dayana and Yael Niv.
62. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning Model-based VS. Model-free Learning - Model-based: Try to model the environment (state transition probabilities, reward probabilities) Eg. Mean estimation - Model-free: Model the value functions directly, without modeling the environment Eg. Action selection, TD learning, Q learning, Sarsa algorithm
63. Emaad Ahmed Manzoor April 28, 2014 Reinforcement Learning References - Reinforcement Learning: An Introduction. Sutton and Barto. - Foundations of Machine Learning. Mehryar Mohri. - Reinforcement Learning and Control. CS229 lecture notes, Andrew Ng.