Human-in-the-Loop Reinforcement Learning (Pieter Abbeel, UC Berkeley | Covariant | The Robot Brains Podcast)

Deep reinforcement learning (Deep RL) has seen many successes, including learning to play Atari games, the classical game of Go, robotic locomotion and manipulation. However, now that Deep RL has become fairly capable of optimizing reward, a new challenge has arisen: How to choose the reward function that is to be optimized? Indeed, this often becomes the key engineering time sink for practitioners. In this talk, I will present some recent progress on human-in-the-loop reinforcement learning. The newly proposed algorithm, PEBBLE, empowers a human supervisor to directly teach an AI agent new skills without the usual extensive reward engineering or curriculum design efforts.

Transcript

1. Human-in-the-Loop Reinforcement Learning Pieter Abbeel UC Berkeley & Covariant & The Robot Brains
2. Fast Progress on Deep RL 2013 Atari (DQN) [Deepmind] Pong Enduro Beamrider Q*bert Pieter Abbeel -- UC Berkeley | Covariant | The Robot Brains
3. [Source: Mnih et al., Nature 2015 (DeepMind) ] Deep Q-Network (DQN): From Pixels to Joystick Commands Pieter Abbeel -- UC Berkeley | Covariant | The Robot Brains
4. Fast Progress on Deep RL AlphaGo Silver et al, Nature 2015 AlphaGoZero Silver et al, Nature 2017 AlphaZero Silver et al, 2017 Tian et al, 2016; Maddison et al, 2014; Clark et al, 2015 2013 Atari (DQN) [Deepmind] 2015 AlphaGo [Deepmind] Pieter Abbeel -- UC Berkeley | Covariant | The Robot Brains
5. Fast Progress on Deep RL [Schulman, Moritz, Levine, Jordan, Abbeel, ICLR 2016] 2013 Atari (DQN) [Deepmind] 2015 AlphaGo [Deepmind] 2016 3D locomotion (TRPO+GAE) [Berkeley]
6. Fast Progress on Deep RL [Levine*, Finn*, Darrell, Abbeel, JMLR 2016] 2013 Atari (DQN) [Deepmind] 2015 AlphaGo [Deepmind] 2016 3D locomotion (TRPO+GAE) [Berkeley] 2016 Real Robot Manipulation (GPS) [Berkeley] Pieter Abbeel -- UC Berkeley | Covariant | The Robot Brains
7. Fast Progress on Deep RL 2013 Atari (DQN) [Deepmind] 2015 AlphaGo [Deepmind] 2016 3D locomotion (TRPO+GAE) [Berkeley] 2016 Real Robot Manipulation (GPS) [Berkeley] 2019 Rubik’s Cube (PPO+DR) [OpenAI] Pieter Abbeel -- UC Berkeley | Covariant | The Robot Brains
8. Challenge: Designing Suitable Reward Decision (actions) Consequences (observations, rewards) Pieter Abbeel -- UC Berkeley | Covariant | The Robot Brains
9. Challenge: Designing Suitable Reward Decision (actions) Consequences (observations, rewards) Hard tasks to define a reward (e.g. cooking) Pieter Abbeel -- UC Berkeley | Covariant | The Robot Brains
10. Challenge: Designing Suitable Reward Decision (actions) Consequences (observations, rewards) Hard tasks to define a reward (e.g. cooking) Reward exploitation https://openai.com/blog/faulty-reward-functions/
11. What is an Alternative Solution? Pieter Abbeel -- UC Berkeley | Covariant | The Robot Brains
12. What is an Alternative Solution? • Putting (non-expert) humans into the agent learning loop! RL algorithm Environment action observation Reward 💰 Human Behaviors of an agent 👩💻 Feedback Pieter Abbeel -- UC Berkeley | Covariant | The Robot Brains
13. What is an Alternative Solution? • Putting (non-expert) humans into the agent learning loop! RL algorithm Environment action observation Human Behaviors of an agent 👩💻 Feedback Preference 👩💻 Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S. and Amodei, D., Reward learning from human preferences and demonstrations in atari. In NeurIPS, 2018. Christiano, P., Leike, J., Brown, T.B., Martic, M., Legg, S. and Amodei, D., Deep reinforcement learning from human preferences. NeurIPS, 2017.
14. What is an Alternative Solution? • Putting (non-expert) humans into the agent learning loop! RL algorithm Environment action observation Human Behaviors of an agent 👩💻 Feedback Human has ability to interactively guide agents according to their progress à Can teach harder tasks, where we can’t easily define the reward à Can avoid reward exploitation Pieter Abbeel -- UC Berkeley | Covariant | The Robot Brains
15. The PEBBLE Algorithm • Step 1. Collect samples via interactions with environment
16. The PEBBLE Algorithm • Step 1. Collect samples via interactions with environment • Step 2. Collect human preferences 👩💻
17. The PEBBLE Algorithm • Step 1. Collect samples via interactions with environment • Step 2. Collect human preferences • Step 3. Optimize a reward model using cross entropy loss 👩💻
18. Learning Reward from Preferences • Fitting a reward model [1] ◦ Main idea: formulate this problem as a binary classification! ◦ By following the Bradley-Terry model [2], we can model a preference predictor as follows: [1] Christiano, P., Leike, J., Brown, T.B., Martic, M., Legg, S. and Amodei, D., Deep reinforcement learning from human preferences. NeurIPS, 2017. [2] Bradley, R.A. and Terry, M.E., Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4), pp.324-345, 1952.
19. The PEBBLE Algorithm • Step 1. Collect samples via interactions with environment • Step 2. Collect human preferences • Step 3. Optimize a reward model using cross entropy loss • Step 4. Optimize a policy using off-policy algorithms 👩💻
20. The PEBBLE Algorithm • Step 1. Collect samples via interactions with environment • Step 2. Collect human preferences • Step 3. Optimize a reward model using cross entropy loss • Step 4. Optimize a policy using off-policy algorithms Repeat step 1 - step 4 👩💻
21. Unsupervised Pre-training: APT • Obtaining a good initial state space coverage is important! ◦ Human can’t convey much meaningful information to the agent Pieter Abbeel -- UC Berkeley | Covariant | The Robot Brains
22. Unsupervised Pre-training: APT • Obtaining a good initial state space coverage is important! ◦ Human can’t convey much meaningful information to the agent Behavior from random exploration policy Behavior from pre-trained policy Pieter Abbeel -- UC Berkeley | Covariant | The Robot Brains
23. Can Humans Teach Novel Behaviors with PEBBLE? Pieter Abbeel -- UC Berkeley | Covariant | The Robot Brains
24. Can Humans Teach Novel Behaviors with PEBBLE? • 40 queries in less than 5 mins Counter clockwise Clockwise Pieter Abbeel -- UC Berkeley | Covariant | The Robot Brains
25. Can Humans Teach Novel Behaviors with PEBBLE? • 200 queries in less than 30 mins Waving right front leg Waving left front leg Pieter Abbeel -- UC Berkeley | Covariant | The Robot Brains
26. Can Humans Teach Novel Behaviors with PEBBLE? • 400 queries in less than one hour Pieter Abbeel -- UC Berkeley | Covariant | The Robot Brains
27. Can We Avoid Reward Exploitation? Pieter Abbeel -- UC Berkeley | Covariant | The Robot Brains
28. Can We Avoid Reward Exploitation? SAC with task reward on walker, walk (use one leg even if score ~=1000) Pieter Abbeel -- UC Berkeley | Covariant | The Robot Brains
29. Can We Avoid Reward Exploitation? • 150 queries in less than 20 mins SAC with task reward on walker, walk (use one leg even if score ~=1000) SAC trained with human feedback (use both legs) Pieter Abbeel -- UC Berkeley | Covariant | The Robot Brains
30. Comparison: Locomotion Tasks • Learning curves (10 random seeds) * Asymptotic performance of PPO and Preference PPO is indicated by dotted lines of the corresponding color
31. n Reinforcement learning algorithms are become effective at optimizing reward n BUT: outside of games, reward can be hard to specify correctly n PEBBLE enables effective human-in-the-loop reinforcement learning Summary Pieter Abbeel -- UC Berkeley | Covariant | The Robot Brains