OpenTalks.AI - Александр Новиков​, Обзор главных работ и результатов в RL в 2020 году

Transcript

1. (Some) cool RL papers from 2020 Alex Novikov
2. What is Reinforcement Learning (RL) Environment State, Reward Reward: 1 Go left Action
3. Recap: how does RL work? In a few (oversimplified) words 1) Try random stuff 2) “Reinforce” (do more in the future) the stuff that worked better according to reward
4. Learning to Summarize with Human Feedback Nisan Stiennon, Long Ouyang, et al., OpenAI [blog post, Neurips paper] 1) Finetune GPT-3 to summarize Reddit posts into TL;DRs 2) Sample a lot of artificial summaries and ask humans to compare pairs of them
5. Learning to Summarize with Human Feedback 1) Finetune GPT-3 to summarize Reddit posts into TL;DRs 2) Sample a lot of artificial summaries and ask humans to compare pairs of them 3) Train a neural net (reward model) to predict human labels
6. Learning to Summarize with Human Feedback 1) Finetune GPT-3 to summarize Reddit posts into TL;DRs 2) Sample a lot of artificial summaries and ask humans to compare pairs of them 3) Train a neural net (reward model) to predict human labels 4) Use RL to finutene the summarizer using the learned reward function (e.g. preferring longer summaries)
7. Autonomous navigation of stratospheric balloons using reinforcement learning Marc G. Bellemare, Salvatore Candido, et al., Google Brain [Nature paper] Training in simulation Deploying in real world
8. Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design Michael Dennis, Natasha Jaques, et al., Google Brain [talk, Neurips paper]
9. Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design
10. Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design
11. Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design
12. Asymmetric self-play for automatic goal discovery in robotic manipulation Matthias Plappert, Raul Sampedro, et al., OpenAI [paper, blogpost] Similarly, can we generate goals?
13. Asymmetric self-play for automatic goal discovery in robotic manipulation Similarly, can we generate goals? Alice end-state = goal for Bob
14. Asymmetric self-play for automatic goal discovery in robotic manipulation Similarly, can we generate goals? Alice end-state = goal for Bob Bob’s reward to reach the same state (preferably faster than Alice) Alice reward is to make Bob fail
15. Asymmetric self-play for automatic goal discovery in robotic manipulation Similarly, can we generate goals? Alice end-state = goal for Bob Bob’s reward to reach the same state (preferably faster than Alice) Alice reward is to make Bob fail Additionally, Bob can cheat and look into how Alice did it
16. Asymmetric self-play for automatic goal discovery in robotic manipulation Bob learns to reach new goals without any finetuning (zero shot)
17. Never Give Up: Learning Directed Exploration Strategies Adrià Puigdomènech Badia, Pablo Sprechmann, et al., DeepMind [ICLR paper] How exploration usually works 1. A separate network tells you how novel your current state it. 2. Add “novelty” to your reward: 3. Push beta to 0 with time to start exploiting.
18. Classic approach for building a novelty network: learn to predict s_t+1 from s_t. Prediction error is novelty. But the agent stucks watching TV.. Never Give Up: Learning Directed Exploration Strategies
19. Never Give Up: Learning Directed Exploration Strategies [Yuri Burda, Harrison Edwards, et al., Exploration by Random Network Distillation, OpenAI & Univ. of Edinburgh, ICLR 2019]
20. Never Give Up: Learning Directed Exploration Strategies Insights 1. Novelty measures update slowly. Instead: normal lifelong novelty (which updates slowly) AND episodic memory (quick, but only within episode) 2. Only count states which are novel in a controllable way (e.g. novel TV picture doesn’t count). 3. Don’t stop exploring when you get better. Instead, have separate exploration and exploitation policies and run all in parallel.
21. Never Give Up: Learning Directed Exploration Strategies Insights 1. Novelty measures update slowly. Instead: normal lifelong novelty (which updates slowly) AND episodic memory (quick, but only within episode) 2. Only count states which are novel in a controllable way (e.g. novel TV picture doesn’t count). 3. Don’t stop exploring when you get better. Instead, have separate exploration and exploitation policies and run all in parallel.
22. Never Give Up: Learning Directed Exploration Strategies Insights 1. Novelty measures update slowly. Instead: normal lifelong novelty (which updates slowly) AND episodic memory (quick, but only within episode) 2. Only count states which are novel in a controllable way (e.g. novel TV picture doesn’t count). 3. Don’t stop exploring when you get better. Instead, have separate exploration and exploitation policies and run all in parallel.
23. Discovering Reinforcement Learning Algorithms Junhyuk Oh, Matteo Hessel, et al., DeepMind [Neurips paper] Use RL to discover an RL algorithm (e.g. something like PPO) automatically
24. Discovering Reinforcement Learning Algorithms Junhyuk Oh, Matteo Hessel, et al., DeepMind [Neurips paper] Use RL to discover an RL algorithm (e.g. something like PPO) automatically Recap of a simple RL algorithm 1. Initialize parameters of policy and of value function 2. While true
25. Discovering Reinforcement Learning Algorithms Junhyuk Oh, Matteo Hessel, et al., DeepMind [Neurips paper] Use RL to discover an RL algorithm (e.g. something like PPO) automatically Recap of a simple RL algorithm 1. Initialize parameters of policy and of value function 2. While true a. Run policy in the episode and collect a trajectory
26. Use RL to discover an RL algorithm (e.g. something like PPO) automatically Recap of a simple RL algorithm 1. Initialize parameters of policy and of value function 2. While true a. Run policy in the episode and collect a trajectory b. Update , where Discovering Reinforcement Learning Algorithms Junhyuk Oh, Matteo Hessel, et al., DeepMind [Neurips paper] Discounted return (“how good did the episode end”)
27. Use RL to discover an RL algorithm (e.g. something like PPO) automatically Recap of a simple RL algorithm 1. Initialize parameters of policy and of value function 2. While true a. Run policy in the episode and collect a trajectory b. Update , where Discovering Reinforcement Learning Algorithms Junhyuk Oh, Matteo Hessel, et al., DeepMind [Neurips paper] future is bad it’s obvious that the future will be bad don’t encourage / penalize taken action [image source] crashed
28. Use RL to discover an RL algorithm (e.g. something like PPO) automatically Recap of a simple RL algorithm 1. Initialize parameters of policy and of value function 2. While true a. Run policy in the episode and collect a trajectory b. Update , where Discovering Reinforcement Learning Algorithms Junhyuk Oh, Matteo Hessel, et al., DeepMind [Neurips paper] not crashed future is ok it’s obvious that the future will be bad the action avoided forecasted crash, awesome
29. Discovering Reinforcement Learning Algorithms Junhyuk Oh, Matteo Hessel, et al., DeepMind [Neurips paper] Use RL to discover an RL algorithm (e.g. something like PPO) automatically Recap of a simple RL algorithm 1. Initialize parameters of policy and of value function 2. While true a. Run policy in the episode and collect a trajectory b. Update , where c. Update by doing a gradient step on , where
30. Discovering Reinforcement Learning Algorithms Use RL to discover an RL algorithm (e.g. something like PPO) automatically Recap of a simple RL algorithm 1. Initialize parameters of policy and of value function 2. While true a. Run policy in the episode and collect a trajectory b. Update , where c. Update by doing a gradient step on , where Keep Learn
31. Discovering Reinforcement Learning Algorithms Use RL to discover an RL algorithm (e.g. something like PPO) automatically Recap of a simple RL algorithm 1. Initialize parameters of policy and of value function 2. While true a. Run policy in the episode and collect a trajectory b. Update , where c. Update by doing a gradient step on , where Keep Learn and
32. Discovering Reinforcement Learning Algorithms 1. Discovered learning algorithm generalizes from grid worlds to Atari 2. Came up with the idea of value function (and how to learn it) on its own Training on gridworld during meta-training Training on Atari during meta-test
33. Offline (batch) RL Can we use RL to learn to drive cars or control datacenter cooling systems?
34. Offline (batch) RL Can we use RL to learn to drive cars or control datacenter cooling systems? The old recipe is too dangerous and slow: 1) Try random stuff (“drive randomly”) 2) Reinforce the stuff that worked better (“try to not repeat actions that led to crashes”)
35. Offline (batch) RL Can we use RL to learn to drive cars or control datacenter cooling systems? Instead, collect human data and try to learn from it Environm ent State, Rewa rd Action Dataset Data collection States, rewards, actions
36. Offline (batch) RL Can we use RL to learn to drive cars or control datacenter cooling systems? Instead, collect human data and try to learn from it Environm ent State, Rewa rd Action Dataset Data collection States, rewards, actions Learning
37. Offline (batch) RL Can we use RL to learn to drive cars or control datacenter cooling systems? Instead, collect human data and try to learn from it Environm ent State, Rewa rd Action Environm ent State, Rewa rd Action Dataset Data collection States, rewards, actions Learning Testing
38. Offline RL pros and cons 1. Can solve harder tasks (safer; no need for collecting billions of frames) 2. (But of course there is no need to apply offline RL to e.g. Atari) 3. Cheaper research (no need for 10k CPUs per run, since data is prerecorded) 4. Existing datasets and code examples to get started 5. Some unique challenges compared to classic RL 6. More low hanging fruits :)
39. How to train offline RL agents? 1. Behaviour Cloning: just train a neural net to predict actions from states via supervised learning ◦ Only works when all data is of high quality, i.e. BC can’t do better than the data. 2. Just apply classic RL -- will that work? ◦ Not well, because of overestimating some actions: classic RL can randomly think that something not presented in the data (“drive into wall”) is worth a try. Most offline RL methods are thus focusing on avoiding the overestimation.
40. Critic regularized regression Ziyu Wang, Alex Novikov, Konrad Zołna, et al., DeepMind [code, Neurips paper]
41. Critic regularized regression
42. Critic regularized regression
43. Critic regularized regression
44. Critic regularized regression It doesn’t make sense to use BC on this non-expert (random) data
45. Critic regularized regression
46. Thanks! Imitate it now! Critic regularized regression
47. Critic regularized regression
48. Critic regularized regression
49. Any questions?