Kaiser * 1 Mohammad Babaeizadeh * 2 3 Piotr Miło´ s * 4 5 Bła˙ zej Osi´ nski * 4 5 3 Roy H Campbell 2 Konrad Czechowski 4 Dumitru Erhan 1 Chelsea Finn 1 Piotr Kozakowski 4 Sergey Levine 1 Ryan Sepassi 1 George Tucker 1 Henryk Michalewski 4 5 Abstract Model-free reinforcement learning (RL) can be used to learn effective policies for complex tasks, such as Atari games, even from image observa- tions. However, this typically requires very large amounts of interaction – substantially more, in fact, than a human would need to learn the same games. How can people learn so quickly? Part of the answer may be that people can learn how the game works and predict which actions will lead to desirable outcomes. In this paper, we explore how video prediction models can similarly enable agents to solve Atari games with orders of magni- tude fewer interactions than model-free methods. We describe Simulated Policy Learning (SimPLe), a complete model-based deep RL algorithm based on video prediction models and present a compar- ison of several model architectures, including a novel architecture that yields the best results in our setting. Our experiments evaluate SimPLe on a range of Atari games and achieve competitive results with only 100K interactions between the agent and the environment (400K frames), which corresponds to about two hours of real-time play. 1. Introduction Human players can learn to play Atari games in min- utes (Tsividis et al., 2017). However, our best model-free reinforcement learning algorithms require tens or hundreds of millions of time steps – the equivalent of several weeks of training in real time. How is it that humans can learn these games so much faster? Perhaps part of the puzzle is that processes that are represented in the game: we know that planes can fly, balls can roll, and bullets can destroy aliens. We can therefore predict the outcomes of our actions. In this paper, we explore how learned video models can enable learning in the Atari Learning Environment (ALE) bench- mark (Bellemare et al., 2015; Machado et al., 2017) with a budget restricted to 100K time steps – roughly to two hours of a play time. Although prior works have proposed training predictive models for next-frame, future-frame, as well as combined future-frame and reward predictions in Atari games (Oh et al., 2015; Chiappa et al., 2017; Leibfried et al., 2016), no prior work has successfully demonstrated model-based control via such predictive models that achieve results that are competitive with model-free RL. Indeed, in a recent sur- vey by Machado et al. this was formulated as the following challenge: “So far, there has been no clear demonstration of successful planning with a learned model in the ALE” (Section 7.2 in Machado et al. (2017)). Using models of environments, or informally giving the agent ability to predict its future, has a fundamental appeal for reinforcement learning. The spectrum of possible appli- cations is vast, including learning policies from the model (Watter et al., 2015; Finn et al., 2016; Finn & Levine, 2016; Ebert et al., 2017; Hafner et al., 2018; Piergiovanni et al., 2018; Rybkin et al., 2018; Sutton & Barto, 2017, Chapter 8), capturing important details of the scene (Ha & Schmidhuber, 2018), encouraging exploration (Oh et al., 2015), creating intrinsic motivation (Schmidhuber, 2010) or counterfactual reasoning (Buesing et al., 2018). One of the exciting bene- fits of model-based learning is the promise to substantially improve sample efficiency of deep reinforcement learning (see Chapter 8 in (Sutton & Barto, 2017)). arXiv:1903.00374v1 [cs.LG] 1 Mar 2019 Trust Region Policy Optimization John Schulman
[email protected] Sergey Levine
[email protected] Philipp Moritz
[email protected] Michael Jordan
[email protected] Pieter Abbeel
[email protected] University of California, Berkeley, Department of Electrical Engineering and Computer Sciences Abstract We describe an iterative procedure for optimizing policies, with guaranteed monotonic improve- ment. By making several approximations to the theoretically-justified procedure, we develop a practical algorithm, called Trust Region Policy Optimization (TRPO). This algorithm is similar to natural policy gradient methods and is effec- tive for optimizing large nonlinear policies such as neural networks. Our experiments demon- strate its robust performance on a wide variety of tasks: learning simulated robotic swimming, hopping, and walking gaits; and playing Atari games using images of the screen as input. De- spite its approximations that deviate from the theory, TRPO tends to give monotonic improve- ment, with little tuning of hyperparameters. Tetris is a classic benchmark problem for approximate dy- namic programming (ADP) methods, stochastic optimiza- tion methods are difficult to beat on this task (Gabillon et al., 2013). For continuous control problems, methods like CMA have been successful at learning control poli- cies for challenging tasks like locomotion when provided with hand-engineered policy classes with low-dimensional parameterizations (Wampler & Popovi´ c, 2009). The in- ability of ADP and gradient-based methods to consistently beat gradient-free random search is unsatisfying, since gradient-based optimization algorithms enjoy much better sample complexity guarantees than gradient-free methods (Nemirovski, 2005). Continuous gradient-based optimiza- tion has been very successful at learning function approxi- mators for supervised learning tasks with huge numbers of parameters, and extending their success to reinforcement learning would allow for efficient training of complex and powerful policies. 477v5 [cs.LG] 20 Apr 2017 Proximal Policy Optimization Algorithms John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov OpenAI {joschu, filip, prafulla, alec, oleg}@openai.com Abstract We propose a new family of policy gradient methods for reinforcement learning, which al- ternate between sampling data through interaction with the environment, and optimizing a “surrogate” objective function using stochastic gradient ascent. Whereas standard policy gra- Ϟσϧ ࣮࣭ΤϛϡϨʔλ Λͬͯɺ ΤʔδΣϯτΛֶशͤ͞Δจ
ํࡦͷֶशʹ110Λ͏ ࠓͷϝΠϯςʔϚͷϋζͩͬͨɾɾ 110ͷߟ͑ํͷϕʔεʹͳ͍ͬͯΔจ
ཧղ͠ͳ͍ͱ110ͷྑ͕͞Θ͔Βͳ͍
ࠓͷཪςʔϚ 5310ͷརΛ׆͔͠ͳ͕Βɺ ओʹ࣮໘Ͱͷվળ͕ͳ͞Εͨख๏
࠷ۙ5310ʹมΘͬͯΑ͘ΘΕ͍ͯΔ
yuishihara
kei18
gpeyre
mkimura
tsantalis
moda0
xtalent
nehi0615
nttcom
kg_wakate
tomoyayama
mza
palkan
paigeccino
bkeepers
marcduiker
malarkey
sstephenson
matthewcrist
eileencodes
mongodb
addyosmani
geeforr
![Trust Region Policy Optimization John Schulman [email protected] Sergey](https://files.speakerdeck.com/presentations/b65b9ca77afe48c8b61b47523aff98c0/slide_10.jpg){kind=link}
![ํࡦޯ๏ͷ෮श ڧԽֶशͷత ํࡦޯ๏ͷͶΒ͍ arg max ⇡ E[Gt ]](https://files.speakerdeck.com/presentations/b65b9ca77afe48c8b61b47523aff98c0/slide_11.jpg){kind=link}
![7BOJMMBQPMJDZHSBEJFOUͷ https://wiseodd.github.io/techblog/2018/03/14/natural-gradient/ &VDMJEڑͰݟͯಉ͡ൣғ͔ΒબͿ r✓ E ⌧⇠⇡✓ [G0 (⌧)]](https://files.speakerdeck.com/presentations/b65b9ca77afe48c8b61b47523aff98c0/slide_12.jpg){kind=link}
