© 2021, Amazon Web Services, Inc. or its Affiliates. Sahika Genc, Principal Applied Scientist, Amazon June 2021 Introducing Amazon SageMaker Kubeflow Reinforcement Learning Pipelines for Robotics 

© 2021, Amazon Web Services, Inc. or its Affiliates. Contributors Max Kelsen • Nicholas Therkelsen-Terry • Leonard O'Sullivan • Matthew Rose Woodside Energy • Kyle Saltmarsh General Electric Aviation • John Karigiannis • Viktor Holovashchenko Amazon Web Services • Sahika Genc • Alex Chung • Ragha Prasad • Anna Luo 

© 2021, Amazon Web Services, Inc. or its Affiliates. Outline • Robotics Reinforcement Learning Workflow • Amazon SageMaker and Kubernetes/Kubeflow • Kubeflow Pipelines with AnyScale’s Ray • Customer use cases 1. General Electric Aviation – Manufacturing 2. Woodside Energy – Plant Operations 

© 2021, Amazon Web Services, Inc. or its Affiliates. Robotics Reinforcement Learning Workflow 

© 2021, Amazon Web Services, Inc. or its Affiliates. Reinforcement Learning for Robotics Reinforcement learning algorithms have the potential to enable robots to acquire complex behaviors adaptively in their environments. Motion planning Warehouse navigation Object manipulation 

© 2021, Amazon Web Services, Inc. or its Affiliates. Role of Simulation in Robotics • Easy to reset and explore various scenarios • Easy to alter the world, introduce obstacles, and tweak mechanics 

© 2021, Amazon Web Services, Inc. or its Affiliates. Successful Sim2Real with Thousands of Unique RL Agents AWS DeepRacer trains in simulation only with custom reward function. Thousands of RL agents have been successfully transferred at hundreds of in-person events around the world. Coming to NeurIPS 2021! 

© 2021, Amazon Web Services, Inc. or its Affiliates. SageMaker & Kubeflow 

© 2021, Amazon Web Services, Inc. or its Affiliates. The Machine Learning workflow is iterative and complex Collect and prepare training data Choose or bring your own ML algorithm Set up and manage environments for training Train, debug, and tune models Manage training runs Deploy model in production Monitor models Validate predictions Scale and manage the production environment 

© 2021, Amazon Web Services, Inc. or its Affiliates. Amazon SageMaker helps you build, train, and deploy models Collect and prepare training data Fully managed data processing jobs/ data labeling workflows Choose or bring your own ML algorithm Collaborative notebooks, built-in algorithms/models Set up and manage environments for training One-click training Train, debug, and tune models Debugging and optimization Manage training runs Visually track and compare experiments Deploy model in production One-click deployment and auto-scaling Monitor models Automatically spot concept drift Validate predictions Add human review of predictions Scale and manage the production environment Fully managed with auto-scaling for 75% less WEB-BASED IDE FOR ML AUTOMATICALLY BUILD AND TRAIN MODELS 

© 2021, Amazon Web Services, Inc. or its Affiliates. SageMaker with Kubernetes & Kubeflow Amazon SageMaker Operators for Kubernetes Components and Kubeflow Pipelines enable the use of fully managed SageMaker machine learning tools across the ML workflow natively from Kubernetes or Kubeflow. 

© 2021, Amazon Web Services, Inc. or its Affiliates. SageMaker Kubeflow Pipelines with Ray Cluster 

© 2021, Amazon Web Services, Inc. or its Affiliates. SageMaker RL helps you build, train, simulate, and deploy on robots Collect and prepare training data continuously Fully managed data processing jobs/ data labeling workflows Choose or bring your own RL algorithm Collaborative notebooks, built-in algorithms/models Set up and manage environments for training One-click training Train, debug, and tune models Debugging and optimization Manage training runs Visually track and compare experiments Deploy model on robots One-click deployment and auto-scaling Monitor models Automatically spot concept drift Validate predictions Add human review of predictions Scale and manage the production environment Fully managed with auto-scaling for 75% less WEB-BASED IDE FOR ML AUTOMATICALLY BUILD AND TRAIN MODELS Choose or bring your own simulation Cloud simulation environment, built-in algorithms/models Ray/RLLib 

© 2021, Amazon Web Services, Inc. or its Affiliates. Kubeflow Pipelines Orchestrates AWS RL Services Ray/RLLib 

© 2021, Amazon Web Services, Inc. or its Affiliates. Ray Cluster Utilizes External Cloud Simulation Servers VPC Amazon SageMaker Ray Cluster Trainer Policy Server Input Policy Rollout Data AWS RoboMaker Simulation Job AWS RoboMaker Simulation Job Rollout Fragments Policy Weights Amazon CloudWatch S3 Bucket Training Metrics Model Checkpoints Episode Visuals Console User AWS RoboMaker Simulation Job Policy Client Policy Episode Runner Agent Simulator Get_action() step() step() 

© 2021, Amazon Web Services, Inc. or its Affiliates. Faster training • Run a large number of simulations in parallel • Run at faster than real-time speed Policy V0 Episode 1 Episode 2 Episode 3 Episode N Epoch 1 Policy V1 Episode 1 Episode 2 Episode 3 Episode N Epoch 2 Policy Vi Policy V_N Episode 1 Episode 2 Episode 3 Episode N Epoch N Scaling Simulation Environments 

© 2021, Amazon Web Services, Inc. or its Affiliates. Benchmarking with Popular Gaming Environment NeurIPS 2020 Competition Track ProcGen BenchMark SageMaker Notebooks to measure sample efficiency & generalization in RL. Link to code repo: https://github.com/aws-samples/sagemaker-rl-procgen-ray 

© 2021, Amazon Web Services, Inc. or its Affiliates. Customer Uses Cases 

© 2021, Amazon Web Services, Inc. or its Affiliates. Use AWS RoboMaker with Amazon SageMaker to train, tune, and deploy reinforcement learning agents, and help improve operations and customer experience. Reinforcement Learning for Robotics Woodside Energy GE Aviation 

© 2021, Amazon Web Services, Inc. or its Affiliates. General Electric Aviation “We needed a framework that would allow us to evaluate different custom curricula in order to effectively train, test, tune, and deploy our RL agents efficiently to real robots at scale in the manufacturing environment. SageMaker RL with Ray/Rlib provides us with this ecosystem and we are excited to have our engineers and scientists using it for both experimentation, development and deployment.” says John Karigiannis, Senior Research Scientist at GE Aviation – Global Robotics & Automation Center 

© 2021, Amazon Web Services, Inc. or its Affiliates. Woodside Energy “Before we could do this effectively, we needed a framework that would allow us to train, test, tune, and deploy these models efficiently. Utilizing Kubeflow components and pipelines with SageMaker and RoboMaker provides us with this framework and we are excited to have our roboticists and data scientists focus their efforts and time on algorithms and implementation.” says Kyle Saltmarsh, Robotics Engineer at Woodside Energy. https://www.youtube.com/watch?v=r8BwBYAZsxQ 

© 2021, Amazon Web Services, Inc. or its Affiliates. To Get Started … • AWS Machine Learning Blog: https://aws.amazon.com/blogs/machine-learning/introducing-amazon-sagemak er-reinforcement-learning-components-for-open-source-kubeflow-pipelines/ • Code: https://github.com/MaxKelsen/kubeflow-pipelines-robomaker-examples 

© 2021, Amazon Web Services, Inc. or its Affiliates. Thank You Sahika Genc, Ph.D.