Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introducing Amazon SageMaker Kubeflow Reinforcement Learning Pipelines for Robotics (Sahika Genc, Amazon)

Introducing Amazon SageMaker Kubeflow Reinforcement Learning Pipelines for Robotics (Sahika Genc, Amazon)

Robots require the integration of technologies such as image recognition, sensing, artificial intelligence, machine learning (ML), and reinforcement learning (RL) in ways that are new to the field of robotics. Orchestrating robotics operations to train, simulate, and deploy RL applications is difficult and time-consuming. Now, with AnyScale’s Ray and SageMaker RL components and pipelines, it’s faster to experiment and manage robotics RL workflows from perception to controls and optimization, and create end-to-end solutions without having to rebuild each time. In this talk, we will talk about two use cases utilizing AnyScale’s Ray with SageMaker RL Kubeflow components where 1) Woodside Energy uses AnyScale’s Ray with an external cloud simulator, AWS RoboMaker, to start exploring using machine learning methods for robotics manipulation for power plant operations, and 2) General Electric Aviation uses AnyScale’s Ray with an open-source simulator, PyBullet, to improve manufacturing plant operations.

Af07bbf978a0989644b039ae6b8904a5?s=128

Anyscale
PRO

July 21, 2021
Tweet

Transcript

  1. © 2021, Amazon Web Services, Inc. or its Affiliates. Sahika

    Genc, Principal Applied Scientist, Amazon June 2021 Introducing Amazon SageMaker Kubeflow Reinforcement Learning Pipelines for Robotics
  2. © 2021, Amazon Web Services, Inc. or its Affiliates. Contributors

    Max Kelsen • Nicholas Therkelsen-Terry • Leonard O'Sullivan • Matthew Rose Woodside Energy • Kyle Saltmarsh General Electric Aviation • John Karigiannis • Viktor Holovashchenko Amazon Web Services • Sahika Genc • Alex Chung • Ragha Prasad • Anna Luo
  3. © 2021, Amazon Web Services, Inc. or its Affiliates. Outline

    • Robotics Reinforcement Learning Workflow • Amazon SageMaker and Kubernetes/Kubeflow • Kubeflow Pipelines with AnyScale’s Ray • Customer use cases 1. General Electric Aviation – Manufacturing 2. Woodside Energy – Plant Operations
  4. © 2021, Amazon Web Services, Inc. or its Affiliates. Robotics

    Reinforcement Learning Workflow
  5. © 2021, Amazon Web Services, Inc. or its Affiliates. Reinforcement

    Learning for Robotics Reinforcement learning algorithms have the potential to enable robots to acquire complex behaviors adaptively in their environments. Motion planning Warehouse navigation Object manipulation
  6. © 2021, Amazon Web Services, Inc. or its Affiliates. Role

    of Simulation in Robotics • Easy to reset and explore various scenarios • Easy to alter the world, introduce obstacles, and tweak mechanics
  7. © 2021, Amazon Web Services, Inc. or its Affiliates. Successful

    Sim2Real with Thousands of Unique RL Agents AWS DeepRacer trains in simulation only with custom reward function. Thousands of RL agents have been successfully transferred at hundreds of in-person events around the world. Coming to NeurIPS 2021!
  8. © 2021, Amazon Web Services, Inc. or its Affiliates. SageMaker

    & Kubeflow
  9. © 2021, Amazon Web Services, Inc. or its Affiliates. The

    Machine Learning workflow is iterative and complex Collect and prepare training data Choose or bring your own ML algorithm Set up and manage environments for training Train, debug, and tune models Manage training runs Deploy model in production Monitor models Validate predictions Scale and manage the production environment
  10. © 2021, Amazon Web Services, Inc. or its Affiliates. Amazon

    SageMaker helps you build, train, and deploy models Collect and prepare training data Fully managed data processing jobs/ data labeling workflows Choose or bring your own ML algorithm Collaborative notebooks, built-in algorithms/models Set up and manage environments for training One-click training Train, debug, and tune models Debugging and optimization Manage training runs Visually track and compare experiments Deploy model in production One-click deployment and auto-scaling Monitor models Automatically spot concept drift Validate predictions Add human review of predictions Scale and manage the production environment Fully managed with auto-scaling for 75% less WEB-BASED IDE FOR ML AUTOMATICALLY BUILD AND TRAIN MODELS
  11. © 2021, Amazon Web Services, Inc. or its Affiliates. SageMaker

    with Kubernetes & Kubeflow Amazon SageMaker Operators for Kubernetes Components and Kubeflow Pipelines enable the use of fully managed SageMaker machine learning tools across the ML workflow natively from Kubernetes or Kubeflow.
  12. © 2021, Amazon Web Services, Inc. or its Affiliates. SageMaker

    Kubeflow Pipelines with Ray Cluster
  13. © 2021, Amazon Web Services, Inc. or its Affiliates. SageMaker

    RL helps you build, train, simulate, and deploy on robots Collect and prepare training data continuously Fully managed data processing jobs/ data labeling workflows Choose or bring your own RL algorithm Collaborative notebooks, built-in algorithms/models Set up and manage environments for training One-click training Train, debug, and tune models Debugging and optimization Manage training runs Visually track and compare experiments Deploy model on robots One-click deployment and auto-scaling Monitor models Automatically spot concept drift Validate predictions Add human review of predictions Scale and manage the production environment Fully managed with auto-scaling for 75% less WEB-BASED IDE FOR ML AUTOMATICALLY BUILD AND TRAIN MODELS Choose or bring your own simulation Cloud simulation environment, built-in algorithms/models Ray/RLLib
  14. © 2021, Amazon Web Services, Inc. or its Affiliates. Kubeflow

    Pipelines Orchestrates AWS RL Services Ray/RLLib
  15. © 2021, Amazon Web Services, Inc. or its Affiliates. Ray

    Cluster Utilizes External Cloud Simulation Servers VPC Amazon SageMaker Ray Cluster Trainer Policy Server Input Policy Rollout Data AWS RoboMaker Simulation Job AWS RoboMaker Simulation Job Rollout Fragments Policy Weights Amazon CloudWatch S3 Bucket Training Metrics Model Checkpoints Episode Visuals Console User AWS RoboMaker Simulation Job Policy Client Policy Episode Runner Agent Simulator Get_action() step() step()
  16. © 2021, Amazon Web Services, Inc. or its Affiliates. Faster

    training • Run a large number of simulations in parallel • Run at faster than real-time speed Policy V0 Episode 1 Episode 2 Episode 3 Episode N Epoch 1 Policy V1 Episode 1 Episode 2 Episode 3 Episode N Epoch 2 Policy Vi Policy V_N Episode 1 Episode 2 Episode 3 Episode N Epoch N Scaling Simulation Environments
  17. © 2021, Amazon Web Services, Inc. or its Affiliates. Benchmarking

    with Popular Gaming Environment NeurIPS 2020 Competition Track ProcGen BenchMark SageMaker Notebooks to measure sample efficiency & generalization in RL. Link to code repo: https://github.com/aws-samples/sagemaker-rl-procgen-ray
  18. © 2021, Amazon Web Services, Inc. or its Affiliates. Customer

    Uses Cases
  19. © 2021, Amazon Web Services, Inc. or its Affiliates. Use

    AWS RoboMaker with Amazon SageMaker to train, tune, and deploy reinforcement learning agents, and help improve operations and customer experience. Reinforcement Learning for Robotics Woodside Energy GE Aviation
  20. © 2021, Amazon Web Services, Inc. or its Affiliates. General

    Electric Aviation “We needed a framework that would allow us to evaluate different custom curricula in order to effectively train, test, tune, and deploy our RL agents efficiently to real robots at scale in the manufacturing environment. SageMaker RL with Ray/Rlib provides us with this ecosystem and we are excited to have our engineers and scientists using it for both experimentation, development and deployment.” says John Karigiannis, Senior Research Scientist at GE Aviation – Global Robotics & Automation Center
  21. © 2021, Amazon Web Services, Inc. or its Affiliates. Woodside

    Energy “Before we could do this effectively, we needed a framework that would allow us to train, test, tune, and deploy these models efficiently. Utilizing Kubeflow components and pipelines with SageMaker and RoboMaker provides us with this framework and we are excited to have our roboticists and data scientists focus their efforts and time on algorithms and implementation.” says Kyle Saltmarsh, Robotics Engineer at Woodside Energy. https://www.youtube.com/watch?v=r8BwBYAZsxQ
  22. © 2021, Amazon Web Services, Inc. or its Affiliates. To

    Get Started … • AWS Machine Learning Blog: https://aws.amazon.com/blogs/machine-learning/introducing-amazon-sagemak er-reinforcement-learning-components-for-open-source-kubeflow-pipelines/ • Code: https://github.com/MaxKelsen/kubeflow-pipelines-robomaker-examples
  23. © 2021, Amazon Web Services, Inc. or its Affiliates. Thank

    You Sahika Genc, Ph.D.