Introducing Amazon SageMaker Kubeflow Reinforcement Learning Pipelines for Robotics (Sahika Genc, Amazon)

Robots require the integration of technologies such as image recognition, sensing, artificial intelligence, machine learning (ML), and reinforcement learning (RL) in ways that are new to the field of robotics. Orchestrating robotics operations to train, simulate, and deploy RL applications is difficult and time-consuming. Now, with AnyScale’s Ray and SageMaker RL components and pipelines, it’s faster to experiment and manage robotics RL workflows from perception to controls and optimization, and create end-to-end solutions without having to rebuild each time. In this talk, we will talk about two use cases utilizing AnyScale’s Ray with SageMaker RL Kubeflow components where 1) Woodside Energy uses AnyScale’s Ray with an external cloud simulator, AWS RoboMaker, to start exploring using machine learning methods for robotics manipulation for power plant operations, and 2) General Electric Aviation uses AnyScale’s Ray with an open-source simulator, PyBullet, to improve manufacturing plant operations.

Transcript

1. © 2021, Amazon Web Services, Inc. or its Affiliates.
   Sahika Genc, Principal Applied Scientist, Amazon
   June 2021
   Introducing Amazon SageMaker
   Kubeflow Reinforcement Learning
   Pipelines for Robotics
   

   View Slide

2. © 2021, Amazon Web Services, Inc. or its Affiliates.
   Contributors
   Max Kelsen
   • Nicholas Therkelsen-Terry
   • Leonard O'Sullivan
   • Matthew Rose
   Woodside Energy
   • Kyle Saltmarsh
   General Electric Aviation
   • John Karigiannis
   • Viktor Holovashchenko
   Amazon Web Services
   • Sahika Genc
   • Alex Chung
   • Ragha Prasad
   • Anna Luo
   

   View Slide

3. © 2021, Amazon Web Services, Inc. or its Affiliates.
   Outline
   • Robotics Reinforcement Learning Workflow
   • Amazon SageMaker and Kubernetes/Kubeflow
   • Kubeflow Pipelines with AnyScale’s Ray
   • Customer use cases
   1. General Electric Aviation – Manufacturing
   2. Woodside Energy – Plant Operations
   

   View Slide

4. © 2021, Amazon Web Services, Inc. or its Affiliates.
   Robotics Reinforcement
   Learning Workflow
   

   View Slide

5. © 2021, Amazon Web Services, Inc. or its Affiliates.
   Reinforcement Learning for Robotics
   Reinforcement learning algorithms have the potential to enable
   robots to acquire complex behaviors adaptively in their environments.
   Motion planning Warehouse navigation Object manipulation
   

   View Slide

6. © 2021, Amazon Web Services, Inc. or its Affiliates.
   Role of Simulation in Robotics
   • Easy to reset and explore various scenarios
   • Easy to alter the world, introduce obstacles, and tweak mechanics
   

   View Slide

7. © 2021, Amazon Web Services, Inc. or its Affiliates.
   Successful Sim2Real with Thousands of Unique RL
   Agents
   AWS DeepRacer trains in simulation only with custom reward function.
   Thousands of RL agents have been successfully transferred at hundreds
   of in-person events around the world. Coming to NeurIPS 2021!
   

   View Slide

8. © 2021, Amazon Web Services, Inc. or its Affiliates.
   SageMaker & Kubeflow
   

   View Slide

9. © 2021, Amazon Web Services, Inc. or its Affiliates.
   The Machine Learning workflow is iterative and complex
   Collect and prepare
   training data
   Choose or bring your
   own ML algorithm
   Set up and manage
   environments for training
   Train, debug, and
   tune models
   Manage
   training runs
   Deploy model
   in production
   Monitor
   models
   Validate
   predictions
   Scale and manage the
   production environment
   

   View Slide

10. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Amazon SageMaker helps you build, train, and deploy models
    Collect and prepare
    training data
    Fully managed data processing jobs/
    data labeling workflows
    Choose or bring your
    own ML algorithm
    Collaborative notebooks,
    built-in algorithms/models
    Set up and manage
    environments for training
    One-click training
    Train, debug, and
    tune models
    Debugging and
    optimization
    Manage
    training runs
    Visually track and
    compare experiments
    Deploy model
    in production
    One-click deployment
    and auto-scaling
    Monitor
    models
    Automatically
    spot concept drift
    Validate
    predictions
    Add human review
    of predictions
    Scale and manage the
    production environment
    Fully managed with
    auto-scaling for 75% less
    WEB-BASED IDE FOR ML AUTOMATICALLY BUILD AND TRAIN MODELS
    

    View Slide

11. © 2021, Amazon Web Services, Inc. or its Affiliates.
    SageMaker with Kubernetes & Kubeflow
    Amazon SageMaker Operators for Kubernetes Components and Kubeflow
    Pipelines enable the use of fully managed SageMaker machine learning tools
    across the ML workflow natively from Kubernetes or Kubeflow.
    

    View Slide

12. © 2021, Amazon Web Services, Inc. or its Affiliates.
    SageMaker Kubeflow
    Pipelines with Ray
    Cluster
    

    View Slide

13. © 2021, Amazon Web Services, Inc. or its Affiliates.
    SageMaker RL helps you build, train, simulate, and deploy on
    robots
    Collect and prepare
    training data continuously
    Fully managed data processing jobs/
    data labeling workflows
    Choose or bring your
    own RL algorithm
    Collaborative notebooks,
    built-in algorithms/models
    Set up and manage
    environments for training
    One-click training
    Train, debug, and
    tune models
    Debugging and
    optimization
    Manage
    training runs
    Visually track and
    compare experiments
    Deploy model
    on robots
    One-click deployment
    and auto-scaling
    Monitor
    models
    Automatically
    spot concept drift
    Validate
    predictions
    Add human review
    of predictions
    Scale and manage the
    production environment
    Fully managed with
    auto-scaling for 75% less
    WEB-BASED IDE FOR ML AUTOMATICALLY BUILD AND TRAIN MODELS
    Choose or bring your
    own simulation
    Cloud simulation environment,
    built-in algorithms/models
    Ray/RLLib
    

    View Slide

14. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Kubeflow Pipelines Orchestrates AWS RL Services
    Ray/RLLib
    

    View Slide

15. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Ray Cluster Utilizes External Cloud Simulation Servers
    VPC
    Amazon SageMaker
    Ray Cluster
    Trainer
    Policy Server
    Input
    Policy
    Rollout Data
    AWS RoboMaker Simulation Job
    AWS RoboMaker Simulation Job
    Rollout
    Fragments
    Policy
    Weights
    Amazon CloudWatch
    S3 Bucket
    Training Metrics
    Model Checkpoints
    Episode Visuals
    Console User
    AWS RoboMaker Simulation Job
    Policy Client
    Policy
    Episode Runner
    Agent
    Simulator
    Get_action()
    step()
    step()
    

    View Slide

16. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Faster training
    • Run a large number of simulations in parallel
    • Run at faster than real-time speed
    Policy V0
    Episode 1
    Episode 2
    Episode 3
    Episode N
    Epoch 1
    Policy V1
    Episode 1
    Episode 2
    Episode 3
    Episode N
    Epoch 2
    Policy Vi Policy V_N
    Episode 1
    Episode 2
    Episode 3
    Episode N
    Epoch N
    Scaling Simulation Environments
    

    View Slide

17. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Benchmarking with Popular Gaming Environment
    NeurIPS 2020 Competition Track ProcGen BenchMark SageMaker Notebooks to
    measure sample efficiency & generalization in RL.
    Link to code repo: https://github.com/aws-samples/sagemaker-rl-procgen-ray
    

    View Slide

18. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Customer Uses Cases
    

    View Slide

19. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Use AWS RoboMaker with Amazon SageMaker to train, tune, and deploy reinforcement learning
    agents, and help improve operations and customer experience.
    Reinforcement Learning for Robotics
    Woodside Energy
    GE Aviation
    

    View Slide

20. © 2021, Amazon Web Services, Inc. or its Affiliates.
    General Electric Aviation
    “We needed a framework that would allow
    us to evaluate different custom curricula in
    order to effectively train, test, tune, and
    deploy our RL agents efficiently to real
    robots at scale in the manufacturing
    environment. SageMaker RL with Ray/Rlib
    provides us with this ecosystem and we are
    excited to have our engineers and scientists
    using it for both experimentation,
    development and deployment.” says John
    Karigiannis, Senior Research Scientist at
    GE Aviation – Global Robotics & Automation
    Center
    

    View Slide

21. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Woodside Energy
    “Before we could do this effectively, we needed a framework that would allow us to train, test, tune,
    and deploy these models efficiently. Utilizing Kubeflow components and pipelines with SageMaker
    and RoboMaker provides us with this framework and we are excited to have our roboticists and
    data scientists focus their efforts and time on algorithms and implementation.” says Kyle
    Saltmarsh, Robotics Engineer at Woodside Energy.
    https://www.youtube.com/watch?v=r8BwBYAZsxQ
    

    View Slide

22. © 2021, Amazon Web Services, Inc. or its Affiliates.
    To Get Started …
    • AWS Machine Learning Blog:
    https://aws.amazon.com/blogs/machine-learning/introducing-amazon-sagemak
    er-reinforcement-learning-components-for-open-source-kubeflow-pipelines/
    • Code: https://github.com/MaxKelsen/kubeflow-pipelines-robomaker-examples
    

    View Slide

23. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Thank You
    Sahika Genc, Ph.D.
    

    View Slide