Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introducing Amazon SageMaker Kubeflow Reinforcement Learning Pipelines for Robotics (Sahika Genc, Amazon)

Introducing Amazon SageMaker Kubeflow Reinforcement Learning Pipelines for Robotics (Sahika Genc, Amazon)

Robots require the integration of technologies such as image recognition, sensing, artificial intelligence, machine learning (ML), and reinforcement learning (RL) in ways that are new to the field of robotics. Orchestrating robotics operations to train, simulate, and deploy RL applications is difficult and time-consuming. Now, with AnyScale’s Ray and SageMaker RL components and pipelines, it’s faster to experiment and manage robotics RL workflows from perception to controls and optimization, and create end-to-end solutions without having to rebuild each time. In this talk, we will talk about two use cases utilizing AnyScale’s Ray with SageMaker RL Kubeflow components where 1) Woodside Energy uses AnyScale’s Ray with an external cloud simulator, AWS RoboMaker, to start exploring using machine learning methods for robotics manipulation for power plant operations, and 2) General Electric Aviation uses AnyScale’s Ray with an open-source simulator, PyBullet, to improve manufacturing plant operations.

Anyscale

July 21, 2021
Tweet

More Decks by Anyscale

Other Decks in Technology

Transcript

  1. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Sahika Genc, Principal Applied Scientist, Amazon
    June 2021
    Introducing Amazon SageMaker
    Kubeflow Reinforcement Learning
    Pipelines for Robotics

    View full-size slide

  2. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Contributors
    Max Kelsen
    • Nicholas Therkelsen-Terry
    • Leonard O'Sullivan
    • Matthew Rose
    Woodside Energy
    • Kyle Saltmarsh
    General Electric Aviation
    • John Karigiannis
    • Viktor Holovashchenko
    Amazon Web Services
    • Sahika Genc
    • Alex Chung
    • Ragha Prasad
    • Anna Luo

    View full-size slide

  3. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Outline
    • Robotics Reinforcement Learning Workflow
    • Amazon SageMaker and Kubernetes/Kubeflow
    • Kubeflow Pipelines with AnyScale’s Ray
    • Customer use cases
    1. General Electric Aviation – Manufacturing
    2. Woodside Energy – Plant Operations

    View full-size slide

  4. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Robotics Reinforcement
    Learning Workflow

    View full-size slide

  5. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Reinforcement Learning for Robotics
    Reinforcement learning algorithms have the potential to enable
    robots to acquire complex behaviors adaptively in their environments.
    Motion planning Warehouse navigation Object manipulation

    View full-size slide

  6. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Role of Simulation in Robotics
    • Easy to reset and explore various scenarios
    • Easy to alter the world, introduce obstacles, and tweak mechanics

    View full-size slide

  7. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Successful Sim2Real with Thousands of Unique RL
    Agents
    AWS DeepRacer trains in simulation only with custom reward function.
    Thousands of RL agents have been successfully transferred at hundreds
    of in-person events around the world. Coming to NeurIPS 2021!

    View full-size slide

  8. © 2021, Amazon Web Services, Inc. or its Affiliates.
    SageMaker & Kubeflow

    View full-size slide

  9. © 2021, Amazon Web Services, Inc. or its Affiliates.
    The Machine Learning workflow is iterative and complex
    Collect and prepare
    training data
    Choose or bring your
    own ML algorithm
    Set up and manage
    environments for training
    Train, debug, and
    tune models
    Manage
    training runs
    Deploy model
    in production
    Monitor
    models
    Validate
    predictions
    Scale and manage the
    production environment

    View full-size slide

  10. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Amazon SageMaker helps you build, train, and deploy models
    Collect and prepare
    training data
    Fully managed data processing jobs/
    data labeling workflows
    Choose or bring your
    own ML algorithm
    Collaborative notebooks,
    built-in algorithms/models
    Set up and manage
    environments for training
    One-click training
    Train, debug, and
    tune models
    Debugging and
    optimization
    Manage
    training runs
    Visually track and
    compare experiments
    Deploy model
    in production
    One-click deployment
    and auto-scaling
    Monitor
    models
    Automatically
    spot concept drift
    Validate
    predictions
    Add human review
    of predictions
    Scale and manage the
    production environment
    Fully managed with
    auto-scaling for 75% less
    WEB-BASED IDE FOR ML AUTOMATICALLY BUILD AND TRAIN MODELS

    View full-size slide

  11. © 2021, Amazon Web Services, Inc. or its Affiliates.
    SageMaker with Kubernetes & Kubeflow
    Amazon SageMaker Operators for Kubernetes Components and Kubeflow
    Pipelines enable the use of fully managed SageMaker machine learning tools
    across the ML workflow natively from Kubernetes or Kubeflow.

    View full-size slide

  12. © 2021, Amazon Web Services, Inc. or its Affiliates.
    SageMaker Kubeflow
    Pipelines with Ray
    Cluster

    View full-size slide

  13. © 2021, Amazon Web Services, Inc. or its Affiliates.
    SageMaker RL helps you build, train, simulate, and deploy on
    robots
    Collect and prepare
    training data continuously
    Fully managed data processing jobs/
    data labeling workflows
    Choose or bring your
    own RL algorithm
    Collaborative notebooks,
    built-in algorithms/models
    Set up and manage
    environments for training
    One-click training
    Train, debug, and
    tune models
    Debugging and
    optimization
    Manage
    training runs
    Visually track and
    compare experiments
    Deploy model
    on robots
    One-click deployment
    and auto-scaling
    Monitor
    models
    Automatically
    spot concept drift
    Validate
    predictions
    Add human review
    of predictions
    Scale and manage the
    production environment
    Fully managed with
    auto-scaling for 75% less
    WEB-BASED IDE FOR ML AUTOMATICALLY BUILD AND TRAIN MODELS
    Choose or bring your
    own simulation
    Cloud simulation environment,
    built-in algorithms/models
    Ray/RLLib

    View full-size slide

  14. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Kubeflow Pipelines Orchestrates AWS RL Services
    Ray/RLLib

    View full-size slide

  15. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Ray Cluster Utilizes External Cloud Simulation Servers
    VPC
    Amazon SageMaker
    Ray Cluster
    Trainer
    Policy Server
    Input
    Policy
    Rollout Data
    AWS RoboMaker Simulation Job
    AWS RoboMaker Simulation Job
    Rollout
    Fragments
    Policy
    Weights
    Amazon CloudWatch
    S3 Bucket
    Training Metrics
    Model Checkpoints
    Episode Visuals
    Console User
    AWS RoboMaker Simulation Job
    Policy Client
    Policy
    Episode Runner
    Agent
    Simulator
    Get_action()
    step()
    step()

    View full-size slide

  16. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Faster training
    • Run a large number of simulations in parallel
    • Run at faster than real-time speed
    Policy V0
    Episode 1
    Episode 2
    Episode 3
    Episode N
    Epoch 1
    Policy V1
    Episode 1
    Episode 2
    Episode 3
    Episode N
    Epoch 2
    Policy Vi Policy V_N
    Episode 1
    Episode 2
    Episode 3
    Episode N
    Epoch N
    Scaling Simulation Environments

    View full-size slide

  17. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Benchmarking with Popular Gaming Environment
    NeurIPS 2020 Competition Track ProcGen BenchMark SageMaker Notebooks to
    measure sample efficiency & generalization in RL.
    Link to code repo: https://github.com/aws-samples/sagemaker-rl-procgen-ray

    View full-size slide

  18. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Customer Uses Cases

    View full-size slide

  19. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Use AWS RoboMaker with Amazon SageMaker to train, tune, and deploy reinforcement learning
    agents, and help improve operations and customer experience.
    Reinforcement Learning for Robotics
    Woodside Energy
    GE Aviation

    View full-size slide

  20. © 2021, Amazon Web Services, Inc. or its Affiliates.
    General Electric Aviation
    “We needed a framework that would allow
    us to evaluate different custom curricula in
    order to effectively train, test, tune, and
    deploy our RL agents efficiently to real
    robots at scale in the manufacturing
    environment. SageMaker RL with Ray/Rlib
    provides us with this ecosystem and we are
    excited to have our engineers and scientists
    using it for both experimentation,
    development and deployment.” says John
    Karigiannis, Senior Research Scientist at
    GE Aviation – Global Robotics & Automation
    Center

    View full-size slide

  21. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Woodside Energy
    “Before we could do this effectively, we needed a framework that would allow us to train, test, tune,
    and deploy these models efficiently. Utilizing Kubeflow components and pipelines with SageMaker
    and RoboMaker provides us with this framework and we are excited to have our roboticists and
    data scientists focus their efforts and time on algorithms and implementation.” says Kyle
    Saltmarsh, Robotics Engineer at Woodside Energy.
    https://www.youtube.com/watch?v=r8BwBYAZsxQ

    View full-size slide

  22. © 2021, Amazon Web Services, Inc. or its Affiliates.
    To Get Started …
    • AWS Machine Learning Blog:
    https://aws.amazon.com/blogs/machine-learning/introducing-amazon-sagemak
    er-reinforcement-learning-components-for-open-source-kubeflow-pipelines/
    • Code: https://github.com/MaxKelsen/kubeflow-pipelines-robomaker-examples

    View full-size slide

  23. © 2021, Amazon Web Services, Inc. or its Affiliates.
    Thank You
    Sahika Genc, Ph.D.

    View full-size slide