Powering Open Data Hub with Ray (Erik Erlandson, Red Hat, AI Center of Excellence)

Ray is quickly gaining momentum as a distributed computing platform that combines a powerful parallel compute model with a cloud native serverless-style scaling model. Open Data Hub (ODH) is a flexible and customizable federation of open source data science tools that is a great fit for taking advantage of Ray compute clusters.

In this talk, Erik will explain how to integrate Ray with Open Data Hub, by configuring ODH profiles that deploy on-demand Ray clusters for Jupyter notebooks. He’ll demonstrate Ray in action as a compute resource for ODH, and explore the potential use cases opened up by self-service notebooks backed by Ray. Along the way he’ll also discuss the logistics of adapting Ray to OpenShift’s security features.

Attendees will learn how Ray integrates with Open Data Hub’s architecture, and how they can power ODH with Ray to solve distributed computing problems in the popular Jupyter environment.

Transcript

1. Powering ODH
   With Ray
   Erik Erlandson, Red Hat, Inc.
   eje@redhat.com
   @ManyAngled
   

   View Slide

2. Or...
   Erik Erlandson, Red Hat, Inc.
   eje@redhat.com
   @ManyAngled
   

   View Slide

3. Jupyter & Ray
   In The Cloud
   Erik Erlandson, Red Hat, Inc.
   eje@redhat.com
   @ManyAngled
   

   View Slide

4. Landscape
   Motivations
   Open Data Hub and Jupyter in Context
   Ray on ODH
   Demo
   Community Collaborations
   

   View Slide

5. Native Ray Libraries
   ● Tune: Scalable Hyperparameter Tuning
   ● RLlib: Scalable Reinforcement Learning
   ● RaySGD: Distributed Training Wrappers
   ● Ray Serve: Scalable and Programmable Serving
   

   View Slide

6. Ray Community Integrations
   ● XGBoost
   ● Dask
   ● Horovod
   ● sklearn
   ● Spacy
   ● huggingface
   https://docs.ray.io/en/master/ray-libraries.html
   

   View Slide

7. Ray Community Integrations
   ● XGBoost
   ● Dask
   ● Horovod
   ● sklearn
   ● Spacy
   ● huggingface
   https://docs.ray.io/en/master/ray-libraries.html
   

   View Slide

8. Literate And Interactive Ray...
   https://docs.ray.io/en/master/ray-libraries.html
   

   View Slide

9. Hosted In The Cloud
   https://docs.ray.io/en/master/ray-libraries.html
   

   View Slide

10. Jupyter + Ray 1.X
    Jupyter +
    Ray Head Pod
    Ray Worker Pods
    

    View Slide

11. Jupyter + Ray 1.X
    Jupyter +
    Ray Head Pod
    Ray Worker Pods
    

    View Slide

12. Jupyter + Ray 2.0
    Ray Worker Pods
    Ray Head Pod
    Jupyter Pod
    

    View Slide

13. Jupyter ...
    

    View Slide

14. Jupyter via Open Data Hub
    

    View Slide

15. Open Data Hub Is ...
    Open Source Downstream
    Reference Platform
    Federated
    Meta Operator
    

    View Slide

16. Open Data Hub Is ...
    Open Source Downstream
    Reference Platform
    Federated
    Meta Operator
    

    View Slide

17. Open Data Hub Is ...
    Open Source Downstream
    Reference Platform
    Federated
    Meta Operator
    

    View Slide

18. Open Data Hub Is ...
    Open Source Downstream
    Reference Platform
    Federated
    Meta Operator
    

    View Slide

19. Open Data Hub Is ...
    Open Source Downstream
    Reference Platform
    Federated
    Meta Operator
    

    View Slide

20. Data Science with ODH
    Set
    goals
    Gather and
    prepare data
    Develop ML
    model
    Deploy ML
    models in app
    dev process
    Implement
    Apps &
    Inference
    ML models
    Monitoring &
    Management
    

    View Slide

21. Data Science with ODH
    Set
    goals
    Gather and
    prepare data
    Develop ML
    model
    Deploy ML
    models in app
    dev process
    Implement
    Apps &
    Inference
    ML models
    Monitoring &
    Management
    

    View Slide

22. Data Science with ODH
    Set
    goals
    Gather and
    prepare data
    Develop ML
    model
    Deploy ML
    models in app
    dev process
    Implement
    Apps &
    Inference
    ML models
    Monitoring &
    Management
    App developer
    IT operations
    Data engineer
    Business
    leadership
    Data scientists
    ML Engineer
    

    View Slide

23. Data Science with ODH
    Set
    goals
    Gather and
    prepare data
    Develop ML
    model
    Deploy ML
    models in app
    dev process
    Implement
    Apps &
    Inference
    ML models
    Monitoring &
    Management
    App developer
    IT operations
    Data engineer
    Business
    leadership
    Data scientists
    ML Engineer
    Seldon
    Jupyter
    Ceph Spark
    TensorFlow
    Kafka
    SuperSet
    Argo/Airflow/Tekton
    Hue
    Prometheus/Grafana
    Argo/Airflow/Tekton Ceph
    Kafka Seldon Middleware
    M
    odel to
    M
    icroservice
    

    View Slide

24. Dog-Fooding ODH at Red Hat
    Application Logs
    Applications in the product release
    pipeline store their runtime logs in our
    system. These groups are also
    engaged for anomaly detection
    Cluster Metrics
    Operational metrics from OpenShift
    clusters. AIOps is engaged here.
    Customer Support Data
    Storage of customer data like
    SOSReports, customer feedback, etc.
    

    View Slide

25. Analogy: Spark on ODH
    ODH
    JupyterHub
    Launcher
    

    View Slide

26. Analogy: Spark on ODH
    ODH
    JupyterHub
    Launcher
    Jupyter
    Environment
    

    View Slide

27. Analogy: Spark on ODH
    ODH
    JupyterHub
    Launcher
    Spark
    SingleUser
    Profile
    Jupyter
    Environment
    

    View Slide

28. Analogy: Spark on ODH
    ODH
    JupyterHub
    Launcher
    Spark
    SingleUser
    Profile
    Spark Cluster
    Service Template
    Jupyter
    Environment
    

    View Slide

29. Analogy: Spark on ODH
    Spark cluster
    ODH
    JupyterHub
    Launcher
    Spark
    SingleUser
    Profile
    Spark Cluster
    Service Template
    Jupyter
    Environment
    

    View Slide

30. Analogy: Spark on ODH
    Spark cluster
    ODH
    JupyterHub
    Launcher
    Spark
    SingleUser
    Profile
    Spark Cluster
    Service Template
    Jupyter
    Environment
    

    View Slide

31. Analogy: Spark on ODH
    Spark cluster
    Spark
    SingleUser
    Profile
    Spark Cluster
    Service Template
    ConfigMap
    ConfigMap
    

    View Slide

32. Ray on ODH?
    Ray cluster
    ODH
    JupyterHub
    Launcher
    Ray
    SingleUser
    Profile
    Ray Cluster
    Service Template
    Jupyter
    Environment
    

    View Slide

33. Ray Single User Profile
    

    View Slide

34. Ray Cluster Service Template
    

    View Slide

35. Demo: Ray on ODH!
    Ray cluster
    ODH
    JupyterHub
    Launcher
    Ray
    SingleUser
    Profile
    Ray Cluster
    Service Template
    Jupyter
    Environment
    

    View Slide

36. Ray on ODH at the Mass-Open Cloud
    Led by Boston University, the MOC is a
    collaborative effort among BU, Harvard,
    UMass Amherst, MIT, and Northeastern
    University, as well as the Massachusetts
    Green High-Performance Computing
    Center (MGHPCC) and Oak Ridge
    National Laboratory (ORNL).
    It is supported by a broad alliance of
    industry partners, including Red Hat.
    

    View Slide

37. Ray on MOC
    ● Maximum 5 workers + 1 head
    ● 1 CPU, 1 GB memory
    ● Pre-installed:
    

    View Slide

38. Operate First
    https://www.operate-first.cloud/
    Developing
    Software In
    The Open
    Operating
    Software and
    Services In
    the Open
    

    View Slide

39. Operate First PRs for Ray
    https://github.com/operate-first/support/issues/102
    

    View Slide

40. Collaboration: IBM
    ● Ray with Code Engine
    ● Ray on IBM OpenShift Clusters
    ● Scikit-Learn pipelines on Ray
    ● Ray Use Cases
    ○ Machine Learning Model Explorations
    ○ Earth Science
    

    View Slide

41. IBM Research at Ray Summit
    Raghu Ganti: Scaling and Unifying SciKit Learn and Spark
    Pipelines using Ray
    Linsong Chu: Serverless Earth Science Data Labeling using
    Unsupervised Deep Learning with Ray
    

    View Slide

42. Roadmap
    ● Community Ray Operator in Catalog
    ● Maintain Ray Images via Project Thoth
    ● Community Use Cases With Jupyter
    ● Formal Integration With KF and ODH
    ● KF Pipeline Nodes Backed by Ray
    

    View Slide

43. Call To Action
    ● Play with Ray on Jupyter up on MOC
    ● File issues and PRs with op-1st
    ● Report Back! eje@redhat.com
    https://www.operate-first.cloud/users/moc-ray-demo/README.md
    https://odh.operate-first.cloud/
    

    View Slide

44. 1
    1
    π ≈
    4 Σ
    Σ( + )
    

    View Slide