Dask on Kubernetes

Transcript

1. Deployment Workshop Dask on Kubernetes Jacob Tomlinson

2. Types of Dask cluster Fixed Ephemeral

3. Fixed clusters More traditional cluster deployments where you set things

   up and leave them running indefinitely. They idle while not in use but are always ready to go when you need them.

4. Ephemeral clusters Dynamic clusters which are created in the moment

   of need and destroyed again when you’re done. These rely on some underlying scheduling system which can quickly provision resources.

5. Dask Helm Chart A chart which launches a fixed size

   Dask cluster alongside a Jupyter notebook. Fixed
6. None

7. Service Worker Worker Worker Scheduler Jupyter Dask Helm Chart Deployment

   Service Deployment Deployment Ingress Ingress 💻 The user creates the cluster once. Then they can connect to it multiple times in the future.

8. Service Worker Worker Worker Scheduler Jupyter Dask Helm Chart Deployment

   Service Deployment Deployment Ingress Ingress If the user disconnects the cluster still exists.

9. dask-kubernetes A collection of cluster managers and utilities for Kubernetes

   Ephemeral

10. KubeCluster() Spawns ephemeral clusters by requesting Pods directly via the

    Kubernetes API.
11. None

12. Worker Worker Worker Scheduler KubeCluster Service Ingress 🏽 💻 Worker

    The user dynamically creates the cluster resources at runtime.

13. Worker Worker Worker Scheduler Service Ingress Worker KubeCluster If the

    user disconnects.

14. KubeCluster The cluster is garbage collected.

15. HelmCluster() Connects to an existing Helm Chart deployment and provides

    the cluster manager interface including log retrieval and manual scaling.
16. None

17. dask-gateway A central hub which launches Dask clusters on behalf

    of users. Can launch onto Kubernetes (and more). Ephemeral/Fixed

18. Node Scheduler Worker Worker Worker Dask Gateway Ingress Worker Service

    Dask Gateway Dask Proxy Service Worker Scheduler Scheduler Worker Worker Node Node Node The user connects to the gateway and requests cluster resources. Dask gateway launches the cluster on their behalf and proxies traffic through.
19. None

20. DaskHub Helm Chart JupyterHub and Dask Gateway packaged as a

    single Helm chart. Provides a central portal for launching Jupyter and Dask together. Ephemeral/Fixed

21. Jupyter Hub Jupyter Proxy Jupyter Scheduler Scheduler Worker Worker Worker

    DaskHub Helm Chart 💻 Dask Gateway Service Service Ingress Jupyter Worker Service Lines omitted for clarity. Things got a bit crazy. Just imagine lines from basically everything to everything. Jupyter Auth Jupyter Spawner Database

22. Deployment Workshop Stay tuned for Deploy JupyterHub with Dask Gateway

    on Kubernetes in 15 minutes Amit Kumar, Adam Lewis 17:20 UTC

23. Dask Deployments on Kubernetes Gateway Dask Helm Chart Fixed Deploys

    a Jupyter Server and Dask cluster via Kubernetes deployments. Can be manually scaled via kubectl/helm/dask_kubernetes.HelmCluster(). https://github.com/dask/helm-chart dask_kubernetes.KubeCluster() Ephemeral Dynamically launch Dask clusters onto Kubernetes and scale adaptively. Gets garbage collected when idle. https://github.com/dask/dask-kubernetes Dask Gateway Ephemeral/Fixed/Centralized Central hub for spawning Dask Clusters. Great for teams and organizations who want to run many clusters for many users. https://gateway.dask.org/

24. Deployment Workshop Thank you! @_jacobtomlinson