Dask on Kubernetes

Transcript

1. Deployment Workshop Dask on Kubernetes Jacob Tomlinson
2. Types of Dask cluster Fixed Ephemeral
3. Fixed clusters More traditional cluster deployments where you set things up and leave them running indefinitely. They idle while not in use but are always ready to go when you need them.
4. Ephemeral clusters Dynamic clusters which are created in the moment of need and destroyed again when you’re done. These rely on some underlying scheduling system which can quickly provision resources.
5. Dask Helm Chart A chart which launches a fixed size Dask cluster alongside a Jupyter notebook. Fixed
7. Service Worker Worker Worker Scheduler Jupyter Dask Helm Chart Deployment Service Deployment Deployment Ingress Ingress 💻 The user creates the cluster once. Then they can connect to it multiple times in the future.
8. Service Worker Worker Worker Scheduler Jupyter Dask Helm Chart Deployment Service Deployment Deployment Ingress Ingress If the user disconnects the cluster still exists.
9. dask-kubernetes A collection of cluster managers and utilities for Kubernetes Ephemeral
10. KubeCluster() Spawns ephemeral clusters by requesting Pods directly via the Kubernetes API.
12. Worker Worker Worker Scheduler KubeCluster Service Ingress 🏽 💻 Worker The user dynamically creates the cluster resources at runtime.
13. Worker Worker Worker Scheduler Service Ingress Worker KubeCluster If the user disconnects.
14. KubeCluster The cluster is garbage collected.
15. HelmCluster() Connects to an existing Helm Chart deployment and provides the cluster manager interface including log retrieval and manual scaling.
17. dask-gateway A central hub which launches Dask clusters on behalf of users. Can launch onto Kubernetes (and more). Ephemeral/Fixed
18. Node Scheduler Worker Worker Worker Dask Gateway Ingress Worker Service Dask Gateway Dask Proxy Service Worker Scheduler Scheduler Worker Worker Node Node Node The user connects to the gateway and requests cluster resources. Dask gateway launches the cluster on their behalf and proxies traffic through.
20. DaskHub Helm Chart JupyterHub and Dask Gateway packaged as a single Helm chart. Provides a central portal for launching Jupyter and Dask together. Ephemeral/Fixed
21. Jupyter Hub Jupyter Proxy Jupyter Scheduler Scheduler Worker Worker Worker DaskHub Helm Chart 💻 Dask Gateway Service Service Ingress Jupyter Worker Service Lines omitted for clarity. Things got a bit crazy. Just imagine lines from basically everything to everything. Jupyter Auth Jupyter Spawner Database
22. Deployment Workshop Stay tuned for Deploy JupyterHub with Dask Gateway on Kubernetes in 15 minutes Amit Kumar, Adam Lewis 17:20 UTC
23. Dask Deployments on Kubernetes Gateway Dask Helm Chart Fixed Deploys a Jupyter Server and Dask cluster via Kubernetes deployments. Can be manually scaled via kubectl/helm/dask_kubernetes.HelmCluster(). https://github.com/dask/helm-chart dask_kubernetes.KubeCluster() Ephemeral Dynamically launch Dask clusters onto Kubernetes and scale adaptively. Gets garbage collected when idle. https://github.com/dask/dask-kubernetes Dask Gateway Ephemeral/Fixed/Centralized Central hub for spawning Dask Clusters. Great for teams and organizations who want to run many clusters for many users. https://gateway.dask.org/
24. Deployment Workshop Thank you! @_jacobtomlinson