Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Dask on Kubernetes

Dask on Kubernetes

Jacob Tomlinson

May 19, 2021

More Decks by Jacob Tomlinson

Other Decks in Technology


  1. Fixed clusters More traditional cluster deployments where you set things

    up and leave them running indefinitely. They idle while not in use but are always ready to go when you need them.
  2. Ephemeral clusters Dynamic clusters which are created in the moment

    of need and destroyed again when you’re done. These rely on some underlying scheduling system which can quickly provision resources.
  3. Dask Helm Chart A chart which launches a fixed size

    Dask cluster alongside a Jupyter notebook. Fixed
  4. Service Worker Worker Worker Scheduler Jupyter Dask Helm Chart Deployment

    Service Deployment Deployment Ingress Ingress 💻 The user creates the cluster once. Then they can connect to it multiple times in the future.
  5. Service Worker Worker Worker Scheduler Jupyter Dask Helm Chart Deployment

    Service Deployment Deployment Ingress Ingress If the user disconnects the cluster still exists.
  6. Worker Worker Worker Scheduler KubeCluster Service Ingress 🏽 💻 Worker

    The user dynamically creates the cluster resources at runtime.
  7. HelmCluster() Connects to an existing Helm Chart deployment and provides

    the cluster manager interface including log retrieval and manual scaling.
  8. dask-gateway A central hub which launches Dask clusters on behalf

    of users. Can launch onto Kubernetes (and more). Ephemeral/Fixed
  9. Node Scheduler Worker Worker Worker Dask Gateway Ingress Worker Service

    Dask Gateway Dask Proxy Service Worker Scheduler Scheduler Worker Worker Node Node Node The user connects to the gateway and requests cluster resources. Dask gateway launches the cluster on their behalf and proxies traffic through.
  10. DaskHub Helm Chart JupyterHub and Dask Gateway packaged as a

    single Helm chart. Provides a central portal for launching Jupyter and Dask together. Ephemeral/Fixed
  11. Jupyter Hub Jupyter Proxy Jupyter Scheduler Scheduler Worker Worker Worker

    DaskHub Helm Chart 💻 Dask Gateway Service Service Ingress Jupyter Worker Service Lines omitted for clarity. Things got a bit crazy. Just imagine lines from basically everything to everything. Jupyter Auth Jupyter Spawner Database
  12. Deployment Workshop Stay tuned for Deploy JupyterHub with Dask Gateway

    on Kubernetes in 15 minutes Amit Kumar, Adam Lewis 17:20 UTC
  13. Dask Deployments on Kubernetes Gateway Dask Helm Chart Fixed Deploys

    a Jupyter Server and Dask cluster via Kubernetes deployments. Can be manually scaled via kubectl/helm/dask_kubernetes.HelmCluster(). https://github.com/dask/helm-chart dask_kubernetes.KubeCluster() Ephemeral Dynamically launch Dask clusters onto Kubernetes and scale adaptively. Gets garbage collected when idle. https://github.com/dask/dask-kubernetes Dask Gateway Ephemeral/Fixed/Centralized Central hub for spawning Dask Clusters. Great for teams and organizations who want to run many clusters for many users. https://gateway.dask.org/