Lightning talk - Managing cluster lifecycle with dask-ctl

Transcript

1. Lightning Talk Managing cluster lifecycle with dask-ctl Jacob Tomlinson

2. github.com/dask-contrib/dask-ctl A set of tools to provide a control plane

   for managing the lifecycle of Dask clusters.

3. Worker Worker Worker Scheduler Cluster Manager Tight lifecycle coupling Today

   the Python process contains the cluster manager, which contains the only references to the resources created. This makes sense if the scheduler and workers are subprocesses on the same machine. But less so when they are remote and independent resources on a cloud or HPC platform. Python Process Remote Cluster resources

4. Worker Worker Worker Scheduler Tight lifecycle coupling What happens to

   the cluster resources if the Python process that created them is killed? Sometimes the OS may clean them up. Sometimes they will time out or hit a wall time. Sometimes they will exist until you manually delete them and cost you money. Cluster resources !??

5. Forcibly restarting your notebook kernel shouldn’t leave cluster resources on

   some cloud or other platform that need to be killed manually.

6. Discovery The big challenge around managing cluster lifecycle is being

   able to discover existing clusters and then reconstruct the cluster manager which represents them. With dask-ctl cluster managers can register a discovery method via the dask_cluster_discovery entrypoint. To support dask-ctl cluster managers should implement this entrypoint and search for clusters (by listing jobs/pods/vm/etc looking for Dask cluster resources) and then return an iterable of cluster names and cluster managers which can recreate them.

7. Reconstruction Once clusters can be discovered then all cluster managers

   need to have a way of reconstructing the representation by the clusters name/uuid. In dask-ctl we try to call a ClusterManager.from_name(name/uuid) class method to reconstruct the cluster object. To support dask-ctl cluster managers also need to implement this method.

8. Lifecycle Once we can discover clusters and reconstruct cluster managers

   to represent then we can always perform lifecycle management operations including: • Getting logs • Connecting clients • Scaling • Deleting Create Scale Compute Delete

9. Utilities Python CLI

10. Next steps • Implement dask-ctl support on as many cluster

    managers as possible. • Add support for cluster discovery to the Dask Jupyter Lab Extension so that discovered clusters are listed in the sidebar. • Stabilize things and move from dask-contrib org to dask.

11. Lightning Talk Thank you! @_jacobtomlinson github.com/dask-contrib/dask-ctl dask-ctl.readthedocs.io