Deploying Dask Distributed

Transcript

1. Deployment Workshop Deploying Dask Distributed Jacob Tomlinson
2. Dask Distributed A centrally managed, distributed, dynamic task scheduler
3. Dask Overview
5. Worker Worker Worker Scheduler Client Protocols TCP UCX Websocket Dask components can communicate via a variety of different protocols.
6. Scheduler Starting a scheduler
7. Connecting a worker Worker Scheduler
8. Client Scheduler Worker Connecting a client
9. Client Scheduler Worker Submitting work
10. Dask Dashboard
11. JupyterLab Extension
12. Cluster Managers Utility classes to simplify cluster creation
13. Local Cluster Scheduler Worker Worker Worker Worker LocalCluster creates everything for you. It will break down a large CPU into multiple workers withy multiple threads as this can be more performant.
14. Client Local Cluster Scheduler Worker Worker Worker Worker
15. Get logs
16. Scaling
17. How do I get more resource? Moving beyond a single machine
18. SSH ... You could SSH to a bunch of machines and start the Dask components manually.
19. SSHCluster Or you could use SSHCluster which will bootstrap a cluster for you on a list of machines. All you need is passwordless SSH configured for each machine.
21. Deployment Workshop Thank you! @_jacobtomlinson