Deploying Dask Distributed

Transcript

1. Deployment Workshop
   Deploying Dask Distributed
   Jacob Tomlinson
   

   View Slide

2. Dask Distributed
   A centrally managed, distributed, dynamic task scheduler
   

   View Slide

3. Dask Overview
   

   View Slide

4. View Slide

5. Worker Worker Worker
   Scheduler
   Client
   Protocols
   TCP
   UCX
   Websocket
   Dask components can
   communicate via a variety of
   different protocols.
   

   View Slide

6. Scheduler
   Starting a scheduler
   

   View Slide

7. Connecting a worker
   Worker
   Scheduler
   

   View Slide

8. Client Scheduler Worker
   Connecting a client
   

   View Slide

9. Client Scheduler Worker
   Submitting work
   

   View Slide

10. Dask Dashboard
    

    View Slide

11. JupyterLab Extension
    

    View Slide

12. Cluster Managers
    Utility classes to simplify cluster creation
    

    View Slide

13. Local Cluster
    Scheduler
    Worker
    Worker
    Worker
    Worker
    LocalCluster creates everything
    for you.
    It will break down a large CPU
    into multiple workers withy
    multiple threads as this can be
    more performant.
    

    View Slide

14. Client
    Local Cluster
    Scheduler
    Worker
    Worker
    Worker
    Worker
    

    View Slide

15. Get logs
    

    View Slide

16. Scaling
    

    View Slide

17. How do I get more resource?
    Moving beyond a single machine
    

    View Slide

18. SSH
    ...
    You could SSH to a bunch of
    machines and start the Dask
    components manually.
    

    View Slide

19. SSHCluster
    Or you could use SSHCluster which will
    bootstrap a cluster for you on a list of
    machines.
    All you need is passwordless SSH
    configured for each machine.
    

    View Slide

20. View Slide

21. Deployment Workshop
    Thank you!
    @_jacobtomlinson
    

    View Slide