Dask Summit: Native Cloud Deployment with Dask-Cloudprovider

Transcript

1. dask-cloudprovider
   

   View Slide

2. Zero to Dask
   on the cloud
   

   View Slide

3. Overview of a cluster manager
   Cluster
   Scheduler
   Worker
   Worker
   Worker
   Client
   Cloud
   resource
   Cloud
   resource
   Cloud
   resource
   Cloud
   resource
   Array
   Dataframe
   Bag
   ML
   Xarray...
   

   View Slide

4. Cloud platform service types
   Functions
   Machine learning
   Kubernetes
   Containers
   Batch
   VMs
   ...as a service
   

   View Slide

5. Ephemeral nature of clusters
   Cluster
   Cluster Cloud
   resource
   Cloud
   resource
   $
   0
   $$
   $$
   0
   

   View Slide

6. AWS
   from dask_cloudprovider import FargateCluster
   cluster = FargateCluster()
   cluster.scale(10)
   from dask_cloudprovider import ECSCluster
   cluster = ECSCluster(
   cluster_arn="arn"
   )
   cluster.scale(10)
   AWS Fargate
   ● Managed container platform
   ● Scale by CPU and Memory
   ● Billing per CPU/memory second
   ● Low account limits (~50 workers)
   AWS Elastic Container Service
   ● Unmanaged container platform
   ● Full control over VM type (GPU, ARM)
   ● Scale by VMs
   ● Billing per VM second
   

   View Slide

7. AzureML
   - Targets Data Scientists from all backgrounds in enterprise settings
   - Easy to use interfaces for interacting with cloud resources (GUI, Python SDK, R SDK, ML CLI)
   - Powerful hundred node clusters of Azure CPU or GPU VMs for various workloads
   

   View Slide

8. AzureML
   - Targets Data Scientists from all backgrounds in enterprise settings
   - Easy to use interfaces for interacting with cloud resources (GUI, Python SDK, R SDK, ML CLI)
   - Powerful hundred node clusters of Azure CPU or GPU VMs for various workloads
   Data science, ML
   Software development
   

   View Slide

9. AzureML
   - Targets Data Scientists from all backgrounds in enterprise settings
   - Easy to use interfaces for interacting with cloud resources (GUI, Python SDK, R SDK, ML CLI)
   - Powerful hundred node clusters of Azure CPU or GPU VMs for various workloads
   Data science, ML
   Software development
   Distributed systems
   and HPC
   

   View Slide

10. AzureML | dask-cloudprovider
    # import from Azure ML Python SDK and Dask
    from azureml.core import Workspace
    from dask.distributed import Client
    from dask_cloudprovider import AzureMLCluster
    # specify Workspace - authenticate interactively or otherwise
    ws = Workspace.from_config() # see https://aka.ms/azureml/workspace
    # get (or create) desired Compute Target and Environment (base image + conda/pip installs)
    ct = ws.compute_targets[‘cpu-cluster’] # see https://aka.ms/azureml/computetarget
    env = ws.environments[‘AzureML-Dask-CPU’] # see https://aka.ms/azureml/environments
    # start cluster, print widget and links
    cluster = AzureMLCluster(ws, ct, env, initial_node_count=100, jupyter=True)
    # optionally, use directly in client
    c = Client(cluster) # optionally, use directly in Client
    

    View Slide

11. V100s!
    DS14_V2
    

    View Slide

12. AzureML | Architecture
    • Derives from the distributed.deploy.cluster.Cluster class
    • Starts the scheduler via an experiment run
    • Headnode also runs a worker (maximize resources utilization)
    • Submits an experiment run for each worker
    • Port forwarding:
    • Port mapping via socat if on the same VNET
    • SSH-tunnel port forward otherwise (needs SSH creds)
    https://github.com/dask/dask-cloudprovider/pull/67
    

    View Slide

13. AzureML | Links
    https://github.com/dask/dask-cloudprovider/pull/67
    - cody.peterson@microsoft.com - Cody - PM @ Azure ML
    - todrabas@microsoft.com - Tom - Senior Data Scientist @ Azure ML
    - https://github.com/lostmygithubaccount/dasky - CPU demos
    - https://github.com/drabastomek/GTC - GPU demos
    - @tomekdrabas @codydkdc - Twitter
    - NVIDIA’s GTC in San Jose and Microsoft’s //build in Seattle
    

    View Slide

14. AzureML | GPU overview
    

    View Slide

15. AzureML | Run architecture
    

    View Slide