dask_cloudprovider import ECSCluster cluster = ECSCluster( cluster_arn="arn" ) cluster.scale(10) AWS Fargate • Managed container platform • Scale by CPU and Memory • Billing per CPU/memory second • Low account limits (~50 workers) AWS Elastic Container Service • Unmanaged container platform • Full control over VM type (GPU, ARM) • Scale by VMs • Billing per VM second
settings - Easy to use interfaces for interacting with cloud resources (GUI, Python SDK, R SDK, ML CLI) - Powerful hundred node clusters of Azure CPU or GPU VMs for various workloads
settings - Easy to use interfaces for interacting with cloud resources (GUI, Python SDK, R SDK, ML CLI) - Powerful hundred node clusters of Azure CPU or GPU VMs for various workloads Data science, ML Software development
settings - Easy to use interfaces for interacting with cloud resources (GUI, Python SDK, R SDK, ML CLI) - Powerful hundred node clusters of Azure CPU or GPU VMs for various workloads Data science, ML Software development Distributed systems and HPC
Starts the scheduler via an experiment run • Headnode also runs a worker (maximize resources utilization) • Submits an experiment run for each worker • Port forwarding: • Port mapping via socat if on the same VNET • SSH-tunnel port forward otherwise (needs SSH creds) https://github.com/dask/dask-cloudprovider/pull/67
@ Azure ML - [email protected] - Tom - Senior Data Scientist @ Azure ML - https://github.com/lostmygithubaccount/dasky - CPU demos - https://github.com/drabastomek/GTC - GPU demos - @tomekdrabas @codydkdc - Twitter - NVIDIA’s GTC in San Jose and Microsoft’s //build in Seattle