Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Deploying Dask Distributed
Search
Jacob Tomlinson
May 19, 2021
Technology
0
210
Deploying Dask Distributed
Jacob Tomlinson
May 19, 2021
Tweet
Share
More Decks by Jacob Tomlinson
See All by Jacob Tomlinson
Dask Demo Day - How RAPIDS users use Dask
jacobtomlinson
0
17
Deploying multi-GPU workloads on Kubernetes in Python
jacobtomlinson
0
140
Overview of Dask on Kubernetes
jacobtomlinson
0
110
k8s-batch-sig - Dask on Kubernetes
jacobtomlinson
0
73
Deploying multi-GPU workloads on Kubernetes in Python
jacobtomlinson
0
110
Dask Overview - Stanford Legion all-hands
jacobtomlinson
0
160
Container Canary - Lightning Talk
jacobtomlinson
0
41
Parallelizing Your ETL with Dask on Kubeflow
jacobtomlinson
0
760
GPU accelerating your computation in Python
jacobtomlinson
0
210
Other Decks in Technology
See All in Technology
データ分析を支える技術 生成AI再入門
ishikawa_satoru
0
380
What is DRE? - Road to SRE NEXT@広島
chanyou0311
3
630
AI研修【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
130
DDDにおける認可の扱いとKotlinにおける実装パターン / authorization-for-ddd-and-kotlin-implement-pattern
urmot
4
390
データベース研修 分析向けSQL入門【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
110
テストケースの自動生成に生成AIの導入を試みた話と生成AIによる今後の期待
shift_evolve
0
190
「単なる OAuth 2.0 を認証に使うと、車が通れるほどのどでかいセキュリティー・ホールができる」のか検証してみた
terara
0
380
Classmethod Odyssey 登壇資料
yamahiro
0
390
ABEMAにおけるLLMを用いたコンテンツベース推薦システム導入と効果検証
cyberagentdevelopers
PRO
1
750
目標設定は好きですか? アジャイルとともに目標と向き合い続ける方法 / Do you like target Management?
kakehashi
10
3k
ギークの理想が7つ集まるエムスリーで夢を叶えよう - エムスリー株式会社
m3_engineering
1
260
AWSサービスメニュー開発をしていてAWSを好きだ!と感じた瞬間
toru_kubota
0
130
Featured
See All Featured
A Tale of Four Properties
chriscoyier
155
22k
Optimizing for Happiness
mojombo
373
69k
Building a Scalable Design System with Sketch
lauravandoore
458
32k
Writing Fast Ruby
sferik
623
60k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
228
16k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
35
6.3k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
29
2.5k
Build your cross-platform service in a week with App Engine
jlugia
227
17k
VelocityConf: Rendering Performance Case Studies
addyosmani
321
23k
The Cult of Friendly URLs
andyhume
75
5.9k
Mobile First: as difficult as doing things right
swwweet
219
8.8k
Embracing the Ebb and Flow
colly
81
4.3k
Transcript
Deployment Workshop Deploying Dask Distributed Jacob Tomlinson
Dask Distributed A centrally managed, distributed, dynamic task scheduler
Dask Overview
None
Worker Worker Worker Scheduler Client Protocols TCP UCX Websocket Dask
components can communicate via a variety of different protocols.
Scheduler Starting a scheduler
Connecting a worker Worker Scheduler
Client Scheduler Worker Connecting a client
Client Scheduler Worker Submitting work
Dask Dashboard
JupyterLab Extension
Cluster Managers Utility classes to simplify cluster creation
Local Cluster Scheduler Worker Worker Worker Worker LocalCluster creates everything
for you. It will break down a large CPU into multiple workers withy multiple threads as this can be more performant.
Client Local Cluster Scheduler Worker Worker Worker Worker
Get logs
Scaling
How do I get more resource? Moving beyond a single
machine
SSH ... You could SSH to a bunch of machines
and start the Dask components manually.
SSHCluster Or you could use SSHCluster which will bootstrap a
cluster for you on a list of machines. All you need is passwordless SSH configured for each machine.
None
Deployment Workshop Thank you! @_jacobtomlinson