Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Dagster & Geomagical
Search
Noah Kantrowitz
February 09, 2021
Programming
0
130
Dagster & Geomagical
Noah Kantrowitz
February 09, 2021
Tweet
Share
More Decks by Noah Kantrowitz
See All by Noah Kantrowitz
What Python Can Learn From Other Languages
coderanger
0
14
What Python Can Learn From Other Languages (with notes)
coderanger
0
72
Swiss Army Django: Small Footprint ETL (with notes) - DjangoCon US
coderanger
0
140
Swiss Army Django: Small Footprint ETL - DjangoCon US
coderanger
0
32
How to look at space: PyCon AU
coderanger
0
71
Swiss Army Django: Small Footprint ETL
coderanger
0
50
Swiss Army Django: Small Footprint ETL (with notes)
coderanger
0
58
Minimum Viable Kubernetes
coderanger
0
21
Minimum Viable Kubernetes (with notes)
coderanger
0
310
Other Decks in Programming
See All in Programming
feature環境をGitHub ActionsとCloudFormationでいい感じに管理する
nealle
2
310
APIのない大学ログインWebサービスをWKWebViewとJavaScriptでアプリ化した話
akidon0000
1
330
はしめてのプログラミングとロボット制御
watawatavoltage
0
290
大規模マルチテナントを解決するYugabyteDBという選択肢
nnaka2992
1
250
Architectures with Lightweight Stores: New Rules and Options
manfredsteyer
PRO
0
100
日付と正規化
megmogmog1965
0
140
I/O Extended Android in Korea 2024 ~ Whats new in Android development tools
pluu
0
250
Microservices rules (July 2024) : what good looks like
cer
PRO
0
1.6k
実用的かつリーズナブルな 「Azure × Gemini × LINE」~キャラクターBot 実装ライブデモ~
tomodo_ysys
1
170
Temporalを取り巻く仕様を整理する
sajikix
0
110
TiDB Serverless ~理想のServerless DBを考える~
soso_15315
1
160
Introduction to GitOps
hwchiu
0
110
Featured
See All Featured
Optimizing for Happiness
mojombo
373
69k
Navigating Team Friction
lara
181
13k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
78
15k
A better future with KSS
kneath
231
17k
ParisWeb 2013: Learning to Love: Crash Course in Emotional UX Design
dotmariusz
105
6.8k
GraphQLとの向き合い方2022年版
quramy
36
13k
Building a Scalable Design System with Sketch
lauravandoore
458
32k
Designing with Data
zakiwarfel
96
5k
What’s in a name? Adding method to the madness
productmarketing
PRO
21
2.9k
Music & Morning Musume
bryan
43
5.9k
Designing for Performance
lara
604
67k
The Illustrated Children's Guide to Kubernetes
chrisshort
39
47k
Transcript
Geomagical & Dagster Dagster Community Meeting
Noah Kantrowitz > @kantrn - coderanger.net > Principal Ops @
Geomagical > Part of the IKEA family > Augmented reality with furniture
Our Product
Starting Point > Celery & RabbitMQ > Each operation as
its own daemon > celery.canvas > Custom DAG compiler
Design Goals > Keeping most of the solid structure >
Improved DAG expressiveness > Low fixed overhead, compatible with autoscaling > More detailed tracking and metrics
Dagster > Met all our requirements for structural simplicity >
DAG compiler was a bit limited but growing fast > Highly responsive team Dagster > No execution setup that met our needs
But dagster_celery? > Solid and pipeline code commingled > Single
runtime environment > Hard to build a workflow around at scale
But dagster_k8s? > Fine for infrequent or non-customer facing tasks
> Do not put kube-apiserver in your hot path > No really, I mean it
None
Autoscaling > KEDA watching RabbitMQ > Zero-scale: only Dagit and
gRPC daemons > task_acks_late = True > worker_prefetch_multiplier = 1
Remote Solids > Independent release cycles for each Solid >
Can run multiple versions in parallel > Testing in isolation
Writing A Remote Solid app = SolidCelery('repo-something') @app.task(bind=True) def something(self,
foo: str) -> str: return f'Hello {foo}'
Proxy Solids @celery_solid(queue='repo-something') def something(context, item): output = yield {
'foo': item['bar'], } item['something'] = output yield Output(item)
Workflow > One git repo per Dagster repo > main.py
which holds "default" Pipeline > solids.py which defines proxy Solids > Misc other pipelines for testing and development
CI/CD Briefly, since this is its own rabbit hole >
Buildkite > kustomize edit set image > ArgoCD
Downsides > Slow cold start > No feedback during long
tasks > New and exciting bugs
How It's Going > Happy with overall progress > Still
dropping some tasks at load > Plan to move forward looks good
Future Plans > Async execution support > Events from solid
workers > Pipeline-level webhooks > Predictive auto-scaling? K8s Operator?
Can I Use This? Kinda sorta geomagical/dagster_geomagical
Thank You Questions?