Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Dagster & Geomagical
Search
Noah Kantrowitz
February 09, 2021
Programming
0
160
Dagster & Geomagical
Noah Kantrowitz
February 09, 2021
Tweet
Share
More Decks by Noah Kantrowitz
See All by Noah Kantrowitz
What Python Can Learn From Other Languages
coderanger
0
63
What Python Can Learn From Other Languages (with notes)
coderanger
0
160
Swiss Army Django: Small Footprint ETL (with notes) - DjangoCon US
coderanger
0
230
Swiss Army Django: Small Footprint ETL - DjangoCon US
coderanger
0
60
How to look at space: PyCon AU
coderanger
0
100
Swiss Army Django: Small Footprint ETL
coderanger
0
92
Swiss Army Django: Small Footprint ETL (with notes)
coderanger
0
82
Minimum Viable Kubernetes
coderanger
0
40
Minimum Viable Kubernetes (with notes)
coderanger
0
450
Other Decks in Programming
See All in Programming
「テストは愚直&&網羅的に書くほどよい」という誤解 / Test Smarter, Not Harder
munetoshi
1
220
おやつのお供はお決まりですか?@WWDC25 Recap -Japan-\(region).swift
shingangan
0
150
型で語るカタ
irof
0
800
CIを整備してメンテナンスを生成AIに任せる
hazumirr
0
190
Gemini CLIの"強み"を知る! Gemini CLIとClaude Codeを比較してみた!
kotahisafuru
2
150
レベル1の開発生産性向上に取り組む − 日々の作業の効率化・自動化を通じた改善活動
kesoji
1
350
Android 16KBページサイズ対応をはじめからていねいに
mine2424
0
640
ソフトウェア設計とAI技術の活用
masuda220
PRO
25
6.7k
The Niche of CDK Grant オブジェクトって何者?/the-niche-of-cdk-what-isgrant-object
hassaku63
1
690
iOS開発スターターキットの作り方
akidon0000
0
170
オンコール⼊⾨〜ページャーが鳴る前に、あなたが備えられること〜 / Before The Pager Rings
yktakaha4
2
1.1k
抽象化という思考のツール - 理解と活用 - / Abstraction-as-a-Tool-for-Thinking
shin1x1
1
330
Featured
See All Featured
The Power of CSS Pseudo Elements
geoffreycrofte
77
5.9k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
7
760
Reflections from 52 weeks, 52 projects
jeffersonlam
351
21k
Documentation Writing (for coders)
carmenintech
72
4.9k
Mobile First: as difficult as doing things right
swwweet
223
9.7k
How GitHub (no longer) Works
holman
314
140k
Six Lessons from altMBA
skipperchong
28
3.9k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
Testing 201, or: Great Expectations
jmmastey
43
7.6k
Building Flexible Design Systems
yeseniaperezcruz
328
39k
Intergalactic Javascript Robots from Outer Space
tanoku
271
27k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
229
22k
Transcript
Geomagical & Dagster Dagster Community Meeting
Noah Kantrowitz > @kantrn - coderanger.net > Principal Ops @
Geomagical > Part of the IKEA family > Augmented reality with furniture
Our Product
Starting Point > Celery & RabbitMQ > Each operation as
its own daemon > celery.canvas > Custom DAG compiler
Design Goals > Keeping most of the solid structure >
Improved DAG expressiveness > Low fixed overhead, compatible with autoscaling > More detailed tracking and metrics
Dagster > Met all our requirements for structural simplicity >
DAG compiler was a bit limited but growing fast > Highly responsive team Dagster > No execution setup that met our needs
But dagster_celery? > Solid and pipeline code commingled > Single
runtime environment > Hard to build a workflow around at scale
But dagster_k8s? > Fine for infrequent or non-customer facing tasks
> Do not put kube-apiserver in your hot path > No really, I mean it
None
Autoscaling > KEDA watching RabbitMQ > Zero-scale: only Dagit and
gRPC daemons > task_acks_late = True > worker_prefetch_multiplier = 1
Remote Solids > Independent release cycles for each Solid >
Can run multiple versions in parallel > Testing in isolation
Writing A Remote Solid app = SolidCelery('repo-something') @app.task(bind=True) def something(self,
foo: str) -> str: return f'Hello {foo}'
Proxy Solids @celery_solid(queue='repo-something') def something(context, item): output = yield {
'foo': item['bar'], } item['something'] = output yield Output(item)
Workflow > One git repo per Dagster repo > main.py
which holds "default" Pipeline > solids.py which defines proxy Solids > Misc other pipelines for testing and development
CI/CD Briefly, since this is its own rabbit hole >
Buildkite > kustomize edit set image > ArgoCD
Downsides > Slow cold start > No feedback during long
tasks > New and exciting bugs
How It's Going > Happy with overall progress > Still
dropping some tasks at load > Plan to move forward looks good
Future Plans > Async execution support > Events from solid
workers > Pipeline-level webhooks > Predictive auto-scaling? K8s Operator?
Can I Use This? Kinda sorta geomagical/dagster_geomagical
Thank You Questions?