Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Dagster & Geomagical
Search
Noah Kantrowitz
February 09, 2021
Programming
0
170
Dagster & Geomagical
Noah Kantrowitz
February 09, 2021
Tweet
Share
More Decks by Noah Kantrowitz
See All by Noah Kantrowitz
What Python Can Learn From Other Languages
coderanger
0
66
What Python Can Learn From Other Languages (with notes)
coderanger
0
170
Swiss Army Django: Small Footprint ETL (with notes) - DjangoCon US
coderanger
0
240
Swiss Army Django: Small Footprint ETL - DjangoCon US
coderanger
0
65
How to look at space: PyCon AU
coderanger
0
110
Swiss Army Django: Small Footprint ETL
coderanger
0
94
Swiss Army Django: Small Footprint ETL (with notes)
coderanger
0
86
Minimum Viable Kubernetes
coderanger
0
41
Minimum Viable Kubernetes (with notes)
coderanger
0
460
Other Decks in Programming
See All in Programming
より安全で効率的な Go コードへ: Protocol Buffers Opaque API の導入
shwatanap
2
770
今だからこそ入門する Server-Sent Events (SSE)
nearme_tech
PRO
3
250
AIを活用し、今後に備えるための技術知識 / Basic Knowledge to Utilize AI
kishida
22
5.9k
Introducing ReActionView: A new ActionView-compatible ERB Engine @ Rails World 2025, Amsterdam
marcoroth
0
710
ファインディ株式会社におけるMCP活用とサービス開発
starfish719
0
2k
楽して成果を出すためのセルフリソース管理
clipnote
0
190
1から理解するWeb Push
dora1998
7
1.9k
AIでLINEスタンプを作ってみた
eycjur
1
230
AIと私たちの学習の変化を考える - Claude Codeの学習モードを例に
azukiazusa1
11
4.4k
Reading Rails 1.0 Source Code
okuramasafumi
0
250
The Past, Present, and Future of Enterprise Java with ASF in the Middle
ivargrimstad
0
170
go test -json そして testing.T.Attr / Kyoto.go #63
utgwkk
3
310
Featured
See All Featured
The Pragmatic Product Professional
lauravandoore
36
6.9k
Building Applications with DynamoDB
mza
96
6.6k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Intergalactic Javascript Robots from Outer Space
tanoku
272
27k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
285
14k
Designing for humans not robots
tammielis
253
25k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
188
55k
We Have a Design System, Now What?
morganepeng
53
7.8k
Fireside Chat
paigeccino
39
3.6k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
930
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
46
7.6k
Scaling GitHub
holman
463
140k
Transcript
Geomagical & Dagster Dagster Community Meeting
Noah Kantrowitz > @kantrn - coderanger.net > Principal Ops @
Geomagical > Part of the IKEA family > Augmented reality with furniture
Our Product
Starting Point > Celery & RabbitMQ > Each operation as
its own daemon > celery.canvas > Custom DAG compiler
Design Goals > Keeping most of the solid structure >
Improved DAG expressiveness > Low fixed overhead, compatible with autoscaling > More detailed tracking and metrics
Dagster > Met all our requirements for structural simplicity >
DAG compiler was a bit limited but growing fast > Highly responsive team Dagster > No execution setup that met our needs
But dagster_celery? > Solid and pipeline code commingled > Single
runtime environment > Hard to build a workflow around at scale
But dagster_k8s? > Fine for infrequent or non-customer facing tasks
> Do not put kube-apiserver in your hot path > No really, I mean it
None
Autoscaling > KEDA watching RabbitMQ > Zero-scale: only Dagit and
gRPC daemons > task_acks_late = True > worker_prefetch_multiplier = 1
Remote Solids > Independent release cycles for each Solid >
Can run multiple versions in parallel > Testing in isolation
Writing A Remote Solid app = SolidCelery('repo-something') @app.task(bind=True) def something(self,
foo: str) -> str: return f'Hello {foo}'
Proxy Solids @celery_solid(queue='repo-something') def something(context, item): output = yield {
'foo': item['bar'], } item['something'] = output yield Output(item)
Workflow > One git repo per Dagster repo > main.py
which holds "default" Pipeline > solids.py which defines proxy Solids > Misc other pipelines for testing and development
CI/CD Briefly, since this is its own rabbit hole >
Buildkite > kustomize edit set image > ArgoCD
Downsides > Slow cold start > No feedback during long
tasks > New and exciting bugs
How It's Going > Happy with overall progress > Still
dropping some tasks at load > Plan to move forward looks good
Future Plans > Async execution support > Events from solid
workers > Pipeline-level webhooks > Predictive auto-scaling? K8s Operator?
Can I Use This? Kinda sorta geomagical/dagster_geomagical
Thank You Questions?