Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Dagster & Geomagical
Search
Noah Kantrowitz
February 09, 2021
Programming
0
170
Dagster & Geomagical
Noah Kantrowitz
February 09, 2021
Tweet
Share
More Decks by Noah Kantrowitz
See All by Noah Kantrowitz
What Python Can Learn From Other Languages
coderanger
0
67
What Python Can Learn From Other Languages (with notes)
coderanger
0
170
Swiss Army Django: Small Footprint ETL (with notes) - DjangoCon US
coderanger
0
240
Swiss Army Django: Small Footprint ETL - DjangoCon US
coderanger
0
65
How to look at space: PyCon AU
coderanger
0
110
Swiss Army Django: Small Footprint ETL
coderanger
0
95
Swiss Army Django: Small Footprint ETL (with notes)
coderanger
0
86
Minimum Viable Kubernetes
coderanger
0
42
Minimum Viable Kubernetes (with notes)
coderanger
0
470
Other Decks in Programming
See All in Programming
Namespace and Its Future
tagomoris
6
710
RDoc meets YARD
okuramasafumi
4
170
Deep Dive into Kotlin Flow
jmatsu
1
370
Putting The Genie in the Bottle - A Crash Course on running LLMs on Android
iurysza
0
140
Flutter with Dart MCP: All You Need - 박제창 2025 I/O Extended Busan
itsmedreamwalker
0
150
実用的なGOCACHEPROG実装をするために / golang.tokyo #40
mazrean
1
300
もうちょっといいRubyプロファイラを作りたい (2025)
osyoyu
1
460
MCPでVibe Working。そして、結局はContext Eng(略)/ Working with Vibe on MCP And Context Eng
rkaga
5
2.3k
プロポーザル駆動学習 / Proposal-Driven Learning
mackey0225
2
1.3k
複雑なフォームに立ち向かう Next.js の技術選定
macchiitaka
2
250
請來的 AI Agent 同事們在寫程式時,怎麼用 pytest 去除各種幻想與盲點
keitheis
0
130
アプリの "かわいい" を支えるアニメーションツールRiveについて
uetyo
0
280
Featured
See All Featured
Practical Orchestrator
shlominoach
190
11k
Measuring & Analyzing Core Web Vitals
bluesmoon
9
590
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
Agile that works and the tools we love
rasmusluckow
330
21k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
4k
[RailsConf 2023] Rails as a piece of cake
palkan
57
5.8k
StorybookのUI Testing Handbookを読んだ
zakiyama
31
6.1k
Documentation Writing (for coders)
carmenintech
74
5k
How STYLIGHT went responsive
nonsquared
100
5.8k
The World Runs on Bad Software
bkeepers
PRO
70
11k
The Cost Of JavaScript in 2023
addyosmani
53
8.9k
Code Review Best Practice
trishagee
71
19k
Transcript
Geomagical & Dagster Dagster Community Meeting
Noah Kantrowitz > @kantrn - coderanger.net > Principal Ops @
Geomagical > Part of the IKEA family > Augmented reality with furniture
Our Product
Starting Point > Celery & RabbitMQ > Each operation as
its own daemon > celery.canvas > Custom DAG compiler
Design Goals > Keeping most of the solid structure >
Improved DAG expressiveness > Low fixed overhead, compatible with autoscaling > More detailed tracking and metrics
Dagster > Met all our requirements for structural simplicity >
DAG compiler was a bit limited but growing fast > Highly responsive team Dagster > No execution setup that met our needs
But dagster_celery? > Solid and pipeline code commingled > Single
runtime environment > Hard to build a workflow around at scale
But dagster_k8s? > Fine for infrequent or non-customer facing tasks
> Do not put kube-apiserver in your hot path > No really, I mean it
None
Autoscaling > KEDA watching RabbitMQ > Zero-scale: only Dagit and
gRPC daemons > task_acks_late = True > worker_prefetch_multiplier = 1
Remote Solids > Independent release cycles for each Solid >
Can run multiple versions in parallel > Testing in isolation
Writing A Remote Solid app = SolidCelery('repo-something') @app.task(bind=True) def something(self,
foo: str) -> str: return f'Hello {foo}'
Proxy Solids @celery_solid(queue='repo-something') def something(context, item): output = yield {
'foo': item['bar'], } item['something'] = output yield Output(item)
Workflow > One git repo per Dagster repo > main.py
which holds "default" Pipeline > solids.py which defines proxy Solids > Misc other pipelines for testing and development
CI/CD Briefly, since this is its own rabbit hole >
Buildkite > kustomize edit set image > ArgoCD
Downsides > Slow cold start > No feedback during long
tasks > New and exciting bugs
How It's Going > Happy with overall progress > Still
dropping some tasks at load > Plan to move forward looks good
Future Plans > Async execution support > Events from solid
workers > Pipeline-level webhooks > Predictive auto-scaling? K8s Operator?
Can I Use This? Kinda sorta geomagical/dagster_geomagical
Thank You Questions?