Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Dagster & Geomagical
Search
Noah Kantrowitz
February 09, 2021
Programming
0
140
Dagster & Geomagical
Noah Kantrowitz
February 09, 2021
Tweet
Share
More Decks by Noah Kantrowitz
See All by Noah Kantrowitz
What Python Can Learn From Other Languages
coderanger
0
38
What Python Can Learn From Other Languages (with notes)
coderanger
0
120
Swiss Army Django: Small Footprint ETL (with notes) - DjangoCon US
coderanger
0
190
Swiss Army Django: Small Footprint ETL - DjangoCon US
coderanger
0
44
How to look at space: PyCon AU
coderanger
0
85
Swiss Army Django: Small Footprint ETL
coderanger
0
71
Swiss Army Django: Small Footprint ETL (with notes)
coderanger
0
66
Minimum Viable Kubernetes
coderanger
0
30
Minimum Viable Kubernetes (with notes)
coderanger
0
390
Other Decks in Programming
See All in Programming
Boost Performance and Developer Productivity with Jakarta EE 11
ivargrimstad
0
620
ナレッジイネイブリングにAIを活用してみる ゆるSRE勉強会 #9
nealle
0
120
Pulsar2 を雰囲気で使ってみよう
anoken
0
250
一休.com のログイン体験を支える技術 〜Web Components x Vue.js 活用事例と最適化について〜
atsumim
0
770
AI Agent系IDEを使って 開発生産性を爆アゲする
ouchi2501
1
100
Ruby on cygwin 2025-02
fd0
0
170
お前もAI鬼にならないか?👹Bolt & Cursor & Supabase & Vercelで人間をやめるぞ、ジョジョー!👺
taishiyade
7
4.1k
Rails アプリ地図考 Flush Cut
makicamel
1
130
GAEログのコスト削減
mot_techtalk
0
120
負債になりにくいCSSをデザイナとつくるには?
fsubal
10
2.5k
Conform を推す - Advocating for Conform
mizoguchicoji
3
710
Visual StudioのGitHub Copilotでいろいろやってみる
tomokusaba
1
190
Featured
See All Featured
Product Roadmaps are Hard
iamctodd
PRO
50
11k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
47
5.2k
Docker and Python
trallard
44
3.3k
Rebuilding a faster, lazier Slack
samanthasiow
80
8.8k
Six Lessons from altMBA
skipperchong
27
3.6k
The Cost Of JavaScript in 2023
addyosmani
47
7.3k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
666
120k
The Cult of Friendly URLs
andyhume
78
6.2k
What's in a price? How to price your products and services
michaelherold
244
12k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
44
7k
Transcript
Geomagical & Dagster Dagster Community Meeting
Noah Kantrowitz > @kantrn - coderanger.net > Principal Ops @
Geomagical > Part of the IKEA family > Augmented reality with furniture
Our Product
Starting Point > Celery & RabbitMQ > Each operation as
its own daemon > celery.canvas > Custom DAG compiler
Design Goals > Keeping most of the solid structure >
Improved DAG expressiveness > Low fixed overhead, compatible with autoscaling > More detailed tracking and metrics
Dagster > Met all our requirements for structural simplicity >
DAG compiler was a bit limited but growing fast > Highly responsive team Dagster > No execution setup that met our needs
But dagster_celery? > Solid and pipeline code commingled > Single
runtime environment > Hard to build a workflow around at scale
But dagster_k8s? > Fine for infrequent or non-customer facing tasks
> Do not put kube-apiserver in your hot path > No really, I mean it
None
Autoscaling > KEDA watching RabbitMQ > Zero-scale: only Dagit and
gRPC daemons > task_acks_late = True > worker_prefetch_multiplier = 1
Remote Solids > Independent release cycles for each Solid >
Can run multiple versions in parallel > Testing in isolation
Writing A Remote Solid app = SolidCelery('repo-something') @app.task(bind=True) def something(self,
foo: str) -> str: return f'Hello {foo}'
Proxy Solids @celery_solid(queue='repo-something') def something(context, item): output = yield {
'foo': item['bar'], } item['something'] = output yield Output(item)
Workflow > One git repo per Dagster repo > main.py
which holds "default" Pipeline > solids.py which defines proxy Solids > Misc other pipelines for testing and development
CI/CD Briefly, since this is its own rabbit hole >
Buildkite > kustomize edit set image > ArgoCD
Downsides > Slow cold start > No feedback during long
tasks > New and exciting bugs
How It's Going > Happy with overall progress > Still
dropping some tasks at load > Plan to move forward looks good
Future Plans > Async execution support > Events from solid
workers > Pipeline-level webhooks > Predictive auto-scaling? K8s Operator?
Can I Use This? Kinda sorta geomagical/dagster_geomagical
Thank You Questions?