Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Dagster & Geomagical
Search
Noah Kantrowitz
February 09, 2021
Programming
0
160
Dagster & Geomagical
Noah Kantrowitz
February 09, 2021
Tweet
Share
More Decks by Noah Kantrowitz
See All by Noah Kantrowitz
What Python Can Learn From Other Languages
coderanger
0
60
What Python Can Learn From Other Languages (with notes)
coderanger
0
160
Swiss Army Django: Small Footprint ETL (with notes) - DjangoCon US
coderanger
0
220
Swiss Army Django: Small Footprint ETL - DjangoCon US
coderanger
0
58
How to look at space: PyCon AU
coderanger
0
100
Swiss Army Django: Small Footprint ETL
coderanger
0
88
Swiss Army Django: Small Footprint ETL (with notes)
coderanger
0
81
Minimum Viable Kubernetes
coderanger
0
37
Minimum Viable Kubernetes (with notes)
coderanger
0
440
Other Decks in Programming
See All in Programming
第9回 情シス転職ミートアップ 株式会社IVRy(アイブリー)の紹介
ivry_presentationmaterials
1
230
KotlinConf 2025 現地で感じたServer-Side Kotlin
n_takehata
1
230
技術同人誌をMCP Serverにしてみた
74th
0
290
プロダクト志向ってなんなんだろうね
righttouch
PRO
0
160
Cursor AI Agentと伴走する アプリケーションの高速リプレイス
daisuketakeda
1
130
LINEヤフー データグループ紹介
lycorp_recruit_jp
0
870
生成AIコーディングとの向き合い方、AIと共創するという考え方 / How to deal with generative AI coding and the concept of co-creating with AI
seike460
PRO
1
330
童醫院敏捷轉型的實踐經驗
cclai999
0
190
PHPで始める振る舞い駆動開発(Behaviour-Driven Development)
ohmori_yusuke
2
180
すべてのコンテキストを、 ユーザー価値に変える
applism118
2
750
20250628_非エンジニアがバイブコーディングしてみた
ponponmikankan
0
350
Deep Dive into ~/.claude/projects
hiragram
7
1.3k
Featured
See All Featured
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
48
2.8k
BBQ
matthewcrist
89
9.7k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Fireside Chat
paigeccino
37
3.5k
Building a Modern Day E-commerce SEO Strategy
aleyda
41
7.3k
RailsConf 2023
tenderlove
30
1.1k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
46
9.6k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
53
2.8k
Music & Morning Musume
bryan
46
6.6k
Gamification - CAS2011
davidbonilla
81
5.3k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
331
22k
Transcript
Geomagical & Dagster Dagster Community Meeting
Noah Kantrowitz > @kantrn - coderanger.net > Principal Ops @
Geomagical > Part of the IKEA family > Augmented reality with furniture
Our Product
Starting Point > Celery & RabbitMQ > Each operation as
its own daemon > celery.canvas > Custom DAG compiler
Design Goals > Keeping most of the solid structure >
Improved DAG expressiveness > Low fixed overhead, compatible with autoscaling > More detailed tracking and metrics
Dagster > Met all our requirements for structural simplicity >
DAG compiler was a bit limited but growing fast > Highly responsive team Dagster > No execution setup that met our needs
But dagster_celery? > Solid and pipeline code commingled > Single
runtime environment > Hard to build a workflow around at scale
But dagster_k8s? > Fine for infrequent or non-customer facing tasks
> Do not put kube-apiserver in your hot path > No really, I mean it
None
Autoscaling > KEDA watching RabbitMQ > Zero-scale: only Dagit and
gRPC daemons > task_acks_late = True > worker_prefetch_multiplier = 1
Remote Solids > Independent release cycles for each Solid >
Can run multiple versions in parallel > Testing in isolation
Writing A Remote Solid app = SolidCelery('repo-something') @app.task(bind=True) def something(self,
foo: str) -> str: return f'Hello {foo}'
Proxy Solids @celery_solid(queue='repo-something') def something(context, item): output = yield {
'foo': item['bar'], } item['something'] = output yield Output(item)
Workflow > One git repo per Dagster repo > main.py
which holds "default" Pipeline > solids.py which defines proxy Solids > Misc other pipelines for testing and development
CI/CD Briefly, since this is its own rabbit hole >
Buildkite > kustomize edit set image > ArgoCD
Downsides > Slow cold start > No feedback during long
tasks > New and exciting bugs
How It's Going > Happy with overall progress > Still
dropping some tasks at load > Plan to move forward looks good
Future Plans > Async execution support > Events from solid
workers > Pipeline-level webhooks > Predictive auto-scaling? K8s Operator?
Can I Use This? Kinda sorta geomagical/dagster_geomagical
Thank You Questions?