Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Dagster & Geomagical
Search
Noah Kantrowitz
February 09, 2021
Programming
0
140
Dagster & Geomagical
Noah Kantrowitz
February 09, 2021
Tweet
Share
More Decks by Noah Kantrowitz
See All by Noah Kantrowitz
What Python Can Learn From Other Languages
coderanger
0
38
What Python Can Learn From Other Languages (with notes)
coderanger
0
120
Swiss Army Django: Small Footprint ETL (with notes) - DjangoCon US
coderanger
0
180
Swiss Army Django: Small Footprint ETL - DjangoCon US
coderanger
0
44
How to look at space: PyCon AU
coderanger
0
85
Swiss Army Django: Small Footprint ETL
coderanger
0
71
Swiss Army Django: Small Footprint ETL (with notes)
coderanger
0
66
Minimum Viable Kubernetes
coderanger
0
30
Minimum Viable Kubernetes (with notes)
coderanger
0
390
Other Decks in Programming
See All in Programming
時計仕掛けのCompose
mkeeda
1
300
苦しいTiDBへの移行を乗り越えて快適な運用を目指す
leveragestech
0
600
Amazon S3 TablesとAmazon S3 Metadataを触ってみた / 20250201-jawsug-tochigi-s3tables-s3metadata
kasacchiful
0
170
Pythonでもちょっとリッチな見た目のアプリを設計してみる
ueponx
1
570
Flutter × Firebase Genkit で加速する生成 AI アプリ開発
coborinai
0
160
CSS Linter による Baseline サポートの仕組み
ryo_manba
1
110
AWS Organizations で実現する、 マルチ AWS アカウントのルートユーザー管理からの脱却
atpons
0
150
一休.com のログイン体験を支える技術 〜Web Components x Vue.js 活用事例と最適化について〜
atsumim
0
490
チームリードになって変わったこと
isaka1022
0
200
Software Architecture
hschwentner
6
2.1k
『GO』アプリ バックエンドサーバのコスト削減
mot_techtalk
0
140
Amazon ECS とマイクロサービスから考えるシステム構成
hiyanger
2
560
Featured
See All Featured
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
Java REST API Framework Comparison - PWX 2021
mraible
28
8.4k
StorybookのUI Testing Handbookを読んだ
zakiyama
28
5.5k
Code Reviewing Like a Champion
maltzj
521
39k
How to train your dragon (web standard)
notwaldorf
91
5.8k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
100
18k
The Straight Up "How To Draw Better" Workshop
denniskardys
232
140k
Gamification - CAS2011
davidbonilla
80
5.1k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
4
330
Designing for humans not robots
tammielis
250
25k
Docker and Python
trallard
44
3.3k
The Pragmatic Product Professional
lauravandoore
32
6.4k
Transcript
Geomagical & Dagster Dagster Community Meeting
Noah Kantrowitz > @kantrn - coderanger.net > Principal Ops @
Geomagical > Part of the IKEA family > Augmented reality with furniture
Our Product
Starting Point > Celery & RabbitMQ > Each operation as
its own daemon > celery.canvas > Custom DAG compiler
Design Goals > Keeping most of the solid structure >
Improved DAG expressiveness > Low fixed overhead, compatible with autoscaling > More detailed tracking and metrics
Dagster > Met all our requirements for structural simplicity >
DAG compiler was a bit limited but growing fast > Highly responsive team Dagster > No execution setup that met our needs
But dagster_celery? > Solid and pipeline code commingled > Single
runtime environment > Hard to build a workflow around at scale
But dagster_k8s? > Fine for infrequent or non-customer facing tasks
> Do not put kube-apiserver in your hot path > No really, I mean it
None
Autoscaling > KEDA watching RabbitMQ > Zero-scale: only Dagit and
gRPC daemons > task_acks_late = True > worker_prefetch_multiplier = 1
Remote Solids > Independent release cycles for each Solid >
Can run multiple versions in parallel > Testing in isolation
Writing A Remote Solid app = SolidCelery('repo-something') @app.task(bind=True) def something(self,
foo: str) -> str: return f'Hello {foo}'
Proxy Solids @celery_solid(queue='repo-something') def something(context, item): output = yield {
'foo': item['bar'], } item['something'] = output yield Output(item)
Workflow > One git repo per Dagster repo > main.py
which holds "default" Pipeline > solids.py which defines proxy Solids > Misc other pipelines for testing and development
CI/CD Briefly, since this is its own rabbit hole >
Buildkite > kustomize edit set image > ArgoCD
Downsides > Slow cold start > No feedback during long
tasks > New and exciting bugs
How It's Going > Happy with overall progress > Still
dropping some tasks at load > Plan to move forward looks good
Future Plans > Async execution support > Events from solid
workers > Pipeline-level webhooks > Predictive auto-scaling? K8s Operator?
Can I Use This? Kinda sorta geomagical/dagster_geomagical
Thank You Questions?