Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Dagster & Geomagical
Search
Noah Kantrowitz
February 09, 2021
Programming
0
120
Dagster & Geomagical
Noah Kantrowitz
February 09, 2021
Tweet
Share
More Decks by Noah Kantrowitz
See All by Noah Kantrowitz
Swiss Army Django: Small Footprint ETL (with notes) - DjangoCon US
coderanger
0
110
Swiss Army Django: Small Footprint ETL - DjangoCon US
coderanger
0
21
How to look at space: PyCon AU
coderanger
0
58
Swiss Army Django: Small Footprint ETL
coderanger
0
42
Swiss Army Django: Small Footprint ETL (with notes)
coderanger
0
46
Minimum Viable Kubernetes
coderanger
0
17
Minimum Viable Kubernetes (with notes)
coderanger
0
290
Applied Science Fiction: Operating a Research-Led Product (with notes)
coderanger
0
22
Applied Science Fiction: Operating a Research-Led Product
coderanger
0
44
Other Decks in Programming
See All in Programming
Netty Chicago Java User Group 2024-04-17
sullis
0
170
Komplexe Oberflächen mit SVG und der Web Animation API
joergneumann
0
660
try! Swift Tokyo 2024 参加報告 / try! Swift Tokyo 2024 Report
hironytic
0
200
Snowflakeで眠ったデータを起こそう!
estie
0
110
Rubyでたのしむクリエイティブコーディング/Enjoy Creative coding with Ruby
chobishiba
1
170
大規模Reactアプリのリアーキテクチャ~8万行のTanStack Query移行の軌跡~
kj455
4
920
単体テストを書かない技術 #phpcon_odawara
o0h
PRO
26
8.1k
Rails と人魚の話/rails-and-mermaid
sanfrecce_osaka
0
100
"config" ってなんだ? / What is "config"?
okashoi
0
230
Goのmultiple errorsについて (2024年4月版)
syumai
1
260
Folding Cheat Sheet #2
philipschwarz
PRO
0
120
PHPの次期バージョンはこの時期どうなっているのか - Internalsの開発体制について - PHPカンファレンス小田原
youkidearitai
PRO
1
190
Featured
See All Featured
jQuery: Nuts, Bolts and Bling
dougneiner
59
7.1k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
220
21k
Facilitating Awesome Meetings
lara
41
5.6k
What's in a price? How to price your products and services
michaelherold
237
11k
Learning to Love Humans: Emotional Interface Design
aarron
266
39k
Unsuck your backbone
ammeep
662
57k
The Cost Of JavaScript in 2023
addyosmani
15
3.8k
Building Applications with DynamoDB
mza
88
5.6k
Practical Orchestrator
shlominoach
181
9.7k
Visualization
eitanlees
135
14k
Git: the NoSQL Database
bkeepers
PRO
422
63k
Six Lessons from altMBA
skipperchong
20
3k
Transcript
Geomagical & Dagster Dagster Community Meeting
Noah Kantrowitz > @kantrn - coderanger.net > Principal Ops @
Geomagical > Part of the IKEA family > Augmented reality with furniture
Our Product
Starting Point > Celery & RabbitMQ > Each operation as
its own daemon > celery.canvas > Custom DAG compiler
Design Goals > Keeping most of the solid structure >
Improved DAG expressiveness > Low fixed overhead, compatible with autoscaling > More detailed tracking and metrics
Dagster > Met all our requirements for structural simplicity >
DAG compiler was a bit limited but growing fast > Highly responsive team Dagster > No execution setup that met our needs
But dagster_celery? > Solid and pipeline code commingled > Single
runtime environment > Hard to build a workflow around at scale
But dagster_k8s? > Fine for infrequent or non-customer facing tasks
> Do not put kube-apiserver in your hot path > No really, I mean it
None
Autoscaling > KEDA watching RabbitMQ > Zero-scale: only Dagit and
gRPC daemons > task_acks_late = True > worker_prefetch_multiplier = 1
Remote Solids > Independent release cycles for each Solid >
Can run multiple versions in parallel > Testing in isolation
Writing A Remote Solid app = SolidCelery('repo-something') @app.task(bind=True) def something(self,
foo: str) -> str: return f'Hello {foo}'
Proxy Solids @celery_solid(queue='repo-something') def something(context, item): output = yield {
'foo': item['bar'], } item['something'] = output yield Output(item)
Workflow > One git repo per Dagster repo > main.py
which holds "default" Pipeline > solids.py which defines proxy Solids > Misc other pipelines for testing and development
CI/CD Briefly, since this is its own rabbit hole >
Buildkite > kustomize edit set image > ArgoCD
Downsides > Slow cold start > No feedback during long
tasks > New and exciting bugs
How It's Going > Happy with overall progress > Still
dropping some tasks at load > Plan to move forward looks good
Future Plans > Async execution support > Events from solid
workers > Pipeline-level webhooks > Predictive auto-scaling? K8s Operator?
Can I Use This? Kinda sorta geomagical/dagster_geomagical
Thank You Questions?