Dagster & Geomagical

Transcript

1. Geomagical & Dagster Dagster Community Meeting
2. Noah Kantrowitz > @kantrn - coderanger.net > Principal Ops @ Geomagical > Part of the IKEA family > Augmented reality with furniture
3. Our Product
4. Starting Point > Celery & RabbitMQ > Each operation as its own daemon > celery.canvas > Custom DAG compiler
5. Design Goals > Keeping most of the solid structure > Improved DAG expressiveness > Low fixed overhead, compatible with autoscaling > More detailed tracking and metrics
6. Dagster > Met all our requirements for structural simplicity > DAG compiler was a bit limited but growing fast > Highly responsive team Dagster > No execution setup that met our needs
7. But dagster_celery? > Solid and pipeline code commingled > Single runtime environment > Hard to build a workflow around at scale
8. But dagster_k8s? > Fine for infrequent or non-customer facing tasks > Do not put kube-apiserver in your hot path > No really, I mean it
10. Autoscaling > KEDA watching RabbitMQ > Zero-scale: only Dagit and gRPC daemons > task_acks_late = True > worker_prefetch_multiplier = 1
11. Remote Solids > Independent release cycles for each Solid > Can run multiple versions in parallel > Testing in isolation
12. Writing A Remote Solid app = SolidCelery('repo-something') @app.task(bind=True) def something(self, foo: str) -> str: return f'Hello {foo}'
13. Proxy Solids @celery_solid(queue='repo-something') def something(context, item): output = yield { 'foo': item['bar'], } item['something'] = output yield Output(item)
14. Workflow > One git repo per Dagster repo > main.py which holds "default" Pipeline > solids.py which defines proxy Solids > Misc other pipelines for testing and development
15. CI/CD Briefly, since this is its own rabbit hole > Buildkite > kustomize edit set image > ArgoCD
16. Downsides > Slow cold start > No feedback during long tasks > New and exciting bugs
17. How It's Going > Happy with overall progress > Still dropping some tasks at load > Plan to move forward looks good
18. Future Plans > Async execution support > Events from solid workers > Pipeline-level webhooks > Predictive auto-scaling? K8s Operator?
19. Can I Use This? Kinda sorta geomagical/dagster_geomagical
20. Thank You Questions?