Streamling: Lightweight, Extensible Streaming on DataFusion @ Berlin Buzzwords 2026

Apache DataFusion is moving beyond batch into streaming. We built Streamling, a Rust streaming engine that uses DataFusion planning and Arrow RecordBatch streams for real-time SQL/WASM transforms. This talk covers how we built it, highlights key features (FFI plugins, WASM transforms, and dynamic tables), and shares production lessons.

- open source: https://streamling.dev/
- talk page: https://2026.berlinbuzzwords.de/session/streamling-lightweight-extensible-streaming-on-datafusion/

Transcript

1. Streamling Lightweight, Extensible Streaming on DataFusion BERLIN BUZZWORDS 2026
2. Hi Xiao Meng • Streaming Team Lead @ Goldsky • Expert Data Engineer @ Activision Rafael Aguiar • Senior Software Eng @ Goldsky • Senior Software Eng @ Shopify
3. Streaming Pipeline specialized in Blockchain Sources What we do
4. Declarative Streaming Pipeline in the Cloud One YAML file. Three named blocks. Each step references the previous by name. SOURCES Unbounded Data sources: raw.transactions: type: kafka topic: raw.event.transaction TRANSFORMS How it's shaped transforms: large_transactions: type: sql primary_key: id sql: | SELECT * FROM raw.transactions WHERE amount > 1000 SINKS Where it lands sinks: pg.large_transactions: from: large_transactions type: postgres schema: public table: large_transactions primary_key: id
5. Runtime Engine: Our Flink Era Product Goldsky Mirror 2024 June Battle Tested De facto Standard for Streaming Processing Fault Tolerance: Checkpoint & Savepoint Flink k8s Operator for Cloud Deployment
7. PART 1 The Engine Internals How we built a streaming SQL engine on DataFusion for workloads that don't need distributed stateful processing.
8. From Flink to Streamling We had a streaming platform. We learned from it. Then we built lighter. 1 BEFORE Our Flink era Flink served us well — mature ecosystem, big community. It also paid us back in operational complexity, and a real $/month floor. 2 TURNING POINT What we actually needed Most of our workloads are CDC fanout, enrichment, dedup, simple transforms. None of them need distributed shuffle. We were paying for distribution we couldn't cash in. 3 NOW A specific point in the design space DataFusion + Arrow matured into a usable substrate 20222024) for streaming. 5s from paused to running across our fleet — customers feel it as the whole platform being faster.
9. Scope as a feature What's in. What's out. And why the cut is where it is. IN SCOPE Stateless transforms + stateful lookups via SQL. WHERE / filter expressions arbitrary predicates Projection & computed columns SELECT, CAST, computed cols Scalar UDFs native, WASM, and HTTP UNION ALL, UNNEST, subqueries standard relational shapes Dynamic-table lookups in-memory or Postgres-backed At-least-once + idempotent sinks _gs_op upsert column OUT OF SCOPE Anything that needs distributed shuffle or per-key state. Streaming JOINs no shuffle, no hash partitioning Streaming aggregations SUM / COUNT / AVG over windows Windowing tumbling, sliding, session Exactly-once delivery we lean on idempotent sinks instead Every "out of scope" item needs either distributed shuffle or per-key state. Both are deliberate non-goals for Streamling today. If you need them — use Flink.
10. DataFusion: the parts we use We extend its extension points. We don't fight it. SQL parser DataFusion Logical planner DataFusion Physical planner DataFusion (+ our ExtensionPlanner) Optimizer rules DataFusion (+ our 3 streaming rules) Vectorized executor DataFusion — pulls RecordBatches THE FOUR EXTENSION POINTS WE TOUCH TableProvider sources and sinks become tables — Kafka, Postgres, ClickHouse… ExecutionPlan custom physical operators — Streaming{Filter,Projection,Unnest}, MultiSink, scan-sharing… ExtensionPlanner teach the planner to lower our nodes — one per extension operator Physical optimizer rules rewrite plans post-planning — we have exactly three Streaming{Filter,Projection,Unnest}
11. A streaming source is a TableProvider that doesn't end The whole trick — if you remember one thing from this talk, this is it. TWO IMPLS. THAT'S IT. Try it yourself Working 150-line toy implementation:
12. Same plan, only the source changes A streaming query plan is a batch plan with an unbounded bottom node. BATCH STREAMING Sink ParquetSink Projection SELECT a, b Filter WHERE … TableScan parquet (Bounded) Sink KafkaSink Projection SELECT a, b Filter WHERE … TableScan kafka (Unbounded) SAME plan shape same SQL same executor Only the bottom node differs. Every operator above the scan — Filter, Projection, UDFs — is stock DataFusion. The streaming-ness lives at the source. RecordBatch is the one currency throughout.
13. Checkpoints Why we need them. Why they ride inside the data. WHY At-least-once across in-flight buffers. Checkpoints track source progress. Until data is durable downstream, the source can't commit offsets. PROTOCOL Marker(N) Coordinator → Sources through plan rides in Schema::metadata Ack(N) Sinks → Coordinator (after flush) Finalizer(N) Coordinator → All commit Sources commit offsets
14. Three paths into the engine One Arrow boundary. Three places to hook in. SANDBOXED WebAssembly via Extism JS / TS js_transform: type: script script: > function process(input) { input.num_trades = 100; return input; } Arrow IPC across the boundary No host APIs — no fs, no net FIT Per-pipeline custom JS/TS without forking the engine. EXTERNAL HTTP handlers via ExternalHandlerExec POST batches to a URL JSON payload Retries with backoff Concurrent across batches CDC envelope w/ op metadata FIT Call your existing service. Enrichment, side calls. NATIVE Plugins via Rust dylibs Source / Transform / Sink / UDF / Preprocessor / SideOutput Zero-copy of data buffers Auto-managed by the runtime Plugins can participate in checkpoints Easy to one-shot with AI (e.g. EventBridge) FIT First-class engine components. Max performance.
15. Live inspect, by default See the data, not just the shape. No edit, no restart, no savepoint. Status: ACTIVE All (60) │ test_transform (30) │ test_source (30) Sampled Live Data — All (60/60 records) { "id": "0xea2a41…_423797", "owner_address": "0xbf6554783aed…", "token_id": "423797", "token_type": "ERC_721", "block_number": 28391735, "balance": "1", "_gs_op": "i" } { "id": "0xea2a41…_429554", "owner_address": "0xbf6554783aed…", "token_id": "429554", "token_type": "ERC_721", "block_number": 28391735, "balance": "1", "_gs_op": "i" } Last record: 1s ago │ q=quit, Tab=tabs, j/k=scroll, /=search, w=web HOW IT WORKS turbo inspect my-pipeline Every operator wraps in WrappingExec. Minimal fixed cost 15 rows / 30s / node. Plus OpenTelemetry metrics from the same WrappingExec hook — throughput, batch sizes, epoch progress.
16. Recap Part 1 in five lines. Then over to Part 2. 01 A specific point in the design space Lighter shape for workloads that don't need distributed stateful processing. 02 DataFusion + Arrow as the substrate A streaming source is a TableProvider that doesn't end. Everything above is stock. 03 Checkpoints At least once semantics via source progress tracking. 04 Three extension paths WebAssembly for sandboxed transforms. HTTP for service calls. Native plugins for first-class components. 05 Observability by default Metrics from every operator. Live inspect into any source or transform. Now: from engine to platform → Part 2
17. PART 2 From Engine to Platform Production Cloud Deployment How to turn YAML into Production Ready Streaming Pipeline Goldsky Turbo
19. Control Plane
21. From Topology to Production Pipeline Pipeline = ( Topology, Runtime Metadata) Tenant = Project name job_mode resource_size config_override streamling-cloud (streamling-agent) project-pipeline Naming Convention Deployment | Job ConfigMap Secret
22. Pipeline Lifecycle CLI apply pause resume delete
24. Audit Database
26. State Processing API  SQL on States DB Operation Simplicity State = ( namespace, key, data, created_at) Very handy when debugging checkpointed related issue! • SQL Query: blast radius evaluation • SQL Update: incident mitigation namespace cmd4sefiwn1kt01vyblreh9ck-polymarket-log-decode key matic_raw_logs__1_0_0__3deujzi:matic.raw.logs:42 data { "offset": 653099710, "updated_at": 1780840551991} createad_at 2026-05-20T00:44:26.263769+00:00
27. Recap 01 Keep it simple. Use naming Conventions & Leverage k8s as much as possible. 02 Record wide request events to derive new features. 03 State DB make operations simple.
28. Contributors
29. Streamling Runtime Open Source Today! goldsky-io/streamling ⭐ Star the repo 🚀 Clone it and Play 🛠 Contribute / Raise Issues Functional Source License (FSL-1.1-ALv2)
30. Questions?