Slide 18
Slide 18 text
Workload
Compiler/Optimizer
Deployment
Relational
Expression
@transformation
def transaction_count(tx: Transactions, wspec: WindowSpec):
return tx[tx.status == "failed"].groupby("account_id").window(wspec).count()
Same code can run on different computation engines
Compile into a relational expression (RE), which
is SQL equivalent
Intermediate
Representation
Compile & optimize RE into the computation
engine
(e.g., Panda, DuckDb, Flink, Spark) best suited for
the job
Spin up and manage computation jobs