Deeply Declarative Data Pipelines (Ryanne Dolan, LinkedIn) | RTA Summit 2023

With Flink and Kubernetes, it’s possible to deploy stream processing jobs with just SQL and YAML. This low-code approach can certainly save a lot of development time. However, there is more to data pipelines than just streaming SQL. We must wire up many different systems, thread through schemas, and, worst-of-all, write a lot of configuration.

In this talk, we’ll explore just how “declarative” we can make streaming data pipelines on Kubernetes. I’ll show how we can go deeper by adding more and more operators to the stack. How deep can we go?

Transcript

1. Deeply Declarative Data Pipelines Ryanne Dolan Snr Staff SWE, LinkedIn
2. What are data pipelines? • transfer data from sources to sinks • replicating (e.g. MirrorMaker) • filtering, routing, joining streams (e.g. Flink) • ETL, rETL (e.g. Kafka Connect, Gobblin) • CDC (e.g. Debezium, Brooklin) • includes batch, but we'll focus on streaming
3. Deeply Declarative? Explicit Concrete Imperative General-purpose Internally simple Hard Implicit Abstract Declarative Domain-specific Internally complex Easy ASM SQL YAML Java
4. 1. Preheat the oven to 450° 2. Mix flour and water ...
5. When do we need data pipelines? online operational database analytics database data pipeline
6. Data pipelines as connectors MySQL HDFS Kafka MySQL source connector (CDC) HDFS sink connector (ETL) Kafka Connect
7. Connectors as configuration { "name": "inventory-connector", "config": { "connector.class": "io.debezium.connector.mysql.MySqlConnector", "tasks.max": "1", "database.hostname": "mysql", "database.port": "3306", "database.user": "debezium", "database.password": "dbz", "database.server.id": "184054", "database.server.name": "dbserver1", "database.include.list": "inventory", "database.history.kafka.bootstrap.servers": "kafka:9092", "database.history.kafka.topic": "schema-changes.inventory" } }
8. Simple transformation pipelines MySQL HDFS Kafka MySQL source connector (CDC) HDFS sink connector (ETL) Filter (SMT) Flatten (SMT)
9. Transformations as configuration { "transforms": "Filter,Flatten", "transforms.Filter.type": "org.apache.kafka.connect.transforms.Filter", "transforms.Filter.predicate": "IsMuj", "transforms.Flatten.type": "org.apache.kafka.connect.transforms.Flatten$Value", "transforms.Flatten.delimiter": "_", "predicates": "IsMuj", "predicates.IsMuj.type": "io.confluent.connect.transforms.predicates.FieldValuePredicate", "predicates.IsMuj.field": "$.name", "predicates.IsMuj.value": "\"muj\"", ... }
10. Data pipelines as configuration MySQL HDFS Kafka MySQL source connector (CDC) HDFS sink connector (ETL) Filter (SMT) Flatten (SMT) JSON JSON
11. Complex data pipelines online operational database event stream analytics database filter and join change stream view
12. Apache Kafka Connect and Kafka Streams online operational database Kafka analytics database Kafka Streams app Kafka Kafka MySQL source connector (CDC) Filter (SMT)
13. Apache Flink DB event stream DB Flink job change stream upserts
14. Data pipelines power important stuff advertiser accounts ad click events very important data who do we charge? how much do we charge? new accounts money stuff
15. How can we make data pipelines easy?
16. How can we build new data pipelines with as little one-off code as possible?
17. Stream processing with Apache Flink DB event stream DB Flink change stream upserts
18. Stream processing with the Flink Datastream API // Join the two datastreams on the ad id field using a time window of one hour and a custom join function DataStream<Tuple2<AdvertiserAccountCDC, AdClickEvent>> joined = accounts .join(clicks) .where(account -> account.id) // Key selector for accounts .equalTo(click -> click.adId) // Key selector for clicks .window(TumblingEventTimeWindows.of(Time.hours(1))) // Window assigner for joining streams based on event time .apply(new JoinFunction<AdvertiserAccountCDC, AdClickEvent, Tuple2<AdvertiserAccountCDC, AdClickEvent>>() { @Override public Tuple2<AdvertiserAccountCDC, AdClickEvent> join(AdvertiserAccountCDC account, AdClickEvent click) { return new Tuple2<>(account, click); // Return a tuple of account and click } });
19. Flink Table API introduces connectors TableFactoryService.find(StreamTableSourceFactory.class, new HashMap<>()) .connector(new DebeziumDynamicTableSourceFactory()) // use Debezium as the connector .option("connector.hostname", "localhost") .option("connector.port", "3306") .option("connector.username", "flinkuser") .option("connector.password", "flinkpw") .option("connector.database-name", "inventory") // monitor all tables under inventory database .format(new JsonFormatFactory()) // use Json format factory .option("json-schema", "...") // specify the JSON schema of the records .option("json-fail-on-missing-field", true) // fail if a field is missing .inAppendMode() .schema(...) // specify the table schema of the source .createTemporaryTable("cdc_source"); // register the source as a temporary table // create a table from the source descriptor Table cdcTable = tableEnv.from("cdc_source"); // filter out delete operations from the table Table filteredTable = cdcTable.where("!op.equals('d')"); // keep only insert and update operations // print the filtered table filteredTable.execute().print();
20. Flink SQL API -- create a CDC source table for inventory database CREATE TABLE inventory ( id INT NOT NULL, name STRING, description STRING, amount INT ) WITH ( 'connector' = 'mysql-cdc', 'hostname' = 'localhost', 'port' = '3306', 'username' = 'flinkuser', 'password' = 'flinkpw', 'database-name' = 'inventory_db', 'table-name' = 'inventory' ); -- filter out delete operations from the table SELECT * FROM inventory WHERE op != "d"; Data Definition Language Standard Query Language
21. Flink SQL for data pipelines INSERT INTO stock SELECT * FROM inventory WHERE quantity > 0; sink source transformation
22. Flink SQL API // create a TableEnvironment for specific planner batch or streaming TableEnvironment tableEnv = ...; // see planners section // register a MySQL table named "products" in Flink SQL String sourceDDL = "CREATE TABLE products (" + " id INT NOT NULL," + " name STRING," + " description STRING," + " weight DECIMAL(10,3)" + ") WITH (" + " 'connector' = 'mysql-cdc'," + " 'hostname' = 'localhost'," + " 'port' = '3306'," + " 'username' = 'flinkuser'," + " 'password' = 'flinkpw'," + " 'database-name' = 'inventory'," + " 'table-name' = 'products'" + // monitor table `products` in database `inventory` ")"; tableEnv.executeSql(sourceDDL); // define a dynamic aggregating query String query = "SELECT id, name, SUM(weight) as total_weight FROM products GROUP BY id, name"; // convert the Table API query result into DataStream DataStream<Row> resultStream = tableEnv.<Row>toRetractStream(tableEnv.sqlQuery(query), Row.class) .filter(x -> x.f0) // only emit update changes (true) and filter out delete changes (false) .map(x -> x.f1); // drop change flag // print the result stream resultStream.print();
23. FlinkDeployment YAML with the Flink K8s Operator apiVersion: flink.apache.org/v1beta1 kind: FlinkDeployment metadata: name: basic-example spec: image: flink:1.16 flinkVersion: v1_16 flinkConfiguration: taskmanager.numberOfTaskSlots: "2" serviceAccount: flink jobManager: resource: memory: "2048m" cpu: 1 taskManager: resource: memory: "2048m" cpu: 1 job: jarURI: local:///opt/flink/examples/streaming/StateMachineExample.jar parallelism: 2 upgradeMode: stateless https://raw.githubusercontent.com/apache/flink-kubernetes-operator/release-1.4/examples/basic.yaml
24. What did a human need to do? • Write Java and compile a jar • Store the jar somewhere (e.g. in Docker image) • Write FlinkDeployment YAML • Deploy the YAML → Flink K8s operator steps in → Flink job running!
25. We can do better.
26. Level 1 Flink SQL in YAML • "Flink SQL Runner" application • FlinkSqlJob CRD • New k8s operator
27. Flink SQL Runner public class FlinkSqlRunner { public static void main(String[] args) { // check the arguments if (args.length != 1) { System.err.println("Usage: FlinkSqlRunner <sql-file>"); return; } // get the sql file name String sqlFile = args[0]; // create a streaming execution environment StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); // create a table environment with blink planner in streaming mode EnvironmentSettings settings = EnvironmentSettings.newInstance() .useBlinkPlanner() .inStreamingMode() .build(); TableEnvironment tableEnv = TableEnvironment.create(settings); // read the sql file line by line and execute each statement try (BufferedReader reader = new BufferedReader(new FileReader(sqlFile))) { String line; while ((line = reader.readLine()) != null) { // skip empty or comment lines if (line.trim().isEmpty() || line.startsWith("--")) { continue; } // execute the statement and print the result if any tableEnv.executeSql(line); } }
28. FlinkSQLJob Custom Resource apiVersion: rta/v1alpha1 kind: FlinkSQLJob metadata: name: itemcount spec: sql: | CREATE TABLE source ( user_id STRING, item_id STRING, category_id STRING, behavior STRING, ts TIMESTAMP(3) ) WITH ( 'connector' = 'kafka', 'topic' = 'user_behavior', 'properties.bootstrap.servers' = 'kafka:9092', 'format' = 'json' ); CREATE TABLE sink ( user_id STRING, item_count BIGINT ) WITH ( 'connector' = 'jdbc', 'url' = 'jdbc:mysql://mysql:3306/flink', 'table-name' = 'user_item_count', 'username' = 'root', 'password' = '' ); INSERT INTO sink SELECT user_id, COUNT(DISTINCT item_id) AS item_count FROM source WHERE behavior = 'buy' GROUP BY TUMBLE(ts, INTERVAL '1' HOUR), user_id; parallelism: 2
29. flink-sql-job-operator FlinkSQLJob (CR YAML) flink-sql-job- -operator SQL script (ConfigMap) FlinkDeployment (CR YAML) flink-kubernetes- -operator Kubernetes JobMgr deployment (native YAML) Job deployment (native YAML) etc...
30. Operators are made up of control loops controller 1 controller 2 controller 3 Kubernetes YAML YAML YAML
31. Deploying a FlinkSQLJob $ kubectl apply -f wordcount.yaml $ kubectl get flinksqljob wordcount -o yaml # Output: apiVersion: rta/v1beta1 kind: FlinkSQLJob metadata: ... spec: ... status: # The current state of the job. state: Running
32. What did a human need to do? • Write SQL in a YAML file • Deploy the YAML → Flink SQL Job operator steps in → Flink K8s operator steps in → Flink job running!
33. Where do we get this stuff? apiVersion: rta/v1alpha1 kind: FlinkSQLJob metadata: name: itemcounts spec: sql: | CREATE TABLE source ( user_id STRING, item_id STRING, category_id STRING, behavior STRING, ts TIMESTAMP(3) ) WITH ( 'connector' = 'kafka', 'topic' = 'user_behavior', 'properties.bootstrap.servers' = 'kafka:9092', 'format' = 'json' ); CREATE TABLE sink ( user_id STRING, item_count BIGINT ) WITH ( 'connector' = 'jdbc', 'url' = 'jdbc:mysql://mysql:3306/flink', 'table-name' = 'user_item_count', 'username' = 'root', 'password' = '' ); INSERT INTO sink SELECT user_id, COUNT(DISTINCT item_id) AS item_count FROM source WHERE behavior = 'buy' GROUP BY TUMBLE(ts, INTERVAL '1' HOUR), user_id; parallelism: 2
34. Level 2 Table YAML • FlinkTable CRD • smarter operator
35. FlinkTable Custom Resource apiVersion: rta/v1alpha1 kind: FlinkTable metadata: name: UserBehavior spec: ddl: | CREATE TABLE UserBehavior ( user_id STRING, item_id STRING, category_id STRING, behavior STRING, ts TIMESTAMP(3) ) WITH ( 'connector' = 'kafka', 'topic' = 'user_behavior', 'properties.bootstrap.servers' = 'kafka:9092', 'format' = 'json' ); apiVersion: rta/v1alpha1 kind: FlinkTable metadata: name: UserItemCounts spec: ddl: | CREATE TABLE UserItemCounts ( user_id STRING, item_count BIGINT ) WITH ( 'connector' = 'jdbc', 'url' = 'jdbc:mysql://mysql:3306/flink', 'table-name' = 'user_item_count', 'username' = 'root', 'password' = '' );
36. Reference FlinkTables in FlinkSQLJobs apiVersion: rta/v1alpha1 kind: FlinkSQLJob metadata: name: itemcounts spec: tables: - UserItemCounts - UserBehavior sql: | INSERT INTO UserItemCounts SELECT user_id, COUNT(DISTINCT item_id) AS item_count FROM UserBehavior WHERE behavior = 'buy' GROUP BY TUMBLE(ts, INTERVAL '1' HOUR), user_id;
37. flink-sql-job-operator FlinkSQLJob (CR YAML) flink-sql-job- -operator DDL+SQL script (ConfigMap) FlinkDeployment (CR YAML) flink-kubernetes- -operator FlinkTable (CR YAML) FlinkTable (CR YAML)
38. What happens when you change or delete a FlinkTable? FlinkSQLJob (CR YAML) flink-sql-job- -operator DDL+SQL script (ConfigMap) FlinkDeployment (CR YAML) flink-kubernetes- -operator FlinkTable (CR YAML) FlinkTable (CR YAML)
39. What happens when you delete a FlinkSQLJob? FlinkSQLJob (CR YAML) flink-sql-job- -operator DDL+SQL script (ConfigMap) FlinkDeployment (CR YAML) flink-kubernetes- -operator FlinkTable (CR YAML) FlinkTable (CR YAML) ownerReferences
40. What happens when you delete a FlinkSQLJob? FlinkSQLJob (CR YAML) flink-sql-job- -operator SQL script (ConfigMap) FlinkDeployment (CR YAML) flink-kubernetes- -operator FlinkTable (CR YAML) FlinkTable (CR YAML) ownerReferences ownerReferences
41. Level 3 Automatic Table Registration • smarter data plane • centralized catalog
42. Why do we need to write these by hand? apiVersion: rta/v1alpha1 kind: FlinkTable metadata: name: UserBehavior spec: ddl: | CREATE TABLE UserBehavior ( user_id STRING, item_id STRING, category_id STRING, behavior STRING, ts TIMESTAMP(3) ) WITH ( 'connector' = 'kafka', 'topic' = 'user_behavior', 'properties.bootstrap.servers' = 'kafka:9092', 'format' = 'json' ); apiVersion: rta/v1alpha1 kind: FlinkTable metadata: name: UserItemCounts spec: ddl: | CREATE TABLE UserItemCounts ( user_id STRING, item_count BIGINT ) WITH ( 'connector' = 'jdbc', 'url' = 'jdbc:mysql://mysql:3306/flink', 'table-name' = 'user_item_count', 'username' = 'root', 'password' = '' );
43. Why do we need to write these by hand? apiVersion: rta/v1alpha1 kind: FlinkTable metadata: name: ImportantMoneyStuff spec: ddl: | CREATE TABLE ImportantMoneyStuff ( account STRING, moneys BIGINT ) WITH ( 'connector' = 'pinot', ... ); Pinot Table ?
44. Metadata catalogs Pinot MySQL Kafka DataHub metadata Data Plane flink-sql-job- -operator FlinkSQLJob (CR YAML)
45. Reference tables in the catalog apiVersion: rta/v1alpha1 kind: FlinkSQLJob metadata: name: itemcounts spec: datasets: - urn:li:dataset:(urn:li:dataPlatform:pinot,project.dataset.UserItemCounts,PROD) - urn:li:dataset:(urn:li:dataPlatform:kafka,project.dataset.UserBehavior,PROD) sql: | INSERT INTO UserItemCounts SELECT user_id, COUNT(DISTINCT item_id) AS item_count FROM UserBehavior WHERE behavior = 'buy' GROUP BY TUMBLE(ts, INTERVAL '1' HOUR), user_id;
46. Level 4 Declarative infra • declarative data plane • even more operators
47. Data plane operators Pinot DataHub metadata flink-sql-job- -operator FlinkSQLJob (CR YAML) PinotTable (CR YAML) pinot-table- -operator Kafka KafkaTopic (CR YAML) kafka-topic- -operator
48. Data plane YAML apiVersion: rta/v1beta1 kind: KafkaTopic metadata: name: my-topic namespace: rta ownershipReferences: - name: kafka-cluster kind: KafkaCluster uid: abc123... - name: my-table kind: PinotTable uid: abc123... spec: clusterRef: name: kafka-cluster name: my-topic partitions: 3 apiVersion: rta/v1alpha1 kind: PinotTable metadata: name: my-table namespace: rta spec: tableType: REALTIME schemaRef: my-schema kafkaRef: name: my-topic
49. Data plane operators Pinot DataHub metadata flink-sql-job- -operator FlinkSQLJob (CR YAML) PinotTable (CR YAML) pinot-table- -operator Kafka KafkaTopic (CR YAML) kafka-topic- -operator
50. Level 5 End-to-End Ownership • pipeline operator • multiple owners
51. Pipeline YAML apiVersion: rta/v1beta1 kind: Pipeline metadata: name: my-pipeline namespace: rta spec: resources: - kind: KafkaTopic name: my-topic - kind: PinotTable name: my-table - kind: AvroSchema name: my-schema sql: | INSERT INTO my-topic ...
52. pipeline- -operator Pipeline operator FlinkSQLJob (CR YAML) flink-sql-job- -operator SQL script (ConfigMap) FlinkDeployment (CR YAML) flink-kubernetes- -operator KafkaTopic (CR YAML) PinotTable (CR YAML) Schema (CR YAML) Pipeline (CR YAML)
53. pipeline- -operator End-to-end ownership FlinkSQLJob (CR YAML) flink-sql-job- -operator SQL script (ConfigMap) FlinkDeployment (CR YAML) flink-kubernetes- -operator KafkaTopic (CR YAML) PinotTable (CR YAML) Schema (CR YAML) Pipeline (CR YAML)
54. pipeline- -operator Multiple owners FlinkSQLJob (CR YAML) flink-sql-job- -operator SQL script (ConfigMap) FlinkDeployment (CR YAML) flink-kubernetes- -operator KafkaTopic (CR YAML) PinotTable (CR YAML) Schema (CR YAML) Pipeline (CR YAML) Pipeline (CR YAML) Pipeline (CR YAML)
55. Thank you! @dolanRyanne in/RyanneDolan All code from Bing Chat All images from OpenAI Dall-E