Quickly Build Kafka Stream ETL System with KSETL

Namil Kim
LINE Plus / Messaging Data Engineering Dept / Engineering Manager

https://linedevday.linecorp.com/2021/ja/sessions/73
https://linedevday.linecorp.com/2021/en/sessions/73
https://linedevday.linecorp.com/2021/ko/sessions/73

Transcript

2. KSETL • KSETL = Kafka Stream ETL – Extract, Transform, Load stream on Kafka • Stream – Unbounded continuously generated data – User log – Sensor data
3. Why Stream Processing? • To process data with low latency • For business requirements – Delivery status, order status for food delivery – Fraud detection in financial transaction • For better performance – Contents recommendation
4. Stream processing system • Batch processing system is not enough – Daily à Hourly à Minuitely à Secondly? • Stream processing system – Reflect the latest data quickly
5. Building systems • Many stream processing systems are needed • Common works – Write and debug programs – Build programs – Deploy programs – Monitor programs • Many data engineers do the similar works
6. KSETL • KSETL – Kafka Stream ETL for LINE – Input and output are Kafka topics – (Kafka is widely used for streams in LINE) • Build stream processing systems easily – Let data engineers build their systems by themselves
7. Goal – Easiness • Express ETL logic easily – Introduce SQL-like syntax • Build systems easily – Create ksqlDB clusters dynamically on k8s
8. Express ETL logic easily • For data engineers without programming expertise – Every data engineer knows SQL • Prototyped using various SQL engines – ksqlDB, FlinkSQL, Spark structured streaming – Left join stream-stream – Query a table and write to Kafka topic
9. ksqlDB • Full features – Join – Window aggregation – User Defined Function (UDF) • Based on Kafka Streams API – Easy to understand • Only for stream processing – No extra parts for batch processing • Everything on Kafka – Good Kafka team in LINE
10. Internals of ksqlDB Join p1 p2 Left topic p1 p2 Right topic p1 p2 Joined topic p1 p1 Join partition1 p2 p2 Join partition2 p1 p2 Left changelog topic Local state store Local state store p1 p2 Right changelog topic
11. Build systems easily • Create ksqlDB clusters dynamically – ODA (On-Demand Applications) • Provide logging/monitoring facilities • Run queries against a ksqlDB cluster
12. KSETL ODA Architecture • Many ksqlDB clusters in a k8s
13. KSETL Logging/Monitoring
14. Summary Tradi&onal KSETL Language Java, Scala SQL Build Compile Interactive shell Deploy CI/CD tools On-demand cluster Monitor Custom tools Prebuilt dashboards
15. Example system • AB test report – LINE runs AB tests before releasing new features – Request logs from LINE server • 50k / sec logs at peak time – Event(impression, click) logs from LINE client – Find client reaction for request and aggregate – Stream join and windowed aggregation required
16. Prev. AB test report • Prev. system to join streams – Event log and request log with the same key – Store event(impression,click) logs to Redis – Delay request logs and lookup Redis to implement join window
17. Join
18. Window aggregation
19. Results • Simple architecture – No Redis to join two streams • Fast release – Interactive development • Fast monitoring and update – Get a performance dashboard – Tune fast
20. Limits • KSETL depends on – ksqlDB – Company-wide Kafka • ksqlDB – Some features are missed (Still in active development) – FlinkSQL may be an alternative • Company-wide Kafka – Good support for all Kafka in LINE – But dynamic topic creation is prohibited
21. Future works • Data import from Hive – Hive tables to enrich Kafka topics • Enhancing query deployment – Better way for executing query scripts
22. Thank you