Predictive Maintenance Pipeline using Kafka Connect, Streams and KSQL

How the data team at Singapore Airlines use Kafka Connect, Kafka Streams and KSQL to build a predictive maintenance pipeline.

Transcript

1. KSQL and Kafka Streams for Predictive Maintenance Pipeline
2. data engineer at
3. at
4. started looking at Kafka in April
5. at
6. Predictive Maintenance
10. Component A - Health Last serviced: 1 October 2018 30% B77W 9V-123
11. aircraft sensors file system source connector ksql kafka streams app ML model web app
13. aircraft sensors file system source connector ksql kafka streams app ML model web app 1 2
14. Source Connector custom connector written in Java
15. KSQL filtering based on plane types, tail number, etc.
16. Kafka Streams basic transformation, calls Machine Learning job
17. Why Kafka?
18. decoupling
19. aircraft sensors file system source connector ksql kafka streams app ML model web app Java Scala C# Python
20. decoupling
21. aircraft sensors file system source connector ksql kafka streams app ML model web app ksql kafka streams app ML model web app ksql kafka streams app ML model web app one pipeline for each aircraft component
22. Why Kafka?
23. offset management
24. Why Kafka Streams?
25. fault-tolerant
27. Why Kafka Streams?
28. scalability
29. aircraft sensors file system source connector ksql kafka streams app ML model web app 1 2
30. aircraft sensors file system source connector ksql kafka streams app ML model web app 1 2 kafka streams app kafka streams app
31. Source: https://kafka.apache.org/0102/documentation/streams/architecture#streams_architecture_threads
32. Why Kafka Streams?
33. easy-to-use APIs
34. val builder: StreamsBuilder = new StreamsBuilder val inputLines: KStream[String, String] = builder.stream[String, String](inputTopic) val outputLines: KStream[String, SomeObject] = inputLines.map[String, SomeObject]((_: String,value: String) => { // perform some ETL (s"$filename", // build resulting object) }) outputLines.to(s"$outputTopic") val streams: KafkaStreams = new KafkaStreams(builder.build(), config) streams.start()
35. create stream 2018_sensor_stream with (kafka_topic=‘2018_sensor_stream’, value_format='avro') as select plane, tail, flight, year, month, date from sensor_stream where year >= 2018;
36. Learning
39. Summary
40. Predictive Maintenance • Using Machine Learning model to determine health of critical aircraft components.
41. Why Kafka? • Decoupled architecture. • Offset management.
42. Why Kafka Streams and KSQL? • Fault tolerant. • Scalable • Easy-to-use APIs.
43. Thank You
44. We are hiring Data Engineers! send your resumes to: victor_wibisono@singaporeair.com.sg