Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Predictive Maintenance Pipeline using Kafka Connect, Streams and KSQL

Predictive Maintenance Pipeline using Kafka Connect, Streams and KSQL

How the data team at Singapore Airlines use Kafka Connect, Kafka Streams and KSQL to build a predictive maintenance pipeline.

8ac3c419308bae98885f5cd89ee492a0?s=128

Victor Wibisono

November 01, 2018
Tweet

More Decks by Victor Wibisono

Other Decks in Programming

Transcript

  1. KSQL and Kafka Streams for Predictive Maintenance Pipeline

  2. data engineer at

  3. at

  4. started looking at Kafka in April

  5. at

  6. Predictive Maintenance

  7. None
  8. None
  9. None
  10. Component A - Health Last serviced: 1 October 2018 30%

    B77W 9V-123
  11. aircraft sensors file system source connector ksql kafka streams app

    ML model web app
  12. None
  13. aircraft sensors file system source connector ksql kafka streams app

    ML model web app 1 2
  14. Source Connector custom connector written in Java

  15. KSQL filtering based on plane types, tail number, etc.

  16. Kafka Streams basic transformation, calls Machine Learning job

  17. Why Kafka?

  18. decoupling

  19. aircraft sensors file system source connector ksql kafka streams app

    ML model web app Java Scala C# Python
  20. decoupling

  21. aircraft sensors file system source connector ksql kafka streams app

    ML model web app ksql kafka streams app ML model web app ksql kafka streams app ML model web app one pipeline for each aircraft component
  22. Why Kafka?

  23. offset management

  24. Why Kafka Streams?

  25. fault-tolerant

  26. None
  27. Why Kafka Streams?

  28. scalability

  29. aircraft sensors file system source connector ksql kafka streams app

    ML model web app 1 2
  30. aircraft sensors file system source connector ksql kafka streams app

    ML model web app 1 2 kafka streams app kafka streams app
  31. Source: https://kafka.apache.org/0102/documentation/streams/architecture#streams_architecture_threads

  32. Why Kafka Streams?

  33. easy-to-use APIs

  34. val builder: StreamsBuilder = new StreamsBuilder val inputLines: KStream[String, String]

    = builder.stream[String, String](inputTopic) val outputLines: KStream[String, SomeObject] = inputLines.map[String, SomeObject]((_: String,value: String) => { // perform some ETL (s"$filename", // build resulting object) }) outputLines.to(s"$outputTopic") val streams: KafkaStreams = new KafkaStreams(builder.build(), config) streams.start()
  35. create stream 2018_sensor_stream with (kafka_topic=‘2018_sensor_stream’, value_format='avro') as select plane, tail,

    flight, year, month, date from sensor_stream where year >= 2018;
  36. Learning

  37. None
  38. None
  39. Summary

  40. Predictive Maintenance • Using Machine Learning model to determine health

    of critical aircraft components.
  41. Why Kafka? • Decoupled architecture. • Offset management.

  42. Why Kafka Streams and KSQL? • Fault tolerant. • Scalable

    • Easy-to-use APIs.
  43. Thank You

  44. We are hiring Data Engineers! send your resumes to: victor_wibisono@singaporeair.com.sg