Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

Predictive Maintenance Pipeline using Kafka Con...

Predictive Maintenance Pipeline using Kafka Connect, Streams and KSQL

How the data team at Singapore Airlines use Kafka Connect, Kafka Streams and KSQL to build a predictive maintenance pipeline.

Victor Wibisono

November 01, 2018
Tweet

More Decks by Victor Wibisono

Other Decks in Programming

Transcript

  1. at

  2. at

  3. aircraft sensors file system source connector ksql kafka streams app

    ML model web app ksql kafka streams app ML model web app ksql kafka streams app ML model web app one pipeline for each aircraft component
  4. aircraft sensors file system source connector ksql kafka streams app

    ML model web app 1 2 kafka streams app kafka streams app
  5. val builder: StreamsBuilder = new StreamsBuilder val inputLines: KStream[String, String]

    = builder.stream[String, String](inputTopic) val outputLines: KStream[String, SomeObject] = inputLines.map[String, SomeObject]((_: String,value: String) => { // perform some ETL (s"$filename", // build resulting object) }) outputLines.to(s"$outputTopic") val streams: KafkaStreams = new KafkaStreams(builder.build(), config) streams.start()