$30 off During Our Annual Pro Sale. View Details »

Predictive Maintenance Pipeline using Kafka Connect, Streams and KSQL

Predictive Maintenance Pipeline using Kafka Connect, Streams and KSQL

How the data team at Singapore Airlines use Kafka Connect, Kafka Streams and KSQL to build a predictive maintenance pipeline.

Victor Wibisono

November 01, 2018
Tweet

More Decks by Victor Wibisono

Other Decks in Programming

Transcript

  1. KSQL and Kafka Streams for
    Predictive Maintenance Pipeline

    View Slide

  2. data engineer
    at

    View Slide

  3. at

    View Slide

  4. started looking at Kafka in April

    View Slide

  5. at

    View Slide

  6. Predictive Maintenance

    View Slide

  7. View Slide

  8. View Slide

  9. View Slide

  10. Component A - Health
    Last serviced: 1 October 2018
    30%
    B77W 9V-123

    View Slide

  11. aircraft
    sensors
    file system
    source
    connector
    ksql
    kafka
    streams app
    ML model web app

    View Slide

  12. View Slide

  13. aircraft
    sensors
    file system
    source
    connector
    ksql
    kafka
    streams app
    ML model web app
    1
    2

    View Slide

  14. Source Connector
    custom connector written in Java

    View Slide

  15. KSQL
    filtering based on plane types, tail number, etc.

    View Slide

  16. Kafka Streams
    basic transformation, calls Machine Learning job

    View Slide

  17. Why Kafka?

    View Slide

  18. decoupling

    View Slide

  19. aircraft
    sensors
    file system
    source
    connector
    ksql
    kafka
    streams app
    ML model web app
    Java
    Scala C#
    Python

    View Slide

  20. decoupling

    View Slide

  21. aircraft
    sensors
    file system
    source
    connector
    ksql
    kafka
    streams app
    ML model web app
    ksql
    kafka
    streams app
    ML model web app
    ksql
    kafka
    streams app
    ML model web app
    one pipeline for each aircraft component

    View Slide

  22. Why Kafka?

    View Slide

  23. offset management

    View Slide

  24. Why Kafka Streams?

    View Slide

  25. fault-tolerant

    View Slide

  26. View Slide

  27. Why Kafka Streams?

    View Slide

  28. scalability

    View Slide

  29. aircraft
    sensors
    file system
    source
    connector
    ksql
    kafka
    streams app
    ML model web app
    1
    2

    View Slide

  30. aircraft
    sensors
    file system
    source
    connector
    ksql
    kafka
    streams app
    ML model web app
    1
    2
    kafka
    streams app
    kafka
    streams app

    View Slide

  31. Source: https://kafka.apache.org/0102/documentation/streams/architecture#streams_architecture_threads

    View Slide

  32. Why Kafka Streams?

    View Slide

  33. easy-to-use APIs

    View Slide

  34. val builder: StreamsBuilder = new StreamsBuilder
    val inputLines: KStream[String, String] = builder.stream[String, String](inputTopic)
    val outputLines: KStream[String, SomeObject] =
    inputLines.map[String, SomeObject]((_: String,value: String) => {
    // perform some ETL
    (s"$filename", // build resulting object)
    })
    outputLines.to(s"$outputTopic")
    val streams: KafkaStreams = new KafkaStreams(builder.build(), config)
    streams.start()

    View Slide

  35. create stream 2018_sensor_stream
    with (kafka_topic=‘2018_sensor_stream’,
    value_format='avro') as
    select plane, tail, flight, year, month,
    date from sensor_stream where year >= 2018;

    View Slide

  36. Learning

    View Slide

  37. View Slide

  38. View Slide

  39. Summary

    View Slide

  40. Predictive Maintenance
    • Using Machine Learning model to determine health
    of critical aircraft components.

    View Slide

  41. Why Kafka?
    • Decoupled architecture.
    • Offset management.

    View Slide

  42. Why Kafka Streams and KSQL?
    • Fault tolerant.
    • Scalable
    • Easy-to-use APIs.

    View Slide

  43. Thank You

    View Slide

  44. We are hiring Data Engineers!
    send your resumes to:
    [email protected]

    View Slide