Predictive Maintenance Pipeline using Kafka Connect, Streams and KSQL

How the data team at Singapore Airlines use Kafka Connect, Kafka Streams and KSQL to build a predictive maintenance pipeline.

Transcript

1. KSQL and Kafka Streams for
   Predictive Maintenance Pipeline
   

   View Slide

2. data engineer
   at
   

   View Slide

3. at
   

   View Slide

4. started looking at Kafka in April
   

   View Slide

5. at
   

   View Slide

6. Predictive Maintenance
   

   View Slide

7. View Slide

8. View Slide

9. View Slide

10. Component A - Health
    Last serviced: 1 October 2018
    30%
    B77W 9V-123
    

    View Slide

11. aircraft
    sensors
    file system
    source
    connector
    ksql
    kafka
    streams app
    ML model web app
    

    View Slide

12. View Slide

13. aircraft
    sensors
    file system
    source
    connector
    ksql
    kafka
    streams app
    ML model web app
    1
    2
    

    View Slide

14. Source Connector
    custom connector written in Java
    

    View Slide

15. KSQL
    filtering based on plane types, tail number, etc.
    

    View Slide

16. Kafka Streams
    basic transformation, calls Machine Learning job
    

    View Slide

17. Why Kafka?
    

    View Slide

18. decoupling
    

    View Slide

19. aircraft
    sensors
    file system
    source
    connector
    ksql
    kafka
    streams app
    ML model web app
    Java
    Scala C#
    Python
    

    View Slide

20. decoupling
    

    View Slide

21. aircraft
    sensors
    file system
    source
    connector
    ksql
    kafka
    streams app
    ML model web app
    ksql
    kafka
    streams app
    ML model web app
    ksql
    kafka
    streams app
    ML model web app
    one pipeline for each aircraft component
    

    View Slide

22. Why Kafka?
    

    View Slide

23. offset management
    

    View Slide

24. Why Kafka Streams?
    

    View Slide

25. fault-tolerant
    

    View Slide

26. View Slide

27. Why Kafka Streams?
    

    View Slide

28. scalability
    

    View Slide

29. aircraft
    sensors
    file system
    source
    connector
    ksql
    kafka
    streams app
    ML model web app
    1
    2
    

    View Slide

30. aircraft
    sensors
    file system
    source
    connector
    ksql
    kafka
    streams app
    ML model web app
    1
    2
    kafka
    streams app
    kafka
    streams app
    

    View Slide

31. Source: https://kafka.apache.org/0102/documentation/streams/architecture#streams_architecture_threads
    

    View Slide

32. Why Kafka Streams?
    

    View Slide

33. easy-to-use APIs
    

    View Slide

34. val builder: StreamsBuilder = new StreamsBuilder
    val inputLines: KStream[String, String] = builder.stream[String, String](inputTopic)
    val outputLines: KStream[String, SomeObject] =
    inputLines.map[String, SomeObject]((_: String,value: String) => {
    // perform some ETL
    (s"$filename", // build resulting object)
    })
    outputLines.to(s"$outputTopic")
    val streams: KafkaStreams = new KafkaStreams(builder.build(), config)
    streams.start()
    

    View Slide

35. create stream 2018_sensor_stream
    with (kafka_topic=‘2018_sensor_stream’,
    value_format='avro') as
    select plane, tail, flight, year, month,
    date from sensor_stream where year >= 2018;
    

    View Slide

36. Learning
    

    View Slide

37. View Slide

38. View Slide

39. Summary
    

    View Slide

40. Predictive Maintenance
    • Using Machine Learning model to determine health
    of critical aircraft components.
    

    View Slide

41. Why Kafka?
    • Decoupled architecture.
    • Offset management.
    

    View Slide

42. Why Kafka Streams and KSQL?
    • Fault tolerant.
    • Scalable
    • Easy-to-use APIs.
    

    View Slide

43. Thank You
    

    View Slide

44. We are hiring Data Engineers!
    send your resumes to:
    victor_wibisono@singaporeair.com.sg
    

    View Slide