Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Stockholm Apache Kafka Meetup Jan 2021

Stockholm Apache Kafka Meetup Jan 2021

Loïc DIVAD

January 26, 2021
Tweet

More Decks by Loïc DIVAD

Other Decks in Technology

Transcript

  1. @loicmdivad SAK Meetup @loicmdivad SAK Meetup 8 1. The Use

    Case 2. The Implementation 3. The what’s next for 2021? Photo by Aron Visuals on Unsplash
  2. @loicmdivad SAK Meetup @loicmdivad SAK Meetup 9 1. The Use

    Case a. Context and motivation b. Emit a final result from a time window c. Kafka Tutorials presentation 2. The Implementation a. Develop b. Test c. Deploy 3. The what’s next for 2021? a. New year resolution at massive scale! Photo by Aron Visuals on Unsplash
  3. @loicmdivad SAK Meetup Window Final Result for real-time alerting Let's

    be an automated drink order service: - Captors are sending pressure alerts 11
  4. @loicmdivad SAK Meetup Window Final Result for real-time alerting Let's

    be an automated drink order service: - Captors are sending pressure alerts - The pressure can be high - But these are just warnings - However, we don’t want them to be too frequents - It could indicate a damaged captors 12
  5. @loicmdivad SAK Meetup Window Final Result for real-time alerting Let's

    be an automated drink order service: - Captors are sending pressure alerts - The pressure can be high - But these are just warnings - However, we don’t want them to be too frequents - It could indicate a damaged captors - Instead of maintaining the overall values, we’d like to emit the final count - This can be done by aggregating the unbounded stream of pressure alerts 13
  6. @loicmdivad SAK Meetup Window Final Result for real-time alerting Let's

    be an automated drink order service: - Captors are sending pressure alerts - The pressure can be high - But these are just warnings - However, we don’t want them to be too frequents - It could indicate a damaged captors - Instead of maintaining the overall values, we’d like to emit the final count - This can be done by aggregating the unbounded stream of pressure alerts 14
  7. @loicmdivad SAK Meetup What if the captor were laggy ??

    19 In this exercise we will probably face out-of-order elements and late data arrivals. We have to take this in account! The solution could be to add a extra period when late events can still join there corresponding aggregate and trigger a event. It’s called a grace period!
  8. @loicmdivad SAK Meetup The streaming app solving this has to:

    • Continually consume and process events • Aggregate events over time • Correctly integrate late events • Emit an event at the closing of each window • Be easy to test and deploy 22
  9. @loicmdivad SAK Meetup Kafka Tutorials Kafka Tutorials is a collection

    of common event streaming use cases, with each tutorial featuring an example scenario and several complete code solutions. It’s the fastest way to learn how to use Kafka with confidence. -- Michael Drogalis 24 from: https://www.confluent.io/blog/announci ng-apache-kafka-tutorials/
  10. @loicmdivad SAK Meetup WFR applied to real-time alerting • Step

    N°1: find an issue you’d like to tackle 25
  11. @loicmdivad SAK Meetup WFR applied to real-time alerting • Step

    N°1: find an issue you’d like to tackle • Step N°2: submit a PR referencing the previous issue 26
  12. @loicmdivad SAK Meetup WFR applied to real-time alerting • Step

    N°1: find an issue you’d like to tackle • Step N°2: submit a PR referencing the previous issue • Step N°3: Negotiate during the review to get your dirty code accepted! 27
  13. @loicmdivad SAK Meetup @loicmdivad SAK Meetup 29 1. The Use

    Case a. Context and motivation b. Emit a final result from a time window c. Kafka Tutorials presentation 2. The Implementation a. Develop b. Test c. Deploy 3. The what’s next for 2021? a. New year resolution at massive scale! Photo by Aron Visuals on Unsplash
  14. @loicmdivad SAK Meetup Application Build 34 Plugins: java, application, jib,

    shadow, avro dependencies: - kafka-streams - kafka-streams-avro-serde - kafka-streams-test-utils - junit source / target compatibility: 1.8
  15. @loicmdivad SAK Meetup 36 1. Simple 2. Expressif 3. Declarative

    The Kafka Streams DSL is built on top of the Streams Processor API. It is the recommended for most users… Most data processing operations can be expressed in just a few lines of DSL code.
  16. @loicmdivad SAK Meetup 38 value key headers Kafka Message in

    value key headers Kafka Message out Null Timestamp
  17. @loicmdivad SAK Meetup 49 Time window specification 1. • Return

    a window definition with the given window size 2. • specifies by how much a window moves forward relative to the previous one 3. • Reject late events that arrive after the specified delay of advanceBy grace
  18. @loicmdivad SAK Meetup Topology should group over date-time windows 60

    Kafka Streams Topology 101/[T1-T2] N1 101/[T3-T4] N2 101/[T4-T5] N3 ⚠ ⚠ ⚠
  19. @loicmdivad SAK Meetup Topology should group by Id 63 Kafka

    Streams Topology 101 / [T1-T2] N1 102 / [T1-T2] N2 103 / [T1-T2] N3 ⚠ ⚠ ⚠
  20. @loicmdivad SAK Meetup PDE should extract Timestamp 66 Kafka Streams

    Topology 101/[T1-T2] TS1 101/[T3-T4] TS2 101/[T4-T5] TS3 ⚠ ⚠ ⚠
  21. @loicmdivad SAK Meetup Package as you are 70 • Build

    a jar with all the dependencies and deploy it on machine having a JVM and where all the the configs files are automatically provided Or • Build a container, deploy the container