Making Sense of Sensors with SQL Streaming @ Make IT 2019

Making Sense of Sensors with SQL Streaming @ Make IT 2019

We are living in a data streaming era, yet until recently it has been particularly hard to leverage existing stream processing technologies. On the one hand, because dealing with data in motion has its inherent challenges.

On the other hand, most frameworks and APIs which are allowing for stream processing are typically very hard to employ and/or operate - NOT so for KSQL, the newest kid in Apache Kafka's ecosystem. Based on a simplified version of an IoT use case this session gives a gentle introduction into KSQL - Kafka's SQL streaming engine for the masses.

Join this whirlwind tour during which we are discussing a fully-fledged streaming IoT architecture. Concretely, we are going to:

* ingest smart home energy data into Apache Kafka,
* use KSQL for flexible, powerful and scalable SQL-only stream processing,
* send raw data, as well as pre-processed results to an operational NoSQL data store,
* reactively, serve data to clients in near real-time and
* finally build informative live charts.

Note: None of the sessions were recorded here so no video link this time...

744f1c2c6cbea2ff5104b0ac512936bd?s=128

Hans-Peter Grahsl

October 14, 2019
Tweet

Transcript

  1. Making Sense of Sensors with SQL Streaming @hpgrahsl | Make

    IT Conference, 14th October 2019, Portorož, Slovenia
  2. $ whoami ✓ name: Hans-Peter Grahsl ✓ location: Graz, Austria

    what I do: ✓ technical trainer @ NETCONOMY ✓ independent consultant & engineer ✓ associate lecturer @ CAMPUS 02 ✓ irregular conference speaker @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 2
  3. streaming @hpgrahsl | Make IT Conference, 14th October 2019, Portorož,

    Slovenia 3
  4. "... data processing that is designed with infinite data sets

    in mind." — Tyler Akidau @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 4
  5. Streaming BIG DEAL @hpgrahsl | Make IT Conference, 14th October

    2019, Portorož, Slovenia 5
  6. EVENTS... events everywhere @hpgrahsl | Make IT Conference, 14th October

    2019, Portorož, Slovenia 6
  7. NOW! @hpgrahsl | Make IT Conference, 14th October 2019, Portorož,

    Slovenia 7
  8. Streaming alleviates Pain ✓ built for data-in-motion ✓ event-centric &

    stream-oriented ✓ fast and accurate answers @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 8
  9. BIGGEST Challenge? @hpgrahsl | Make IT Conference, 14th October 2019,

    Portorož, Slovenia 9
  10. "Jungle" with many more

  11. My choice for today

  12. Apache Kafka @hpgrahsl | Make IT Conference, 14th October 2019,

    Portorož, Slovenia 12
  13. Apache Kafka ✓ pub / sub to event streams ✓

    (permanently) store event streams ✓ process streams in real-time ➔ horizontal scalability ➔ high fault-tolerance @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 13
  14. EVENT STREAMING PLATFORM @hpgrahsl | Make IT Conference, 14th October

    2019, Portorož, Slovenia 14
  15. APIs for Everything @hpgrahsl | Make IT Conference, 14th October

    2019, Portorož, Slovenia 15
  16. Everything built for Streaming @hpgrahsl | Make IT Conference, 14th

    October 2019, Portorož, Slovenia 16
  17. Kafka's streaming SQL engine @hpgrahsl | Make IT Conference, 14th

    October 2019, Portorož, Slovenia 17
  18. declarative stream processing language @hpgrahsl | Make IT Conference, 14th

    October 2019, Portorož, Slovenia 18
  19. skyrocketing developer productivity @hpgrahsl | Make IT Conference, 14th October

    2019, Portorož, Slovenia 19
  20. unleash streaming for the masses @hpgrahsl | Make IT Conference,

    14th October 2019, Portorož, Slovenia 20
  21. KSQL in a Nutshell ✓ built on top of Kafka

    Streams ✓ NO(!) coding skills required ✓ SQL only ➔ not embedded ✓ extremely low entry barrier ✓ familiar syntax & semantics ✓ concise & expressive @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 21
  22. KSQL in a Nutshell the usual suspects OOTB: ✓ projections,

    filters ✓ joins, aggregations ✓ windowing something is missing? roll your own: ✓ UDF & UDAF ✓ UDTF pending @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 22
  23. @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia

    23
  24. @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia

    24
  25. @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia

    25
  26. @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia

    26
  27. @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia

    27
  28. @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia

    28
  29. KSQL Queries ✓ per-record streaming with ms latency ✓ compiled

    into Kafka Streams apps ✓ distributed execution on KSQL servers ✓ 2 modes: interactive vs. headless @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 29
  30. KSQL Interactive Mode ✓ KSQL servers accessed via REST API

    ✓ offers ad-hoc stream analytics ✓ share streams & tables across users ✓ used for exploration and during development @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 30
  31. @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia

    31
  32. @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia

    32
  33. @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia

    33
  34. @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia

    34
  35. KSQL Headless Mode ✓ application == SQL file ✓ KSQL

    servers run streaming queries ✓ use case specific isolation ✓ "locked-down" ➡ NO REST API access ✓ used for production deployments @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 35
  36. @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia

    36
  37. SHOW ME! @hpgrahsl | Make IT Conference, 14th October 2019,

    Portorož, Slovenia 37
  38. Demo Scenario @hpgrahsl | Make IT Conference, 14th October 2019,

    Portorož, Slovenia 38
  39. Demo Scenario @hpgrahsl | Make IT Conference, 14th October 2019,

    Portorož, Slovenia 39
  40. Demo Scenario @hpgrahsl | Make IT Conference, 14th October 2019,

    Portorož, Slovenia 40
  41. Demo Scenario @hpgrahsl | Make IT Conference, 14th October 2019,

    Portorož, Slovenia 41
  42. Demo Scenario @hpgrahsl | Make IT Conference, 14th October 2019,

    Portorož, Slovenia 42
  43. Demo Scenario @hpgrahsl | Make IT Conference, 14th October 2019,

    Portorož, Slovenia 43
  44. KSQL in Action @hpgrahsl | Make IT Conference, 14th October

    2019, Portorož, Slovenia 44
  45. "You think that's a database table you are querying now?"

    — Morpheus @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 45
  46. "Instead, only try to realize the truth... there is no

    database table." — Spoon Boy @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 46
  47. MISSION accomplished @hpgrahsl | Make IT Conference, 14th October 2019,

    Portorož, Slovenia 47
  48. KSQL Wrap-Up ✓ SQL... and nothing but SQL ✓ provides

    exactly-once semantics ✓ deployable anywhere in any way ✓ no additional infrastructure ✓ use cases of any size (XS ... XXXL) ✓ scalable & fault-tolerant @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 48
  49. Your obsession tells you to do batching. I tell you

    to walk away and stream with KSQL The choice is yours folks! @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 49
  50. reach out to me @hpgrahsl @hpgrahsl | Make IT Conference,

    14th October 2019, Portorož, Slovenia 50
  51. THANK YOU Q & A ? @hpgrahsl | Make IT

    Conference, 14th October 2019, Portorož, Slovenia 51