Making Sense of Sensors with SQL Streaming @ Make IT 2019

We are living in a data streaming era, yet until recently it has been particularly hard to leverage existing stream processing technologies. On the one hand, because dealing with data in motion has its inherent challenges.

On the other hand, most frameworks and APIs which are allowing for stream processing are typically very hard to employ and/or operate - NOT so for KSQL, the newest kid in Apache Kafka's ecosystem. Based on a simplified version of an IoT use case this session gives a gentle introduction into KSQL - Kafka's SQL streaming engine for the masses.

Join this whirlwind tour during which we are discussing a fully-fledged streaming IoT architecture. Concretely, we are going to:

* ingest smart home energy data into Apache Kafka,
* use KSQL for flexible, powerful and scalable SQL-only stream processing,
* send raw data, as well as pre-processed results to an operational NoSQL data store,
* reactively, serve data to clients in near real-time and
* finally build informative live charts.

Note: None of the sessions were recorded here so no video link this time...

Transcript

1. Making Sense of Sensors with SQL Streaming @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia
2. $ whoami ✓ name: Hans-Peter Grahsl ✓ location: Graz, Austria what I do: ✓ technical trainer @ NETCONOMY ✓ independent consultant & engineer ✓ associate lecturer @ CAMPUS 02 ✓ irregular conference speaker @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 2
3. streaming @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 3
4. "... data processing that is designed with infinite data sets in mind." — Tyler Akidau @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 4
5. Streaming BIG DEAL @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 5
6. EVENTS... events everywhere @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 6
7. NOW! @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 7
8. Streaming alleviates Pain ✓ built for data-in-motion ✓ event-centric & stream-oriented ✓ fast and accurate answers @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 8
9. BIGGEST Challenge? @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 9
10. "Jungle" with many more
11. My choice for today
12. Apache Kafka @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 12
13. Apache Kafka ✓ pub / sub to event streams ✓ (permanently) store event streams ✓ process streams in real-time ➔ horizontal scalability ➔ high fault-tolerance @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 13
14. EVENT STREAMING PLATFORM @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 14
15. APIs for Everything @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 15
16. Everything built for Streaming @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 16
17. Kafka's streaming SQL engine @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 17
18. declarative stream processing language @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 18
19. skyrocketing developer productivity @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 19
20. unleash streaming for the masses @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 20
21. KSQL in a Nutshell ✓ built on top of Kafka Streams ✓ NO(!) coding skills required ✓ SQL only ➔ not embedded ✓ extremely low entry barrier ✓ familiar syntax & semantics ✓ concise & expressive @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 21
22. KSQL in a Nutshell the usual suspects OOTB: ✓ projections, filters ✓ joins, aggregations ✓ windowing something is missing? roll your own: ✓ UDF & UDAF ✓ UDTF pending @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 22
23. @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 23
24. @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 24
25. @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 25
26. @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 26
27. @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 27
28. @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 28
29. KSQL Queries ✓ per-record streaming with ms latency ✓ compiled into Kafka Streams apps ✓ distributed execution on KSQL servers ✓ 2 modes: interactive vs. headless @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 29
30. KSQL Interactive Mode ✓ KSQL servers accessed via REST API ✓ offers ad-hoc stream analytics ✓ share streams & tables across users ✓ used for exploration and during development @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 30
31. @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 31
32. @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 32
33. @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 33
34. @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 34
35. KSQL Headless Mode ✓ application == SQL file ✓ KSQL servers run streaming queries ✓ use case specific isolation ✓ "locked-down" ➡ NO REST API access ✓ used for production deployments @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 35
36. @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 36
37. SHOW ME! @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 37
38. Demo Scenario @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 38
39. Demo Scenario @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 39
40. Demo Scenario @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 40
41. Demo Scenario @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 41
42. Demo Scenario @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 42
43. Demo Scenario @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 43
44. KSQL in Action @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 44
45. "You think that's a database table you are querying now?" — Morpheus @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 45
46. "Instead, only try to realize the truth... there is no database table." — Spoon Boy @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 46
47. MISSION accomplished @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 47
48. KSQL Wrap-Up ✓ SQL... and nothing but SQL ✓ provides exactly-once semantics ✓ deployable anywhere in any way ✓ no additional infrastructure ✓ use cases of any size (XS ... XXXL) ✓ scalable & fault-tolerant @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 48
49. Your obsession tells you to do batching. I tell you to walk away and stream with KSQL The choice is yours folks! @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 49
50. reach out to me @hpgrahsl @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 50
51. THANK YOU Q & A ? @hpgrahsl | Make IT Conference, 14th October 2019, Portorož, Slovenia 51