Making Sense of Sensors with SQL Streaming @ Make IT 2019

Making Sense of Sensors with SQL Streaming @ Make IT 2019

We are living in a data streaming era, yet until recently it has been particularly hard to leverage existing stream processing technologies. On the one hand, because dealing with data in motion has its inherent challenges.

On the other hand, most frameworks and APIs which are allowing for stream processing are typically very hard to employ and/or operate - NOT so for KSQL, the newest kid in Apache Kafka's ecosystem. Based on a simplified version of an IoT use case this session gives a gentle introduction into KSQL - Kafka's SQL streaming engine for the masses.

Join this whirlwind tour during which we are discussing a fully-fledged streaming IoT architecture. Concretely, we are going to:

* ingest smart home energy data into Apache Kafka,
* use KSQL for flexible, powerful and scalable SQL-only stream processing,
* send raw data, as well as pre-processed results to an operational NoSQL data store,
* reactively, serve data to clients in near real-time and
* finally build informative live charts.

Note: None of the sessions were recorded here so no video link this time...


Hans-Peter Grahsl

October 14, 2019