Building Streaming Data Pipelines with Elasticsearch, Apache Kafka, and KSQL

2bded62396ea66c84bd10e91c718dea9?s=47 Robin Moffatt
January 31, 2018

Building Streaming Data Pipelines with Elasticsearch, Apache Kafka, and KSQL

Companies new and old are all recognising the importance of a low-latency, scalable, fault-tolerant data backbone, in the form of the Apache Kafka streaming platform. With Kafka, developers can integrate multiple sources and systems, which enables low latency analytics, event driven architectures and the population of multiple downstream systems. These data pipelines can be built using configuration alone.

In this talk, we’ll see how easy it is to stream data from sources such as databases into Kafka using the Kafka Connect API. We’ll use KSQL to filter, aggregate and join it to other data, and then stream this from Kafka out into targets such as Elasticsearch, and see how time-based indices can be used. All of this can be accomplished without a single line of code!

2bded62396ea66c84bd10e91c718dea9?s=128

Robin Moffatt

January 31, 2018
Tweet