Streaming ETL in Practice with PostgreSQL, Apache Kafka, and KSQL

2bded62396ea66c84bd10e91c718dea9?s=47 rmoff
July 03, 2018

Streaming ETL in Practice with PostgreSQL, Apache Kafka, and KSQL

Have you ever thought that you needed to be a programmer to do stream processing and build streaming data pipelines? Think again!

Companies new and old are all recognising the importance of a low-latency, scalable, fault-tolerant data backbone, in the form of the Apache Kafka® streaming platform. With Kafka, developers can integrate multiple sources and systems, which enables low latency analytics, event driven architectures and the population of multiple downstream systems. These data pipelines can be built using configuration alone.

In this talk, we’ll see how easy it is to stream data from a database such as PostgreSQL into Kafka using CDC and Kafka Connect. In addition, we’ll use KSQL to filter, aggregate and join it to other data, and then stream this from Kafka out into multiple targets such as Elasticsearch and S3. All of this can be accomplished without a single line of code!

Why should Java geeks have all the fun?

2bded62396ea66c84bd10e91c718dea9?s=128

rmoff

July 03, 2018
Tweet