Apache Kafka is a streaming platform that enables integration of data across the enterprise, and ships with its own stream processing capabilities. But how do we get data in and out of Kafka, and at the same time write SQL-like stream processing for transformations of the data, both in an easy, scalable and standardised manner? Enter Kafka Connect, and KSQL from Confluent.
Part of Apache Kafka, Kafka Connect enables the integration of data from multiple sources, including Oracle. As well as ingest, Kafka Connect has connectors with support for numerous targets, including Hadoop, S3 and Elasticsearch. Building on Kafka Streams, KSQL from Confluent enables stream processing using a SQL-like language.
This presentation will briefly recap the purpose of Kafka, and then dive into Kafka Connect with practical examples of data pipelines that can be built with it and are in production at companies around the world already. We'll also look at the Single Message Transform (SMT) capabilities and how they can make Kafka Connect even more flexible and powerful, along with the newly-announced KSQL for powerful query-based stream processing.