When NOT to use Apache Kafka? kai-waehner.de | @KaiWaehner | Field CTO @ Confluent

Data Streaming with Apache Kafka DWH APP STREAM PROCESSING CONNECTORS ksqlDB KStreams APP Streaming ETL Data Processing Real-time Analytics Stateless and Stateful Business Applications Fully-managed Pipelines Connectivity to Data Infrastructure, SaaS, AI/ML Data Governance Connectivity Filtering and Routing Change Data Capture Built-in Scale and Fault Tolerance Oracle DB ORACLE CDC SOURCE PREMIUM CONNECTOR Real-time Data Sharing across Hybrid and Multi-Cloud Storage Backpressure Handling Slow Consumers Replayability kai-waehner.de | @KaiWaehner | When NOT to use Apache Kafka?

Markets DaaS Digital replatforming/ Legacy Modernization Customer 360 Faster transactional processing / analysis incl. Machine Learning / AI Microservices Architecture Online Fraud Detection Online Security (syslog, log aggregation, Splunk replacement) Middleware replacement Website / Core Operations / Payments (Central Nervous System) Real-time app updates Customer Experience Core Business Platform Operational Efficiency (Agility) Migrate to Cloud Fraud Detection Regulatory Increase Revenue (make money) Decrease Costs (save money) Mitigate Risk (protect money) Business Value 10 business use case Strategic Driver 20 business use case Data Eng. / Infrastructure use case Use Cases for Data Streaming by Business Value kai-waehner.de | @KaiWaehner | When NOT to use Apache Kafka?

When NOT to use Apache Kafka? kai-waehner.de | @KaiWaehner | When NOT to use Apache Kafka?

Kafka is a Database BUT NOT for Complex Analytics kai-waehner.de | @KaiWaehner | When NOT to use Apache Kafka? Durable Fault-tolerant Tiered Storage Compacted Topics Exactly-once Semantics RocksDB on Client Side ksqlDB Interactive Queries “You Name It” Connect

Kafka is NOT a Proxy for Millions of Clients kai-waehner.de | @KaiWaehner | When NOT to use Apache Kafka? “Last Mile” Integration is usually a Proxy (like HTTP or MQTT)

Kafka is NOT an API Management Platform kai-waehner.de | @KaiWaehner | When NOT to use Apache Kafka? Orders Customers Payments Stock API (HTTP/REST) Data Streaming Data Integration Real-Time Apps API Gateway API Lifecycle Data Sharing Monetization REST Proxy Stream Exchange

Kafka is NOT the right tool for processing large messages * kai-waehner.de | @KaiWaehner | When NOT to use Apache Kafka? Claim Check Enterprise Integration Pattern: * BUT works well for some use cases, e.g.: - Splitting large legacy CSV files - Externalizing large payloads on-the-fly - Image processing at the edge - Uploading large files into the DWH Pre-Processing and Data Correlation e.g. enrich with other metadata (ksqlDB) Store big files in data lake (e.g. AWS S3) Consume and correlate metadata (Kafka Streams) Automated Orchestration (Kafka Clients) Real time analytics and other business applications (Kafka Clients + other tools) Send metadata including link to video in object store (Kafka Producer) Download big files from data lake

Kafka is NOT an IoT Platform * kai-waehner.de | @KaiWaehner | When NOT to use Apache Kafka? Siemens S7 Kafka Connect Storage Kafka Streams / ksqlDB Stateless + Stateful REST Proxy HTTP(S) SCADA DCS ERP MES Cloud Factory * BUT Kafka is a fundamental part of most IoT projects, e.g.: - Scalable real-time data hub for IoT data AND IT data - Edge and hybrid cloud - Direct integration with IoT protocols - Integration via 3rd party with IoT protocols Analytics Database Data Lake CRM Kafka Connect Cluster Linking

Kafka is NOT for hard real-time requirements kai-waehner.de | @KaiWaehner | When NOT to use Apache Kafka? OT - Connected Vehicle (Car, Train, Drone) OT - Manufacturing (Field Bus, PLC, Machine, Robot) IT – Enterprise Software (Data Center, Cloud, Car IT) Central Data Center / Public Cloud Vehicle Data Robot Data All Data C C++ Rust C C++ Rust Java Python Go [#] Hard Real Time = Deterministic network with zero spikes + zero latency [#] Soft Real Time + Near Real Time + Batch Cluster Linking Cluster Linking

TL;DR == de facto standard for DATA STREAMING

Kai Waehner Field CTO [email protected] @KaiWaehner confluent.io kai-waehner.de linkedin.com/in/kaiwaehner Questions? Feedback? Let’s connect!