When NOT to use Apache Kafka?

Apache Kafka is the de facto standard for data streaming to process data in motion. With its significant adoption growth across all industries, I get a very valid question every week: When NOT to use Apache Kafka? What limitations does the event streaming platform have? When does Kafka simply not provide the needed capabilities? How to qualify Kafka out as it is not the right tool for the job?

This session explores the DOs and DONTs. Separate sections explain when to use Kafka, when NOT to use Kafka, and when to MAYBE use Kafka.

No matter if you think about open source Apache Kafka, a cloud service like Confluent Cloud, or another technology using the Kafka protocol like Redpanda or Pulsar, check out this slide deck.

A detailed article about this topic:
https://www.kai-waehner.de/blog/2022/01/04/when-not-to-use-apache-kafka/

Transcript

1. When NOT to use Apache Kafka? kai-waehner.de | @KaiWaehner | Field CTO @ Confluent
2. Data Streaming with Apache Kafka DWH APP STREAM PROCESSING CONNECTORS ksqlDB KStreams APP Streaming ETL Data Processing Real-time Analytics Stateless and Stateful Business Applications Fully-managed Pipelines Connectivity to Data Infrastructure, SaaS, AI/ML Data Governance Connectivity Filtering and Routing Change Data Capture Built-in Scale and Fault Tolerance Oracle DB ORACLE CDC SOURCE PREMIUM CONNECTOR Real-time Data Sharing across Hybrid and Multi-Cloud Storage Backpressure Handling Slow Consumers Replayability kai-waehner.de | @KaiWaehner | When NOT to use Apache Kafka?
3. Markets DaaS Digital replatforming/ Legacy Modernization Customer 360 Faster transactional processing / analysis incl. Machine Learning / AI Microservices Architecture Online Fraud Detection Online Security (syslog, log aggregation, Splunk replacement) Middleware replacement Website / Core Operations / Payments (Central Nervous System) Real-time app updates Customer Experience Core Business Platform Operational Efficiency (Agility) Migrate to Cloud Fraud Detection Regulatory Increase Revenue (make money) Decrease Costs (save money) Mitigate Risk (protect money) Business Value 10 business use case Strategic Driver 20 business use case Data Eng. / Infrastructure use case Use Cases for Data Streaming by Business Value kai-waehner.de | @KaiWaehner | When NOT to use Apache Kafka?
4. When NOT to use Apache Kafka? kai-waehner.de | @KaiWaehner | When NOT to use Apache Kafka?
5. Kafka is a Database BUT NOT for Complex Analytics kai-waehner.de | @KaiWaehner | When NOT to use Apache Kafka? Durable Fault-tolerant Tiered Storage Compacted Topics Exactly-once Semantics RocksDB on Client Side ksqlDB Interactive Queries “You Name It” Connect
6. Kafka is NOT a Proxy for Millions of Clients kai-waehner.de | @KaiWaehner | When NOT to use Apache Kafka? “Last Mile” Integration is usually a Proxy (like HTTP or MQTT)
7. Kafka is NOT an API Management Platform kai-waehner.de | @KaiWaehner | When NOT to use Apache Kafka? Orders Customers Payments Stock API (HTTP/REST) Data Streaming Data Integration Real-Time Apps API Gateway API Lifecycle Data Sharing Monetization REST Proxy Stream Exchange
8. Kafka is NOT the right tool for processing large messages * kai-waehner.de | @KaiWaehner | When NOT to use Apache Kafka? Claim Check Enterprise Integration Pattern: * BUT works well for some use cases, e.g.: - Splitting large legacy CSV files - Externalizing large payloads on-the-fly - Image processing at the edge - Uploading large files into the DWH Pre-Processing and Data Correlation e.g. enrich with other metadata (ksqlDB) Store big files in data lake (e.g. AWS S3) Consume and correlate metadata (Kafka Streams) Automated Orchestration (Kafka Clients) Real time analytics and other business applications (Kafka Clients + other tools) Send metadata including link to video in object store (Kafka Producer) Download big files from data lake
9. Kafka is NOT an IoT Platform * kai-waehner.de | @KaiWaehner | When NOT to use Apache Kafka? Siemens S7 Kafka Connect Storage Kafka Streams / ksqlDB Stateless + Stateful REST Proxy HTTP(S) SCADA DCS ERP MES Cloud Factory * BUT Kafka is a fundamental part of most IoT projects, e.g.: - Scalable real-time data hub for IoT data AND IT data - Edge and hybrid cloud - Direct integration with IoT protocols - Integration via 3rd party with IoT protocols Analytics Database Data Lake CRM Kafka Connect Cluster Linking
10. Kafka is NOT for hard real-time requirements kai-waehner.de | @KaiWaehner | When NOT to use Apache Kafka? OT - Connected Vehicle (Car, Train, Drone) OT - Manufacturing (Field Bus, PLC, Machine, Robot) IT – Enterprise Software (Data Center, Cloud, Car IT) Central Data Center / Public Cloud Vehicle Data Robot Data All Data C C++ Rust C C++ Rust Java Python Go [#] Hard Real Time = Deterministic network with zero spikes + zero latency [#] Soft Real Time + Near Real Time + Batch Cluster Linking Cluster Linking
11. TL;DR == de facto standard for DATA STREAMING
12. Kai Waehner Field CTO kai.waehner@confluent.io @KaiWaehner confluent.io kai-waehner.de linkedin.com/in/kaiwaehner Questions? Feedback? Let’s connect!