Things Standard Protocols Broker • Queuing and pub/sub • Rich feature set • JMS 2.0 compliance • Best-in-class perf • Based on Apache ActiveMQ Artemis Interconnect • Message routing • Secure messaging backbone for hybrid cloud • Based on Apache Qpid Dispatch Router Streams • Durable pub/sub • Replayable streams • Highly scalable • Based on Apache Kafka Online • Scalable, “self-service” messaging-as-a-service utility based on OpenShift • Available for self-managed and Red Hat-managed deployments Polyglot Clients Common Management Red Hat AMQ 2019
count-based message retention model When message is consumed it is deleted from broker “Smart broker, dumb client” Broker knows about all consumers Can perform per consumer ﬁltering Throughput up to 4K/10K mes./sec.
1 2 3 Time-based message retention model by default Messages are retained according to topic conﬁg (time or capacity) Also “compacted topic” – like a “last-value topic” “Dumb broker, smart client” Client maintains position in message stream Message stream can be replayed Throughput up to 1 million mes./sec.
running Apache Kafka and Zookeeper Operators for managing and conﬁguring Apache Kafka clusters, topics and users Simpliﬁed deployment on OpenShift Broker, Connect, Streams, Mirror Maker, Java clients & management tools Apache Zookeeper (as a Kafka dependency) Enterprise distribution of Apache Kafka Red Hat AMQ Streams
containers with Kubernetes Orchestration Orchestrates computing, networking, and storage infrastructure on behalf of user workloads Modularity Better software management through modularity Simpliﬁed Deployment Simpliﬁes the deployment and update of software at scale High Availability & Auto Scalability Provides high availability, horizontal autoscaling, rolling updates, canary deployments
of Apache Kafka on OpenShift Using the OpenShift native mechanisms for; Provisioning the cluster Managing the topics and users Providing a better integration with applications running on OpenShift Microservices, data streaming, event-sourcing, etc.
and stable network address A way for brokers to discover each other and communicate Durable state on brokers and storage recovery To have brokers accessible from clients, directly It runs alongside a Zookeeper ensemble which requires; Each node has the conﬁguration of the others To have nodes able to communicate each others Accessing Kafka isn’t so simple
and network Together with Headless services for internal discovery Services for accessing the cluster Secrets and ConﬁgMap for handling conﬁgurations PersistentVolume and PersistentVolumeClaim for durable storage Openshift/Kubernetes primitives help but still not easy
and manage other complex applications Contains domain-speciﬁc domain knowledge Operator works based on input from Custom Resource Deﬁnitions (CRDs) User describes the desired state Controller applies this state to the application It watches the *desired* state and the *actual* state and makes forward progress to reconcile OperatorHub.io Observe Analyze Act
broker High scale, high throughput, built-in partitioning, replication, and fault-tolerance. Some limitations compared to traditional broker (ﬁltering, standard protocols, JMS …) Website Activity Tracker Rebuild user activity tracking pipeline as a set of real-time publish-subscribe feeds. Activity is published to central topics with one topic per activity type Metrics Aggregation of statistics from distributed applications to produce centralized feeds of operational data. Log Aggregation Abstracts details of ﬁles an gives event data as stream of messages. Offers good performance, stronger durability guarantees due to replication. Stream Processing Enables continuous, real-time applications built to react to, process, or transform streams. Data Integration Captures streams of events or data changes and feed these to other data systems (see Debezium project).
distribution of Apache Kafka (v2.3.0) Broker, Connect, Streams, Mirror Maker, Java clients & management tools Apache Zookeeper (as a Kafka dependency) Simpliﬁed deployment and management on OpenShift Based on OSS project called Strimzi Container images for running Apache Kafka and Zookeeper Operators for managing and conﬁguring Apache Kafka clusters, topics and users