Apache Kafka with Red Hat AMQ Streams

Red Hat Webinars, 2019. https://onlinexperiences.com/scripts/Server.nxp?LASCmd=AI:4;F:QS!10100&ShowUUID=25AB8D7E-DC73-494D-AB3A-4FE72D43ED5A

Follow me on twitter (@systemcraftsman) or subscribe to https://www.systemcraftsman.com/join/ to get any updates from me.

Transcript

1. 1
   Distributed streaming made easy on the cloud
   Apache Kafka with
   Red Hat AMQ
   Streams
   Aykut M. Bulgu
   Middleware Consultant - Red Hat
   @systemcraftsman
   APACHE KAFKA WITH RED HAT AMQ STREAMS
   

   View Slide

2. 2
   Who am I?
   #oc apply -f aykutbulgu.yaml
   apiVersion: redhat/v1.5
   kind: Middleware Consultant
   metadata:
   name: Aykut Bulgu
   namespace: Red Hat Consulting, CEMEA
   Annotations:
   twitter: @systemcraftsman
   organizer: Software Craftsmanship Turkey
   founder: System Craftsman
   labels:
   married: yes
   children: daughter
   interests: openshift, kubernetes, spring boot, middleware,
   infinispan, kafka, strimzi, camel
   spec:
   replicas: 1
   containers:
   - image: aykut:latest
   

   View Slide

3. 3
   Messaging
   Messaging ≠ ≠ Messaging
   Low-latency
   pub/sub
   Cross-cloud
   backbone
   Temporal
   decoupling
   Load
   levelling
   Load
   balancing
   Enterprise
   application
   integration
   IoT device
   connectivity
   Message-driv
   en beans
   Event-driven
   microservices
   Long-term
   message
   storage
   Replayable
   streams
   Event
   sourcing
   Geo-aware
   routing
   Database
   change data
   capture
   

   View Slide

4. 4
   Flexible messaging for the enterprise, cloud and Internet of Things
   Standard
   Protocols Broker
   • Queuing and pub/sub
   • Rich feature set
   • JMS 2.0 compliance
   • Best-in-class perf
   • Based on Apache
   ActiveMQ Artemis
   Interconnect
   • Message routing
   • Secure messaging
   backbone for hybrid
   cloud
   • Based on Apache
   Qpid Dispatch Router
   Streams
   • Durable pub/sub
   • Replayable streams
   • Highly scalable
   • Based on Apache
   Kafka
   Online
   • Scalable, “self-service” messaging-as-a-service utility based on OpenShift
   • Available for self-managed and Red Hat-managed deployments
   Polyglot
   Clients
   Common
   Management
   Red Hat AMQ 2019
   

   View Slide

5. What is Apache Kafka?
   5
   A publish/subscribe messaging system.
   A data streaming platform
   A distributed, horizontally-scalable, fault-tolerant, commit log
   

   View Slide

6. 6
   Developed at LinkedIn back in 2010, open sourced in 2011
   Distributed by design
   High throughput
   Designed to be fast, scalable, durable and highly available
   Data partitioning (sharding)
   Ability to handle huge number of consumers
   What is Apache Kafka?
   

   View Slide

7. Traditional Messaging
   7
   Queue
   Producer Consumer
   1
   2
   3
   Reference count-based message retention model
   When message is consumed it is deleted from broker
   “Smart broker, dumb client”
   Broker knows about all consumers
   Can perform per consumer filtering
   Throughput up to 4K/10K mes./sec.
   

   View Slide

8. Apache Kafka
   8
   Kafka Topic
   Producer Consumer
   1
   2
   3 1
   2
   3
   Time-based message retention model by default
   Messages are retained according to topic config (time or capacity)
   Also “compacted topic” – like a “last-value topic”
   “Dumb broker, smart client”
   Client maintains position in message stream
   Message stream can be replayed
   Throughput up to 1 million mes./sec.
   

   View Slide

9. Kafka Concepts - Producers
   9
   old new
   0 1 2 3 4 5 6 7 8 9
   0 1 2 3 4 5 6
   0 1 2 3 4 5 6 7 8
   Partition 0
   Partition 1
   Partition 2
   Producer
   Topic
   

   View Slide

10. Kafka Concepts - Consumers
    10
    old new
    0 1 2 3 4 5 6 7 8 9
    0 1 2 3 4 5 6
    0 1 2 3 4 5 6 7 8
    Partition 0
    Partition 1
    Partition 2
    Consumer
    Topic
    

    View Slide

11. Kafka Concepts - High Availability
    11
    Broker 1
    T1 - P1
    T1 - P2
    T2 - P1
    T2 - P2
    Broker 2
    T1 - P1
    T1 - P2
    T2 - P1
    T2 - P2
    Broker 3
    T1 - P1
    T1 - P2
    T2 - P1
    T2 - P2
    Leaders and followers spread across the cluster
    

    View Slide

12. Kafka Concepts - High Availability
    12
    If a broker with leader partition goes down, a new leader partition is elected on different node
    Broker 1
    T1 - P1
    T1 - P2
    T2 - P1
    T2 - P2
    Broker 2
    T1 - P1
    T1 - P2
    T2 - P1
    T2 - P2
    Broker 3
    T1 - P1
    T1 - P2
    T2 - P1
    T2 - P2
    

    View Slide

13. Kafka Concepts - Interaction with Leaders
    13
    Broker 1
    T1 - P1
    T1 - P2
    T2 - P1
    T2 - P2
    Broker 2
    T1 - P1
    T1 - P2
    T2 - P1
    T2 - P2
    Broker 3
    T1 - P1
    T1 - P2
    T2 - P1
    T2 - P2
    Producer P2
    Consumer C3
    Consumer C1
    Producer P1
    Consumer C2
    

    View Slide

14. 14
    Based on OSS project called Strimzi
    Container images for running Apache Kafka and Zookeeper
    Operators for managing and configuring Apache Kafka clusters, topics and users
    Simplified deployment on OpenShift
    Broker, Connect, Streams, Mirror Maker, Java clients & management tools
    Apache Zookeeper (as a Kafka dependency)
    Enterprise distribution of Apache Kafka
    Red Hat AMQ Streams
    

    View Slide

15. 15
    Enterprise Kubernetes
    A comprehensive enterprise-grade application platform, built for
    containers with Kubernetes
    Orchestration
    Orchestrates computing, networking, and storage infrastructure on
    behalf of user workloads
    Modularity
    Better software management through modularity
    Simplified Deployment
    Simplifies the deployment and update of software at scale
    High Availability & Auto Scalability
    Provides high availability, horizontal autoscaling, rolling updates,
    canary deployments
    

    View Slide

16. Goals for AMQ Streams on OpenShift
    16
    Simplifying the deployment of Apache Kafka on OpenShift
    Using the OpenShift native mechanisms for;
    Provisioning the cluster
    Managing the topics and users
    Providing a better integration with applications running on OpenShift
    Microservices, data streaming, event-sourcing, etc.
    

    View Slide

17. Challanges
    17
    A Kafka cluster requires;
    A stable broker identity and stable network address
    A way for brokers to discover each other and communicate
    Durable state on brokers and storage recovery
    To have brokers accessible from clients, directly
    It runs alongside a Zookeeper ensemble which requires;
    Each node has the configuration of the others
    To have nodes able to communicate each others
    Accessing Kafka isn’t so simple
    

    View Slide

18. How Openshift Helps
    18
    Openshift provides;
    StatefulSets for stable identity and network
    Together with Headless services for internal discovery
    Services for accessing the cluster
    Secrets and ConfigMap for handling configurations
    PersistentVolume and PersistentVolumeClaim for durable storage
    Openshift/Kubernetes primitives help but still not easy
    

    View Slide

19. The Operator Pattern
    19
    An application used to create, configure and manage other complex applications
    Contains domain-specific domain knowledge
    Operator works based on input from Custom Resource Definitions (CRDs)
    User describes the desired state
    Controller applies this state to the application
    It watches the *desired* state and the *actual* state and makes forward progress to reconcile
    OperatorHub.io
    Observe
    Analyze
    Act
    

    View Slide

20. AMQ Streams Operators
    20
    Cluster
    Operator
    Kafka CR
    Kafka
    Zookeeper
    Deploys & manages
    cluster
    Topic
    Operator
    User
    Operator
    Topic CR
    User CR
    Manages
    topics & users
    

    View Slide

21. Accessing Kafka
    21
    Broker 1
    T1 - P1
    T1 - P2
    T2 - P1
    T2 - P2
    Broker 2
    T1 - P1
    T1 - P2
    T2 - P1
    T2 - P2
    Broker 3
    T1 - P1
    T1 - P2
    T2 - P1
    T2 - P2
    Producer P2
    Consumer C3
    Consumer C1
    Producer P1
    Consumer C2
    

    View Slide

22. Kafka’s Discovery Protocol
    22
    

    View Slide

23. Openshift Cluster Internal Access
    23
    

    View Slide

24. Openshift Cluster External Access
    24
    

    View Slide

25. AMQ Streams on OpenShift
    25
    Tolerations
    Memory and CPU
    resources
    High
    Availability
    Mirroring
    Affinity
    Authentication
    Storage
    Encryption
    Scale Down
    JVM
    Configuration
    Logging
    Metrics
    Off cluster
    access
    Scale Up
    Authorization
    Healthchecks
    Source2Image
    Configuration
    

    View Slide

26. AMQ Streams Use Cases
    26
    Messaging
    Replacement of traditional message broker High scale, high throughput, built-in partitioning, replication, and
    fault-tolerance. Some limitations compared to traditional broker (filtering, standard protocols, JMS …)
    Website Activity Tracker
    Rebuild user activity tracking pipeline as a set of real-time publish-subscribe feeds. Activity is published to central
    topics with one topic per activity type
    Metrics
    Aggregation of statistics from distributed applications to produce centralized feeds of operational data.
    Log Aggregation
    Abstracts details of files an gives event data as stream of messages. Offers good performance, stronger durability
    guarantees due to replication.
    Stream Processing
    Enables continuous, real-time applications built to react to, process, or transform streams.
    Data Integration
    Captures streams of events or data changes and feed these to other data systems (see Debezium project).
    

    View Slide

27. Demo
    27
    

    View Slide

28. Summary
    28
    AMQ Streams 1.3.0 is generally available now
    Productized distribution of Apache Kafka (v2.3.0)
    Broker, Connect, Streams, Mirror Maker, Java clients & management tools
    Apache Zookeeper (as a Kafka dependency)
    Simplified deployment and management on OpenShift
    Based on OSS project called Strimzi
    Container images for running Apache Kafka and Zookeeper
    Operators for managing and configuring Apache Kafka clusters, topics and users
    

    View Slide

29. Q/A
    29
    

    View Slide

30. CONFIDENTIAL Designator
    linkedin.com/company/red-hat
    youtube.com/user/RedHatVideos
    facebook.com/redhatinc
    twitter.com/RedHat
    30
    Red Hat AMQ Streams
    https://access.redhat.com/products/red-hat-amq
    Strimzi
    http://strimzi.io/
    @strimziio
    Apache Kafka
    https://kafka.apache.org/
    Thank you
    APACHE KAFKA WITH RED HAT AMQ STREAMS
    

    View Slide