Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Running Apache Kafka on Red Hat Openshift with AMQ Streams

Running Apache Kafka on Red Hat Openshift with AMQ Streams

Lab delivered at Red Hat Summit 2019

Marius Bogoevici

May 08, 2019
Tweet

More Decks by Marius Bogoevici

Other Decks in Technology

Transcript

  1. RUNNING APACHE KAFKA ON
    OPENSHIFT WITH AMQ STREAMS
    May 2019
    Marius Bogoevici
    Paolo Patierno
    Gunnar Morling
    Emmanuel Bernard

    View full-size slide

  2. AGENDA
    Running a Kafka cluster
    on OpenShift
    Managing access and
    security
    Replication and
    monitoring
    Goal: Learn the practical aspects of deploying and operating Kafka clusters on OpenShift

    View full-size slide

  3. Lab Environment
    OpenShift
    Workstation
    SSH
    Web Console
    Lab
    Machine
    CLI

    View full-size slide

  4. Module 1: Deploying and managing
    Kafka clusters

    View full-size slide

  5. LABS
    1. AMQ Streams on OpenShift from 0 to 60
    a. Deploying the operator and a minimal cluster
    2. Production-ready topologies
    a. Deploying persistent, scaled-up clusters
    b. Scaling clusters
    3. Managing Topics
    a. Creating and altering topics using CRDs

    View full-size slide

  6. What is Apache Kafka?
    A publish/subscribe messaging system
    A data streaming platform
    A distributed, horizontally-scalable, fault-tolerant, commit log

    View full-size slide

  7. Kafka Concepts
    Producers

    View full-size slide

  8. Kafka Concepts
    Consumers

    View full-size slide

  9. Kafka Concepts
    High Availability
    Broker 1
    T1 - P1
    T1 - P2
    T2 - P1
    T2 - P2
    Broker 2
    T1 - P1
    T1 - P2
    T2 - P1
    T2 - P2
    Broker 3
    T1 - P1
    T1 - P2
    T2 - P1
    T2 - P2
    Leaders and followers spread across the cluster

    View full-size slide

  10. Kafka Concepts
    High Availability
    If a broker with leader partition goes down, a new leader partition is elected on different node
    Broker 1
    T1 - P1
    T1 - P2
    T2 - P1
    T2 - P2
    Broker 2
    T1 - P1
    T1 - P2
    T2 - P1
    T2 - P2
    Broker 3
    T1 - P1
    T1 - P2
    T2 - P1
    T2 - P2

    View full-size slide

  11. Kafka on OpenShift
    • As more application workloads move to OpenShift, it makes sense to bring Kafka to the
    same environment
    • Serve as the foundation for event-driven microservices
    • Benefit from OpenShift core strengths
    • However Kafka is stateful which requires:
    • a stable broker identity
    • a way for the brokers to discover each other on the network
    • durable broker state (i.e., the messages)
    • the ability to recover broker state after a failure
    • Kubernetes primitives help but still not easy

    View full-size slide

  12. Stateful Sets and Persistent Volumes
    ● Description:
    ○ Provides an identity to each pod of the set
    that corresponds to that pod’s persistent
    volume(s)
    ○ If a StatefulSet pod is lost, a new pod with
    the same virtual identity is reinstated and
    the associated storage is reattached
    ● Benefits
    ○ Alleviate complex, state-related problems
    ○ Automation of manual process
    ○ Easy to run stateful applications at scale

    View full-size slide

  13. The Operator pattern
    ● Operator: application used to create, configure and
    manage other complex applications
    ○ Contains domain-specific operational knowledge
    ● Based on Custom Resource Definitions (CRDs)
    ○ Extends the the Kubernetes native resource API
    ○ User describes the desired state
    ○ Controller applies this state to the application
    ● It watches the *desired* state and the *actual* state
    and makes forward progress to reconcile
    ○ This is how Kubernetes works too
    Observe
    Analyze
    Act

    View full-size slide

  14. Strimzi: Provisioning Kafka on Kubernetes
    What is Strimzi ?
    ● Open source project focused on running Apache Kafka on Kubernetes and OpenShift
    ● Available as a part of Red Hat AMQ
    ● Licensed under Apache License 2.0
    ● Web site: http://strimzi.io/
    ● GitHub: https://github.com/strimzi
    ● Slack: strimzi.slack.com
    ● Mailing list: [email protected]
    ● Twitter: @strimziio

    View full-size slide

  15. AMQ Streams Operators
    Cluster
    Operator
    Kafka CR
    Kafka
    Zookeeper
    Deploys & manages
    cluster
    Topic
    Operator
    User
    Operator
    Topic CR
    User CR
    Manages
    topics & users

    View full-size slide

  16. Lab Environment
    OpenShift
    Workstation
    SSH
    Web Console
    Lab
    Machine
    CLI

    View full-size slide

  17. Activation key: amqs-ocp
    OpenShift: master00-.generic.opentlc.com
    User: admin
    Password: r3dh4t1!
    Workstation: workstation-.rhpds.opentlc.com
    User: lab-user
    Password: r3dh4t1! (should not be necessary)
    https://github.com/RedHatWorkshops/workshop-amq -streams

    View full-size slide

  18. Module 2: Internal/External access
    and security

    View full-size slide

  19. LABS
    1. Accessing the cluster from inside and outside OpenShift
    a. Configuration options for internal and external access
    b. Understand the underlying OpenShift resources
    i. Services
    ii. Routes
    2. Managing security
    a. Setting up secure clusters
    b. Managing users and resources with CRDs

    View full-size slide

  20. Kafka Concepts
    How clients interact with brokers
    Broker 1
    T1 - P1
    T1 - P2
    T2 - P1
    T2 - P2
    Broker 2
    T1 - P1
    T1 - P2
    T2 - P1
    T2 - P2
    Broker 3
    T1 - P1
    T1 - P2
    T2 - P1
    T2 - P2
    Producer P2
    Consumer C3
    Consumer C1
    Producer P1
    Consumer C2

    View full-size slide

  21. OPENSHIFT TECHNICAL OVERVIEW
    21
    services provide internal load-balancing and service discovery
    (illustrate the use of services for intra-cluster access)
    POD
    CONTAINER
    POD
    CONTAINER
    POD
    CONTAINER
    BACKEND SERVICE
    POD
    CONTAINER
    role: backend
    role: backend
    role: backend
    role: backend
    role: frontend
    10.110.1.11 10.120.2.22 10.130.3.33
    10.140.4.44
    172.30.170.110

    View full-size slide

  22. OPENSHIFT TECHNICAL OVERVIEW
    22
    POD
    routes add services to the external load-balancer and provide external urls
    (show how routes are used for external cluster access)
    CONTAINER
    POD
    CONTAINER
    POD
    CONTAINER
    BACKEND SERVICE
    ROUTE
    app-prod.mycompany.com
    > curl http://app-prod.mycompany.com

    View full-size slide

  23. Kafka Users and ACL
    lines
    ACL
    Producer Consumer
    secure-topic-writer
    User CR
    secure-topic-reader
    User CR
    users
    Access rules
    secret secret
    Cluster

    View full-size slide

  24. Module 3: Replication and Monitoring

    View full-size slide

  25. LABS
    1. Replication with MirrorMaker
    a. Setting up an additional target cluster
    b. Configuring MirrorMaker to copy data
    2. Monitoring
    a. Exporting metrics for Prometheus
    b. Visualizing cluster metrics with Grafana

    View full-size slide

  26. MirrorMaker overview
    MirrorMaker
    production-ready production-ready-target
    lines lines
    Consumer
    Producer
    Cluster Cluster

    View full-size slide

  27. Kafka & Prometheus overview
    Kafka Pods Zookeeper Pods
    Prometheus
    prometheus-jmx-exporter prometheus-jmx-exporter
    JVM
    JVM
    Grafana

    View full-size slide