$30 off During Our Annual Pro Sale. View Details »

Running Apache Kafka on Red Hat Openshift with AMQ Streams

Running Apache Kafka on Red Hat Openshift with AMQ Streams

Lab delivered at Red Hat Summit 2019

Marius Bogoevici

May 08, 2019

More Decks by Marius Bogoevici

Other Decks in Technology



    Marius Bogoevici Paolo Patierno Gunnar Morling Emmanuel Bernard
  2. AGENDA Running a Kafka cluster on OpenShift Managing access and

    security Replication and monitoring Goal: Learn the practical aspects of deploying and operating Kafka clusters on OpenShift
  3. Lab Environment OpenShift Workstation SSH Web Console Lab Machine CLI

  4. Module 1: Deploying and managing Kafka clusters

  5. LABS 1. AMQ Streams on OpenShift from 0 to 60

    a. Deploying the operator and a minimal cluster 2. Production-ready topologies a. Deploying persistent, scaled-up clusters b. Scaling clusters 3. Managing Topics a. Creating and altering topics using CRDs
  6. What is Apache Kafka? A publish/subscribe messaging system A data

    streaming platform A distributed, horizontally-scalable, fault-tolerant, commit log
  7. Kafka Concepts Producers

  8. Kafka Concepts Consumers

  9. Kafka Concepts High Availability Broker 1 T1 - P1 T1

    - P2 T2 - P1 T2 - P2 Broker 2 T1 - P1 T1 - P2 T2 - P1 T2 - P2 Broker 3 T1 - P1 T1 - P2 T2 - P1 T2 - P2 Leaders and followers spread across the cluster
  10. Kafka Concepts High Availability If a broker with leader partition

    goes down, a new leader partition is elected on different node Broker 1 T1 - P1 T1 - P2 T2 - P1 T2 - P2 Broker 2 T1 - P1 T1 - P2 T2 - P1 T2 - P2 Broker 3 T1 - P1 T1 - P2 T2 - P1 T2 - P2
  11. Kafka on OpenShift • As more application workloads move to

    OpenShift, it makes sense to bring Kafka to the same environment • Serve as the foundation for event-driven microservices • Benefit from OpenShift core strengths • However Kafka is stateful which requires: • a stable broker identity • a way for the brokers to discover each other on the network • durable broker state (i.e., the messages) • the ability to recover broker state after a failure • Kubernetes primitives help but still not easy
  12. Stateful Sets and Persistent Volumes • Description: ◦ Provides an

    identity to each pod of the set that corresponds to that pod’s persistent volume(s) ◦ If a StatefulSet pod is lost, a new pod with the same virtual identity is reinstated and the associated storage is reattached • Benefits ◦ Alleviate complex, state-related problems ◦ Automation of manual process ◦ Easy to run stateful applications at scale
  13. The Operator pattern • Operator: application used to create, configure

    and manage other complex applications ◦ Contains domain-specific operational knowledge • Based on Custom Resource Definitions (CRDs) ◦ Extends the the Kubernetes native resource API ◦ User describes the desired state ◦ Controller applies this state to the application • It watches the *desired* state and the *actual* state and makes forward progress to reconcile ◦ This is how Kubernetes works too Observe Analyze Act
  14. Strimzi: Provisioning Kafka on Kubernetes What is Strimzi ? •

    Open source project focused on running Apache Kafka on Kubernetes and OpenShift • Available as a part of Red Hat AMQ • Licensed under Apache License 2.0 • Web site: http://strimzi.io/ • GitHub: https://github.com/strimzi • Slack: strimzi.slack.com • Mailing list: strimzi@redhat.com • Twitter: @strimziio
  15. AMQ Streams Operators Cluster Operator Kafka CR Kafka Zookeeper Deploys

    & manages cluster Topic Operator User Operator Topic CR User CR Manages topics & users
  16. Lab Environment OpenShift Workstation SSH Web Console Lab Machine CLI

  17. Activation key: amqs-ocp OpenShift: master00-<guid>.generic.opentlc.com User: admin Password: r3dh4t1! Workstation:

    workstation-<guid>.rhpds.opentlc.com User: lab-user Password: r3dh4t1! (should not be necessary) https://github.com/RedHatWorkshops/workshop-amq -streams
  18. Module 2: Internal/External access and security

  19. LABS 1. Accessing the cluster from inside and outside OpenShift

    a. Configuration options for internal and external access b. Understand the underlying OpenShift resources i. Services ii. Routes 2. Managing security a. Setting up secure clusters b. Managing users and resources with CRDs
  20. Kafka Concepts How clients interact with brokers Broker 1 T1

    - P1 T1 - P2 T2 - P1 T2 - P2 Broker 2 T1 - P1 T1 - P2 T2 - P1 T2 - P2 Broker 3 T1 - P1 T1 - P2 T2 - P1 T2 - P2 Producer P2 Consumer C3 Consumer C1 Producer P1 Consumer C2
  21. OPENSHIFT TECHNICAL OVERVIEW 21 services provide internal load-balancing and service

    discovery (illustrate the use of services for intra-cluster access) POD CONTAINER POD CONTAINER POD CONTAINER BACKEND SERVICE POD CONTAINER role: backend role: backend role: backend role: backend role: frontend
  22. OPENSHIFT TECHNICAL OVERVIEW 22 POD routes add services to the

    external load-balancer and provide external urls (show how routes are used for external cluster access) CONTAINER POD CONTAINER POD CONTAINER BACKEND SERVICE ROUTE app-prod.mycompany.com > curl http://app-prod.mycompany.com
  23. Kafka Users and ACL lines ACL Producer Consumer secure-topic-writer User

    CR secure-topic-reader User CR users Access rules secret secret Cluster
  24. Module 3: Replication and Monitoring

  25. LABS 1. Replication with MirrorMaker a. Setting up an additional

    target cluster b. Configuring MirrorMaker to copy data 2. Monitoring a. Exporting metrics for Prometheus b. Visualizing cluster metrics with Grafana
  26. MirrorMaker overview MirrorMaker production-ready production-ready-target lines lines Consumer Producer Cluster

  27. Kafka & Prometheus overview Kafka Pods Zookeeper Pods Prometheus prometheus-jmx-exporter

    prometheus-jmx-exporter JVM JVM Grafana
  28. None