Event-driven microservices with Apache Kafka

Running event-driven microservices with Apache Kafka, on Kubernetes and Openshift. The talk presents Strimzi, which provides a "Kubernetes Operator" to manage your Apache Kafka cluster, on Kubernetes or Openshift!

http://strimzi.io/

Materials: https://github.com/matzew/kafka-presentation

Transcript

1. Event-driven microservices with Apache Kafka RH_PREStemp_light_v2_0816 1 Matthias Wessendorf Principal Software Engineer @mwessendorf
2. Enterprise Java Standards History J2EE 1.2 J2EE 1.3 J2EE 1.4 Java EE 5 Java EE 6 Java EE 7 Java EE 8 2000 2005 2010 2015 2020 Release Cadence
3. MicroProfile Background • Began as a collection of independent discussions ◦ Many innovative “microservices” efforts in existing Java EE projects ▪ WildFly Swarm (NOW: Thorntail) ▪ WebSphere Liberty ▪ Payara ▪ TomEE ◦ Projects already leveraging both Java EE and non-Java EE technologies ◦ Creating new features/capabilities to address microservices architectures • Quickly realized there is common ground • Java EE technologies are already being used for microservices, but we can do better
4. MicroProfile Release Philosophy Release 1.0 JAX-RS CDI JSON-P Build consensus Standardize Rapidly iterate and innovate Sept 2016
5. MicroProfile Release Status
6. What is Apache Kafka? A publish/subscribe messaging system? A streaming data platform? A distributed, horizontally-scalable, fault-tolerant, commit log?
7. DEMO : WebSocket to Kafka
8. Apache Kafka Concepts •Messages are sent to and received from a topic • Topics are split into one or more partitions (aka shards) • All actual work is done on partition level, topic is just a virtual object •Each message is written only into a one selected partition • Partitioning is usually done based on the message key • Message ordering within the partition is fixed •Retention • Based on size / message age • Compacted based on message key
9. Kafka concepts Topics & partitions with Producers old new 0 1 2 3 4 5 6 7 8 9 1 0 1 1 0 1 2 3 4 5 6 0 1 2 3 4 5 6 7 8 9 1 0 Producer Partition 0 Partition 1 Partition 2
10. Kafka concepts Topics & partitions with Consumers old new 0 1 2 3 4 5 6 7 8 9 1 0 1 1 0 1 2 3 4 5 6 0 1 2 3 4 5 6 7 8 9 1 0 Consumer Partition 0 Partition 1 Partition 2
11. Kafka concepts Consumer Groups Group1 C1 C2 Group2 C3 C4 C5 C6 Broker 1 T1 - P1 T1 - P2 Broker 2 T1 - P3 T1 - P4 Cluster •Logical grouping of consumers • The group receives the message... • Consumer might have a partition assigned •Separate scaling of groups • Scaling on use-case… ▪ Non-time-sensitive (down) ▪ Time-sensitive (up)
12. Kafka concepts High availability Broker 1 T1 - P1 T1 - P2 T2 - P1 T2 - P2 Broker 2 T1 - P1 T1 - P2 T2 - P1 T2 - P2 Broker 3 T1 - P1 T1 - P2 T2 - P1 T2 - P2 Leaders and followers spread across the cluster
13. Kafka concepts High availability Broker 1 T1 - P1 T1 - P2 T2 - P1 T2 - P2 Broker 2 T1 - P1 T1 - P2 T2 - P1 T2 - P2 Broker 3 T1 - P1 T1 - P2 T2 - P1 T2 - P2 If a broker with leader partition goes down, a new leader partition is elected on different node
14. DEMO : WebSocket behind the scenes The JAVA API
15. Apache Kafka on OpenShift What is Strimzi ? • Open source project focused on running Apache Kafka on Kubernetes and OpenShift • Provides: ◦ Docker images for running Apache Kafka and Zookeeper ◦ Tooling for managing and configuring Apache Kafka clusters and topics • Follows the Kubernetes “operator” model • Licensed under Apache License 2.0 • Web site: http://strimzi.io/ • GitHub: https://github.com/strimzi • Slack: strimzi.slack.com • Mailing list: strimzi@redhat.com • Twitter: @strimziio
16. Apache Kafka on OpenShift The challenges • Apache Kafka is *stateful* which means we require … ◦ … a stable broker identity ◦ … a way for the brokers to discover each other on the network ◦ … durable broker state (i.e., the messages) ◦ … the ability to recover broker state after a failure • All the above are true for Apache Zookeeper as well • StatefulSets, PersistentVolumeClaims, Services can help but …
17. It’s not easy!
18. Strimzi on OpenShift Goals • Simplifying the Apache Kafka deployment on OpenShift • Using the OpenShift native mechanisms for... ◦ Provisioning the cluster ◦ Managing the topics • … thereby removing the need to use Kafka command-line tools • Providing a better integration with applications running on OpenShift ◦ microservices, data streaming, event-sourcing, etc.
19. Strimzi on OpenShift The “Operator” model • An application used to create, configure and manage other complex applications ◦ Contains specific domain / application knowledge • Controller operates based on input from Config Maps or Custom Resource Definitions ◦ User describes the desired state ◦ Controller applies this state to the application • It watches the *desired* state and the *actual* state … ◦ … taking appropriate actions Observe Analyze Act
20. Strimzi on OpenShift Config Map versus Custom Resource Definitions • Operator are currently using Config Maps for configuration ◦ Main advantage of Config Maps is no need for special permissions to install Strimzi/AMQ Streams on OpenShift • CRDs have some advantages as well ◦ Flexible data structure ◦ Possibility to set permissions for the CRD resources • Support for CRDs is on backlog for the future
21. Cluster Operator Creating and managing Apache Kafka clusters Zookeeper Kafka Cluster Operator Config Map Manages
22. DEMO : CLUSTER DEPLOYMENT
23. Topic Operator Creating and managing Kafka topics Zookeeper Kafka Topic Operator Config Map Manages topics
24. Topic Operator Creating and managing Kafka topics Zookeeper (Topic Operator’s own storage) Kafka topics Topic Operator (3-way diff) Config Map
25. DEMO : TOPICS MANAGEMENT
26. Apache Kafka and Microprofile CDI to the rescue • Kafka library is easy to integrate • Wiring of Producers/Consumers with CDI • Contexts and Dependency Injection (CDI) for the Java EE* platform ◦ Contexts ▪ The ability to bind the lifecycle and interactions of stateful components to well-defined but extensible lifecycle contexts ◦ Dependency Injection ▪ The ability to inject components into an application in a typesafe way, including the ability to choose at deployment time which implementation of a particular interface to inject • THE FOUNDATION for frameworks, extensions and integration with other technologies!
27. Apache Kafka and Microprofile
28. Apache Kafka and Microprofile
29. Apache Kafka and Microprofile
30. DEMO : Kafka-CDI Extension
31. Outlook - KStreams API • KStream API ◦ Based on vanilla Apache Kafka client API ◦ Processor API ◦ DSL / functional programming for filter/map/reduce streams ◦ No external Streaming cluster needed ...
32. Outlook II A rich ecosystem for Apache Kafka • Eclipse Vert.x ◦ Reactive wrappers for Apache Kafka client API • Debezium.io: CDC platform ◦ Not just Apache Kafka! Gunnar knows more ◦ Contains KafkaCluster for unit tests • AMQP-Kafka-Bridge (Strimzi) • ...
33. Resources • Strimzi : http://strimzi.io/ • Apache Kafka : https://kafka.apache.org/ • Kafka-CDI: https://github.com/aerogear/kafka-cdi • Demo : https://github.com/matzew/kafka-presentation
34. THANK YOU plus.google.com/+RedHat linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHatNews 34