Building Event-Driven Apps on Kubernetes with Kafka

Event-driven and reactive architectures offer key advantages over HTTP/REST, such as resiliency, elasticity and extensibility. Kubernetes is the de facto platform for running microservices today, and the emerging Kubernetes Event-Driven Autoscaling (KEDA) project lets you scale your containers based on event-driven metrics from Kafka. In this session I’ll show how to build an event-driven application on Kubernetes, using Kafka as the event backbone, all scaled using KEDA for improved resilience. We'll also look at Knative Eventing as an emerging framework for building event-driven apps on Kubernetes.

Transcript

1. Building event-driven apps with Kafka on Kubernetes — Ian Partridge
2. Ian Partridge • Senior engineer in IBM Hybrid Cloud • Background • Runtime systems development • Open source software • Cloud & Kubernetes Twitter: @alfa GitHub: @ianpartridge 2
3. Agenda • Microservice architectures • REST • Event-driven • KEDA • Coffeeshop demo • Knative Eventing • Blockchain demo 3
4. Microservice Architecture A microservice architecture structures an application as a collection of services that are: • Highly maintainable and testable • Loosely coupled • Independently deployable containers • Organized around business capabilities • Owned by a small team 4
5. REST Microservices • REpresentational State Transfer became the de facto way of building microservices • Microservices communicate over HTTP(S) • Client/server model • REST APIs use standard verbs: • GET, POST, PUT, DELETE • JSON became the standard data format • REST APIs are documented using OpenAPI • Standard description format 5 $ http swapi.dev/api/planets/1/ HTTP/1.0 200 OK Content-Type: application/json { "climate": "Arid", "diameter": "10465", "gravity": "1 standard", "name": "Tatooine", "orbital_period": "304", "population": "200000", "residents": [ "https://swapi.dev/api/people/1/", "https://swapi.dev/api/people/2/", ... ], "rotation_period": "23", "surface_water": "1", "terrain": "Dessert", "url": "https://swapi.dev/api/planets/1/" }
6. Event-driven Architectures • Instead of command/response, focus on events • An event is a record of fact • Events can be transmitted via messages • Messages can be stored durably • Push vs pull • Messages can be multiplexed 6
7. Event-Driven Advantages 7 µS µS µS µS REST Resilience Failure of µService causes failure of request Event backbone stores state enabling requests to continue after µService restart Data Push µS µS µS µS REST Data becomes immediately available to Client Client must poll to discover data/state changes Long Running Requests Edge Data Sync µS µS µS µS REST Network availability determines request success Requests queued until network availability HTTP requests timeout at 100 seconds* “Workaround” to return a request ID µS µS µS µS REST Client must poll against request ID to discover request result Data becomes available to client once request completes
8. Event-Driven Advantages 8 Multiple Clients µS µS µS µS REST Clients must poll for data changes or be called one at a time - any outage during calls means a notification is lost µS REST µS One produced event is consumed by two clients Stream Processing Intelligent Applications (ML) µService receives continuous stream of data and has access to history µS µS REST µS µS µService has to be repeatedly called with new data and manage/store history µS µS REST µS µS ML ML
10. 10 • Joint Microsoft / Red Hat project • v1.0 for KubeCon 2019 ( https://youtu.be/ZK2SS_GXF-g ) • Donated to CNCF for incubation • Headed for OpenShift: • “Our goal is to continue making KEDA a first class citizen in OpenShift, easy to consume and install, so you won’t need to be an expert and can focus on your application, freeing developers from infrastructure concerns. With the work being done to build a Kubernetes Operator for KEDA using the Operator SDK, and the built-in support for Operators in OpenShift 4 we are making the installation and management of KEDA in OpenShift easier.” • https://www.redhat.com/en/blog/celebrating-keda-10-providing-event-driven-scale-capability-any- container-workload Kubernetes Event-driven Autoscaling (KEDA)
11. 11 • 2 roles: • Activate/deactivate Deployment for scale to zero • Expose event metrics (like queue length) to the HPA • Lightweight • 2 CRDs (ScaledObject, TriggerAuthentication) • Deploy via Helm or YAML • Lots of scalars • Kafka, AWS SQS, Azure Event Hub, GCP Pub/Sub, Liiklus, NATS, Prometheus, RabbitMQ, Redis Lists Kubernetes Event-driven Autoscaling (KEDA)
12. 12 Coffeeshop scenario Coffeeshop UI Kafka event bus HTTP Barista Kafka Baristas REST over HTTP * Kafka pub/sub occurs using MicroProfile Reactive Messaging Kafka pub/sub Kafka pub/sub https://github.com/ianpartridge/coffeeshop-demo
14. Knative • Knative Serving • Run serverless containers on Kubernetes with ease, Knative takes care of the details of networking, autoscaling (even to zero), and revision tracking. • Knative Eventing • Universal subscription, delivery, and management of events. Build modern apps by attaching compute to a data stream with declarative event connectivity and developer- friendly object model. 14
15. Knative Eventing • Knative Eventing services are loosely coupled • Events can be transient or durable • Integrates with Knative Services and k8s Services • Key terminology • Broker - mediates events, forwarding and persisting as appropriate • Trigger - filter on events • Service - receive events 15
16. Blockchain Scenario 16 event-source Broker Trigger event-display
17. Blockchain Scenario 17 event-source Broker Trigger event-display Trigger classified- display Trigger classifier