Kafka Streams from the grounds up to the Cloud

[Talk given at Spring One Platform, Dec 5, 2017, San Francisco]

In this session we will introduce the Kafka Streams API and the Kafka Streams processing engine, followed by the Kafka Streams support in the Spring portfolio - showing how to easily write and deploy Kafka Streams applications using Spring Cloud Stream and deploy them on various cloud platforms using Spring Cloud Data Flow.

Transcript

1. Kafka Streams: From the grounds up to the Cloud Marius Bogoevici, Chief Architect, Red Hat Spring One Platform, Dec 4, 2017 @mariusbogoevici
2. Marius Bogoevici • Chief Architect, Data Streaming at Red Hat • Spring ecosystem contributor since 2008 ◦ Spring Integration • Spring team member between 2014 and 2017 ◦ Spring XD, Spring Integration Kafka ◦ Spring Cloud Stream project lead • Co-author “Spring Integration in Action”, Manning, 2012
3. Kafka: from messaging system to streaming platform (based on https://www.confluent.io/blog/apache-kafka-goes-1-0/) Distributed log Replication, Fault tolerance Connect and Streams Transactions, Exactly once
4. How about applications that are both producers and consumers and perform complex computations? Kafka as a distributed messaging system
5. Kafka Streams • Client library for stream processing ◦ Embed stream processing features into regular Java applications (microservice model) ◦ Create sophisticated topologies of independent applications • Functional transformations via DSL: ◦ Mapping, filtering, flatMap ◦ Aggregation, joins (multiple topics) ◦ Windowing • Kafka-to-Kafka semantics • One-record-at-a-time processing (no microbatching) • Stateful processing support • Transactions/exactly once Kafka Cluster Application Kafka Streams
6. Kafka Streams - important concepts • KStream ◦ Record stream abstraction ◦ Read from/written to external topic or produced from other KStream via operators such as map/filter • KTable/GlobalKTable ◦ Changelog stream abstraction (key is meaningful) ◦ Read from external topic as a sequence of updates ◦ Produced from other tables or stream joins, aggregations etc • State Store ◦ Key-value store for intermediate aggregation data, KTable materialized views, arbitrary key-value data produced during ◦ Replicated externally • Time windowing
7. Kafka Streams - high level DSL KStream words = builder.stream(“words”) KTable countsTable = words.flatMapValues(value -> Arrays.asList(value.toLowerCase().split("\\W+"))) .map((key, value) -> new KeyValue<>(value, value)) .groupByKey(Serdes.String(), Serdes.String()) .count(timeWindows, "WordCounts"); KStream counts = counts.toStream() counts.to(“counts”)
8. Kafka Streams stateful processing (default stores) Kafka Cluster Application Kafka Streams In-memory state store Local disk • Pluggable state store model • Key-value data store • Default strategy: ◦ In-memory (fast access) ◦ Local disk (for fast recovery) ◦ Replicated to Kafka (for resilience) • Tightly integrated with Kafka: state updates are correlated with offset commits changelog
9. Spring Cloud Stream • Event-driven microservice framework • Developer focus on writing business code • Middleware-agnostic programming model • Binders: ◦ Kafka ◦ RabbitMQ ◦ AWS Kinesis ◦ Google Pub Sub ◦ Apache Artemis (community) • Easy to deploy with Spring Cloud Data Flow
10. Spring Cloud Stream KStream Processor (since 1.3) counts words Spring Cloud Stream KStream API output input Spring Boot Programming model (developer focus) Application model (configuration options, StreamConfig based on Spring Boot properties, KStreamBuilder, KStream binder) Externalized configuration, uberjar construction, health monitoring endpoints
11. Kafka Streams in the Cloud Application Kafka Streams Application Kafka Streams Docker uberjar Spring Cloud Data Flow
12. Kafka Streams stateful and stateless deployments Kafka Cluster Application Kafka Streams In-memory state store Local disk • Changes propagated to changelog topic • Stored locally for recovery/restart • Fully stateless deployments require to replay the topic on restart/failover • State store recovery can be optimized by providing access to stateful deployments changelog
13. Kafka Streams with Kubernetes StatefulSets Application Kafka Streams Pod Application Kafka Streams Pod Application Kafka Streams Pod volume-word-count-0 word-count-1 word-count-2 volume-word-count-1 volume-word-count-2 word-count-0
14. Demo time! twitter:@mariusbogoevici Email: mariusb@redhat.com http://cloud.spring.io/spring-cloud-stream/ https://github.com/EnMasseProject/barnabas