@systemcraftsman MicroProfile Reactive Messaging with Quarkus and Apache Kafka Aykut M. Bulgu Trainer & Instructor | Technology Consultant www.SystemCraftsman.com

@systemcraftsman Me as Code #oc apply -f aykutbulgu.yaml apiVersion: redhat/v3.2 kind: Services Content Architect metadata: name: Aykut Bulgu namespace: Red Hat Global Learning Services annotations: twitter: @systemcraftsman email: - [email protected] - [email protected] organizer: Software Craftsmanship Turkey founder: System Craftsman labels: married: yes children: 1 interests: - tech (mostly kafka) - aikido - gamification - stoicism spec: replicas: 1 containers: - image: aykut:latest

@systemcraftsman The Specification

@systemcraftsman MicroProfile Open-source community specification for Cloud-Native Java applications. A community of individuals, organizations, and vendors collaborating.

@systemcraftsman MicroProfile 4.0

@systemcraftsman Why this specification?

@systemcraftsman Monoliths https://speakerdeck.com/mbogoevici/data-strategies-for-microservice-architectures?slide=4

@systemcraftsman (Micro)Services & Data Problem https://speakerdeck.com/mbogoevici/data-strategies-for-microservice-architectures?slide=5

@systemcraftsman Request-Driven Architecture Strong Coupling

@systemcraftsman Request-Driven Architecture Latency Availability Performance No fault tolerance Tightly coupled

@systemcraftsman Event-Driven Architecture and Messaging

@systemcraftsman Reactive Systems

@systemcraftsman What is wrong with JMS and Message Driven Beans? Heavyweight (mostly because of XA) Incompatible with popular brokers like Kafka No support for asynchronous IO

@systemcraftsman Concepts

@systemcraftsman Concepts

@systemcraftsman Overall Architecture

@systemcraftsman Concepts Internal Channels: In-memory messaging that creates a reactive chain of processing of beans External Channels: Channels are to be connected to external systems like Kafka or AMQP broker

@systemcraftsman Concepts Internal Channels: In-memory messaging that creates a reactive chain of processing of beans External Channels: Channels are to be connected to external systems like Kafka or AMQP broker External channels Internal channels

@systemcraftsman Concepts org.eclipse.microprofile.reactive.messaging.Message

@systemcraftsman Concepts org.eclipse.microprofile.reactive.messaging.Incoming

@systemcraftsman Concepts org.eclipse.microprofile.reactive.messaging.Acknowledgment

@systemcraftsman Concepts org.eclipse.microprofile.reactive.messaging.Outgoing

@systemcraftsman Concepts org.eclipse.microprofile.reactive.messaging.Emitter

@systemcraftsman Concepts org.eclipse.microprofile.reactive.messaging.connector.IncomingConnectorFactory org.eclipse.microprofile.reactive.messaging.connector.OutgoingConnectorFactory mp.messaging.[incoming | outgoing].[channel-name].connector=[connector-name] mp.messaging.[incoming | outgoing].[channel-name].[attribute]=[value] application.properties Java

@systemcraftsman The Implementation

@systemcraftsman Implementations Lightbend Alpakka SmallRye Reactive Messaging Open Liberty 19.0.0.9+ usage (via SmallRye Reactive Messaging)

@systemcraftsman SmallRye Reactive Messaging

@systemcraftsman Configuring for Kafka kafka.bootstrap.servers=localhost:9092 #mp.messaging.incoming.prices.bootstrap.servers=localhost:9092 mp.messaging.incoming.prices.connector=smallrye-kafka mp.messaging.outgoing.prices.connector=smallrye-kafka application.properties

@systemcraftsman Receiving Messages from Kafka

@systemcraftsman Configuration kafka.bootstrap.servers=localhost:9092 mp.messaging.incoming.prices.connector=smallrye-kafka mp.messaging.incoming.prices.topic=prices application.properties

@systemcraftsman Using @Incoming import org.eclipse.microprofile.reactive.messaging.Incoming; import javax.enterprise.context.ApplicationScoped; @ApplicationScoped public class PriceConsumer { @Incoming("prices") public void consume(double price) { // process your price. } } Java file

@systemcraftsman Using @Incoming @Incoming("prices") public CompletionStage consume(Message msg) { // access record metadata var metadata = msg.getMetadata(IncomingKafkaRecordMetadata.class).orElseThrow(); // process the message payload. double price = msg.getPayload(); // Acknowledge the incoming message (commit the offset) return msg.ack(); } Java file

@systemcraftsman Using @Channel import io.smallrye.mutiny.Multi; import io.smallrye.reactive.messaging.annotations.Channel; ...some imports omitted... import org.jboss.resteasy.annotations.SseElementType; @Path("/prices") public class PriceResource { @Inject @Channel("prices") Multi prices; @GET @Path("/prices") @Produces(MediaType.SERVER_SENT_EVENTS) @SseElementType("text/plain") public Multi stream() { return prices; } } Java file

@systemcraftsman Using @Channel @Inject @Channel("prices") Multi streamOfPayloads; @Inject @Channel("prices") Multi> streamOfMessages; @Inject @Channel("prices") Publisher publisherOfPayloads; @Inject @Channel("prices") Publisher> publisherOfMessages; Java file

@systemcraftsman Acknowledgment Strategies Acknowledgment.Strategy.PRE_PROCESSING Acknowledgment.Strategy.POST_PROCESSING Acknowledgment.Strategy.NONE Acknowledgment.Strategy.MANUAL

@systemcraftsman Acknowledgment Strategies @Incoming("prices") @Acknowledgment(Acknowledgment.Strategy.PRE_PROCESSING) public void process(double price) { // process price } @Incoming("prices") public CompletionStage process(Message msg) { // process price return msg.ack(); } Java file Java file

@systemcraftsman Commit Strategies throttled keeps track of received messages and commits an offset of the latest acked message in sequence (meaning, all previous messages were also acked). This is the default if enable.auto.commit is not explicitly set to true. latest commits the record offset received by the Kafka consumer as soon as the associated message is acknowledged (if the offset is higher than the previously committed offset). ignore performs no commit. This strategy is the default strategy when the consumer is explicitly configured with enable.auto.commit to true.

@systemcraftsman Error Handling fail fail the application, no more records will be processed (default strategy). The offset of the record that has not been processed correctly is not committed. ignore the failure is logged, but the processing continue. The offset of the record that has not been processed correctly is committed. dead-letter-queue the offset of the record that has not been processed correctly is committed, but the record is written to a Kafka dead letter topic.

@systemcraftsman For High Throughput Use the throttled policy Or set enable.auto.commit to true and use @Acknowledgment(Acknowledgment.Strategy.NONE).

@systemcraftsman Consumer Groups Single consumer thread inside a consumer group Multiple consumer threads inside a consumer group Multiple consumer applications inside a consumer group Pub/Sub: Multiple consumer groups subscribed to a topic 4 Patterns for consuming:

@systemcraftsman Consumer Groups Single consumer thread inside a consumer group quarkus.application.name or kafka.group.id

@systemcraftsman Consumer Groups Multiple consumer threads inside a consumer group mp.messaging.incoming.[channel-name].partitions

@systemcraftsman Consumer Groups Multiple consumer applications inside a consumer group mp.messaging.incoming.[channel-name].group.id

@systemcraftsman Consumer Groups Pub/Sub: Multiple consumer groups subscribed to a topic mp.messaging.incoming.somechannel.group.id=someid mp.messaging.incoming.somechannel.group.id=someotherid

@systemcraftsman Sending Messages to Kafka

@systemcraftsman Configuration kafka.bootstrap.servers=localhost:9092 mp.messaging.outgoing.prices.connector=smallrye-kafka mp.messaging.outgoing.prices.topic=prices application.properties

@systemcraftsman Configuration kafka.bootstrap.servers=localhost:9092 mp.messaging.incoming.prices.connector=smallrye-kafka mp.messaging.incoming.prices.topic=prices mp.messaging.outgoing.prices-out.connector=smallrye-kafka mp.messaging.outgoing.prices-out.topic=prices application.properties

@systemcraftsman Using @Outgoing import io.smallrye.mutiny.Multi; import org.eclipse.microprofile.reactive.messaging.Outgoing; import javax.enterprise.context.ApplicationScoped; import java.time.Duration; import java.util.Random; @ApplicationScoped public class KafkaPriceProducer { private final Random random = new Random(); @Outgoing("prices") public Multi generate() { // Build an infinite stream of random prices // It emits a price every second return Multi.createFrom().ticks().every(Duration.ofSeconds(1)) .map(x -> random.nextDouble()); } } Java file

@systemcraftsman Using @Outgoing @Outgoing("prices") public Multi> generate() { return Multi.createFrom().ticks().every(Duration.ofSeconds(1)) .map(x -> Message.of(random.nextDouble()) .addMetadata(OutgoingKafkaRecordMetadata.builder() .withKey("my-key") .withTopic("my-key-prices") .withHeaders(new RecordHeaders().add("my-header", "value".getBytes())) .build())); } Java file

@systemcraftsman Using @Outgoing @Outgoing("prices") T generate(); // T excluding void @Outgoing("prices") Message generate(); @Outgoing("prices") Uni generate(); @Outgoing("prices") Uni> generate(); @Outgoing("prices") CompletionStage generate(); @Outgoing("prices") CompletionStage> generate(); Java file

@systemcraftsman Using @Emitter import org.eclipse.microprofile.reactive.messaging.Channel; import org.eclipse.microprofile.reactive.messaging.Emitter; ...some imports omitted... @Path("/prices") public class PriceResource { @Inject @Channel("prices") Emitter priceEmitter; @POST @Consumes(MediaType.TEXT_PLAIN) public void addPrice(Double price) { CompletionStage ack = priceEmitter.send(price); } } Java file

@systemcraftsman Using @Emitter import java.util.concurrent.CompletableFuture; import org.eclipse.microprofile.reactive.messaging.Channel; import org.eclipse.microprofile.reactive.messaging.Emitter; ...some imports omitted... @Path("/prices") public class PriceResource { @Inject @Channel("prices") Emitter priceEmitter; @POST @Consumes(MediaType.TEXT_PLAIN) public void addPrice(Double price) { priceEmitter.send(Message.of(price) .withAck(() -> { // Called when the message is acked return CompletableFuture.completedFuture(null); }) .withNack(throwable -> { // Called when the message is nacked return CompletableFuture.completedFuture(null); })); } } Java file

@systemcraftsman Write Acknowledgement mp.messaging.outgoing.[channel-name].acks=[0 | 1 | all] mp.messaging.outgoing.[channel-name].waitForWriteCompletion=[true* | false]

@systemcraftsman Retries mp.messaging.outgoing.[channel-name].retries=[value] mp.messaging.outgoing.[channel-name].retry.backoff.ms=[value] mp.messaging.outgoing.[channel-name].delivery.timeout.ms=[value] mp.messaging.outgoing.[channel-name].max.in.flight.requests.per.connection=[value]

@systemcraftsman Processing Messages

@systemcraftsman Processing Messages import org.eclipse.microprofile.reactive.messaging.Incoming; import org.eclipse.microprofile.reactive.messaging.Outgoing; import javax.enterprise.context.ApplicationScoped; @ApplicationScoped public class PriceProcessor { private static final double CONVERSION_RATE = 0.88; @Incoming("price-in") @Outgoing("price-out") public double process(double price) { return price * CONVERSION_RATE; } } Java file

@systemcraftsman Processing Messages import javax.enterprise.context.ApplicationScoped; import org.eclipse.microprofile.reactive.messaging.Incoming; import org.eclipse.microprofile.reactive.messaging.Outgoing; import io.smallrye.mutiny.Multi; @ApplicationScoped public class PriceProcessor { private static final double CONVERSION_RATE = 0.88; @Incoming("price-in") @Outgoing("price-out") public Multi process(Multi prices) { return prices.filter(p -> p > 100).map(p -> p * CONVERSION_RATE); } } Java file

@systemcraftsman Serialization/Deserialization (SerDe)

@systemcraftsman JSON public class Fruit { public String name; public int price; public Fruit() { } public Fruit(String name, int price) { this.name = name; this.price = price; } } @ApplicationScoped public class FruitProcessor { private static final double CONVERSION_RATE = 0.88; @Incoming("fruit-in") @Outgoing("fruit-out") @Broadcast public Fruit process(Fruit fruit) { fruit.price = fruit.price * CONVERSION_RATE; return fruit; } } Java file Java file

@systemcraftsman SerDe via Jackson io.quarkus quarkus-jackson # Configure the Kafka source (we read from it) mp.messaging.incoming.fruit-in.connector=smallrye-kafka mp.messaging.incoming.fruit-in.topic=fruit-in mp.messaging.incoming.fruit-in.value.deserializer=com.acme. fruit.jackson.FruitDeserializer # Configure the Kafka sink (we write to it) mp.messaging.outgoing.fruit-out.connector=smallrye-kafka mp.messaging.outgoing.fruit-out.topic=fruit-out mp.messaging.outgoing.fruit-out.value.serializer=io.quarkus .kafka.client.serialization.ObjectMapperSerializer package com.acme.fruit.jackson; import io.quarkus.kafka.client.serialization.ObjectMapperDeseri alizer; public class FruitDeserializer extends ObjectMapperDeserializer { public FruitDeserializer() { super(Fruit.class); } } Java file pom.xml application.properties

@systemcraftsman SerDe via JSON-B io.quarkus quarkus-jsonb # Configure the Kafka source (we read from it) mp.messaging.incoming.fruit-in.connector=smallrye-kafka mp.messaging.incoming.fruit-in.topic=fruit-in mp.messaging.incoming.fruit-in.value.deserializer=com.acme. fruit.jackson.FruitDeserializer # Configure the Kafka sink (we write to it) mp.messaging.outgoing.fruit-out.connector=smallrye-kafka mp.messaging.outgoing.fruit-out.topic=fruit-out mp.messaging.outgoing.fruit-out.value.serializer=io.quarkus .kafka.client.serialization.JsonbSerializer package com.acme.fruit.jsonb; import io.quarkus.kafka.client.serialization.JsonbDeserializer; public class FruitDeserializer extends JsonbDeserializer { public FruitDeserializer() { super(Fruit.class); } } application.properties pom.xml Java file

@systemcraftsman Avro io.apicurio.registry.utils.serde.AvroKafkaSerializer io.apicurio.registry.utils.serde.AvroKafkaDeserializer io.quarkus quarkus-avro io.apicurio apicurio-registry-utils-serde # Configure the Kafka source (we read from it) mp.messaging.incoming.fruit-in.connector=smallrye-kafka mp.messaging.incoming.fruit-in.topic=fruit-in mp.messaging.incoming.fruit-in.value.deserializer=io.apicur io.registry.utils.serde.AvroKafkaDeserializer # Configure the Kafka sink (we write to it) mp.messaging.outgoing.fruit-out.connector=smallrye-kafka mp.messaging.outgoing.fruit-out.topic=fruit-out mp.messaging.outgoing.fruit-out.value.serializer=io.apicuri o.registry.utils.serde.AvroKafkaSerializer application.properties pom.xml

@systemcraftsman Talk is Cheap, Show me the Demo!

@systemcraftsman Resources ● MicroProfile Reactive Messaging Spec 2.0 ● Quarkus Apache Kafka Reference Guide ● SmallRye Reactive Messaging ● Twitter: @systemcraftsman ● Email: [email protected] ● Demo: https://github.com/RedHatTraining/AD482-ToT-CoffeeShop

@systemcraftsman Thank You! www.SystemCraftsman.com