MicroProfile Reactive Messaging with Quarkus and Apache Kafka

JakartaOne Turkish, Aug 21st 2021

Follow me on twitter (@systemcraftsman) or subscribe to https://www.systemcraftsman.com/join/ to get any updates from me.

Transcript

1. @systemcraftsman MicroProfile Reactive Messaging with Quarkus and Apache Kafka Aykut M. Bulgu Trainer & Instructor | Technology Consultant www.SystemCraftsman.com
2. @systemcraftsman Me as Code #oc apply -f aykutbulgu.yaml apiVersion: redhat/v3.2 kind: Services Content Architect metadata: name: Aykut Bulgu namespace: Red Hat Global Learning Services annotations: twitter: @systemcraftsman email: - aykut@systemcraftsman.com - abulgu@redhat.com organizer: Software Craftsmanship Turkey founder: System Craftsman labels: married: yes children: 1 interests: - tech (mostly kafka) - aikido - gamification - stoicism spec: replicas: 1 containers: - image: aykut:latest
3. @systemcraftsman The Specification
4. @systemcraftsman MicroProfile Open-source community specification for Cloud-Native Java applications. A community of individuals, organizations, and vendors collaborating.
5. @systemcraftsman MicroProfile 4.0
6. @systemcraftsman Why this specification?
7. @systemcraftsman Monoliths https://speakerdeck.com/mbogoevici/data-strategies-for-microservice-architectures?slide=4
8. @systemcraftsman (Micro)Services & Data Problem https://speakerdeck.com/mbogoevici/data-strategies-for-microservice-architectures?slide=5
9. @systemcraftsman Request-Driven Architecture Strong Coupling
10. @systemcraftsman Request-Driven Architecture Latency Availability Performance No fault tolerance Tightly coupled
11. @systemcraftsman Event-Driven Architecture and Messaging
12. @systemcraftsman Reactive Systems
13. @systemcraftsman What is wrong with JMS and Message Driven Beans? Heavyweight (mostly because of XA) Incompatible with popular brokers like Kafka No support for asynchronous IO
14. @systemcraftsman Concepts
15. @systemcraftsman Concepts
16. @systemcraftsman Overall Architecture
17. @systemcraftsman Concepts Internal Channels: In-memory messaging that creates a reactive chain of processing of beans External Channels: Channels are to be connected to external systems like Kafka or AMQP broker
18. @systemcraftsman Concepts Internal Channels: In-memory messaging that creates a reactive chain of processing of beans External Channels: Channels are to be connected to external systems like Kafka or AMQP broker External channels Internal channels
19. @systemcraftsman Concepts org.eclipse.microprofile.reactive.messaging.Message
20. @systemcraftsman Concepts org.eclipse.microprofile.reactive.messaging.Incoming
21. @systemcraftsman Concepts org.eclipse.microprofile.reactive.messaging.Acknowledgment
22. @systemcraftsman Concepts org.eclipse.microprofile.reactive.messaging.Outgoing
23. @systemcraftsman Concepts org.eclipse.microprofile.reactive.messaging.Emitter
24. @systemcraftsman Concepts org.eclipse.microprofile.reactive.messaging.connector.IncomingConnectorFactory org.eclipse.microprofile.reactive.messaging.connector.OutgoingConnectorFactory mp.messaging.[incoming | outgoing].[channel-name].connector=[connector-name] mp.messaging.[incoming | outgoing].[channel-name].[attribute]=[value] application.properties Java
25. @systemcraftsman The Implementation
26. @systemcraftsman Implementations Lightbend Alpakka SmallRye Reactive Messaging Open Liberty 19.0.0.9+ usage (via SmallRye Reactive Messaging)
27. @systemcraftsman SmallRye Reactive Messaging
28. @systemcraftsman Configuring for Kafka kafka.bootstrap.servers=localhost:9092 #mp.messaging.incoming.prices.bootstrap.servers=localhost:9092 mp.messaging.incoming.prices.connector=smallrye-kafka mp.messaging.outgoing.prices.connector=smallrye-kafka application.properties
29. @systemcraftsman Receiving Messages from Kafka
30. @systemcraftsman Configuration kafka.bootstrap.servers=localhost:9092 mp.messaging.incoming.prices.connector=smallrye-kafka mp.messaging.incoming.prices.topic=prices application.properties
31. @systemcraftsman Using @Incoming import org.eclipse.microprofile.reactive.messaging.Incoming; import javax.enterprise.context.ApplicationScoped; @ApplicationScoped public class PriceConsumer { @Incoming("prices") public void consume(double price) { // process your price. } } Java file
32. @systemcraftsman Using @Incoming @Incoming("prices") public CompletionStage<Void> consume(Message<Double> msg) { // access record metadata var metadata = msg.getMetadata(IncomingKafkaRecordMetadata.class).orElseThrow(); // process the message payload. double price = msg.getPayload(); // Acknowledge the incoming message (commit the offset) return msg.ack(); } Java file
33. @systemcraftsman Using @Channel import io.smallrye.mutiny.Multi; import io.smallrye.reactive.messaging.annotations.Channel; ...some imports omitted... import org.jboss.resteasy.annotations.SseElementType; @Path("/prices") public class PriceResource { @Inject @Channel("prices") Multi<Double> prices; @GET @Path("/prices") @Produces(MediaType.SERVER_SENT_EVENTS) @SseElementType("text/plain") public Multi<Double> stream() { return prices; } } Java file
34. @systemcraftsman Using @Channel @Inject @Channel("prices") Multi<Double> streamOfPayloads; @Inject @Channel("prices") Multi<Message<Double>> streamOfMessages; @Inject @Channel("prices") Publisher<Double> publisherOfPayloads; @Inject @Channel("prices") Publisher<Message<Double>> publisherOfMessages; Java file
35. @systemcraftsman Acknowledgment Strategies Acknowledgment.Strategy.PRE_PROCESSING Acknowledgment.Strategy.POST_PROCESSING Acknowledgment.Strategy.NONE Acknowledgment.Strategy.MANUAL
36. @systemcraftsman Acknowledgment Strategies @Incoming("prices") @Acknowledgment(Acknowledgment.Strategy.PRE_PROCESSING) public void process(double price) { // process price } @Incoming("prices") public CompletionStage<Void> process(Message<Double> msg) { // process price return msg.ack(); } Java file Java file
37. @systemcraftsman Commit Strategies throttled keeps track of received messages and commits an offset of the latest acked message in sequence (meaning, all previous messages were also acked). This is the default if enable.auto.commit is not explicitly set to true. latest commits the record offset received by the Kafka consumer as soon as the associated message is acknowledged (if the offset is higher than the previously committed offset). ignore performs no commit. This strategy is the default strategy when the consumer is explicitly configured with enable.auto.commit to true.
38. @systemcraftsman Error Handling fail fail the application, no more records will be processed (default strategy). The offset of the record that has not been processed correctly is not committed. ignore the failure is logged, but the processing continue. The offset of the record that has not been processed correctly is committed. dead-letter-queue the offset of the record that has not been processed correctly is committed, but the record is written to a Kafka dead letter topic.
39. @systemcraftsman For High Throughput Use the throttled policy Or set enable.auto.commit to true and use @Acknowledgment(Acknowledgment.Strategy.NONE).
40. @systemcraftsman Consumer Groups Single consumer thread inside a consumer group Multiple consumer threads inside a consumer group Multiple consumer applications inside a consumer group Pub/Sub: Multiple consumer groups subscribed to a topic 4 Patterns for consuming:
41. @systemcraftsman Consumer Groups Single consumer thread inside a consumer group quarkus.application.name or kafka.group.id
42. @systemcraftsman Consumer Groups Multiple consumer threads inside a consumer group mp.messaging.incoming.[channel-name].partitions
43. @systemcraftsman Consumer Groups Multiple consumer applications inside a consumer group mp.messaging.incoming.[channel-name].group.id
44. @systemcraftsman Consumer Groups Pub/Sub: Multiple consumer groups subscribed to a topic mp.messaging.incoming.somechannel.group.id=someid mp.messaging.incoming.somechannel.group.id=someotherid
45. @systemcraftsman Sending Messages to Kafka
46. @systemcraftsman Configuration kafka.bootstrap.servers=localhost:9092 mp.messaging.outgoing.prices.connector=smallrye-kafka mp.messaging.outgoing.prices.topic=prices application.properties
47. @systemcraftsman Configuration kafka.bootstrap.servers=localhost:9092 mp.messaging.incoming.prices.connector=smallrye-kafka mp.messaging.incoming.prices.topic=prices mp.messaging.outgoing.prices-out.connector=smallrye-kafka mp.messaging.outgoing.prices-out.topic=prices application.properties
48. @systemcraftsman Using @Outgoing import io.smallrye.mutiny.Multi; import org.eclipse.microprofile.reactive.messaging.Outgoing; import javax.enterprise.context.ApplicationScoped; import java.time.Duration; import java.util.Random; @ApplicationScoped public class KafkaPriceProducer { private final Random random = new Random(); @Outgoing("prices") public Multi<Double> generate() { // Build an infinite stream of random prices // It emits a price every second return Multi.createFrom().ticks().every(Duration.ofSeconds(1)) .map(x -> random.nextDouble()); } } Java file
49. @systemcraftsman Using @Outgoing @Outgoing("prices") public Multi<Message<Double>> generate() { return Multi.createFrom().ticks().every(Duration.ofSeconds(1)) .map(x -> Message.of(random.nextDouble()) .addMetadata(OutgoingKafkaRecordMetadata.<String>builder() .withKey("my-key") .withTopic("my-key-prices") .withHeaders(new RecordHeaders().add("my-header", "value".getBytes())) .build())); } Java file
50. @systemcraftsman Using @Outgoing @Outgoing("prices") T generate(); // T excluding void @Outgoing("prices") Message<T> generate(); @Outgoing("prices") Uni<T> generate(); @Outgoing("prices") Uni<Message<T>> generate(); @Outgoing("prices") CompletionStage<T> generate(); @Outgoing("prices") CompletionStage<Message<T>> generate(); Java file
51. @systemcraftsman Using @Emitter import org.eclipse.microprofile.reactive.messaging.Channel; import org.eclipse.microprofile.reactive.messaging.Emitter; ...some imports omitted... @Path("/prices") public class PriceResource { @Inject @Channel("prices") Emitter<Double> priceEmitter; @POST @Consumes(MediaType.TEXT_PLAIN) public void addPrice(Double price) { CompletionStage<Void> ack = priceEmitter.send(price); } } Java file
52. @systemcraftsman Using @Emitter import java.util.concurrent.CompletableFuture; import org.eclipse.microprofile.reactive.messaging.Channel; import org.eclipse.microprofile.reactive.messaging.Emitter; ...some imports omitted... @Path("/prices") public class PriceResource { @Inject @Channel("prices") Emitter<Double> priceEmitter; @POST @Consumes(MediaType.TEXT_PLAIN) public void addPrice(Double price) { priceEmitter.send(Message.of(price) .withAck(() -> { // Called when the message is acked return CompletableFuture.completedFuture(null); }) .withNack(throwable -> { // Called when the message is nacked return CompletableFuture.completedFuture(null); })); } } Java file
53. @systemcraftsman Write Acknowledgement mp.messaging.outgoing.[channel-name].acks=[0 | 1 | all] mp.messaging.outgoing.[channel-name].waitForWriteCompletion=[true* | false]
54. @systemcraftsman Retries mp.messaging.outgoing.[channel-name].retries=[value] mp.messaging.outgoing.[channel-name].retry.backoff.ms=[value] mp.messaging.outgoing.[channel-name].delivery.timeout.ms=[value] mp.messaging.outgoing.[channel-name].max.in.flight.requests.per.connection=[value]
55. @systemcraftsman Processing Messages
56. @systemcraftsman Processing Messages import org.eclipse.microprofile.reactive.messaging.Incoming; import org.eclipse.microprofile.reactive.messaging.Outgoing; import javax.enterprise.context.ApplicationScoped; @ApplicationScoped public class PriceProcessor { private static final double CONVERSION_RATE = 0.88; @Incoming("price-in") @Outgoing("price-out") public double process(double price) { return price * CONVERSION_RATE; } } Java file
57. @systemcraftsman Processing Messages import javax.enterprise.context.ApplicationScoped; import org.eclipse.microprofile.reactive.messaging.Incoming; import org.eclipse.microprofile.reactive.messaging.Outgoing; import io.smallrye.mutiny.Multi; @ApplicationScoped public class PriceProcessor { private static final double CONVERSION_RATE = 0.88; @Incoming("price-in") @Outgoing("price-out") public Multi<Double> process(Multi<Integer> prices) { return prices.filter(p -> p > 100).map(p -> p * CONVERSION_RATE); } } Java file
58. @systemcraftsman Serialization/Deserialization (SerDe)
59. @systemcraftsman JSON public class Fruit { public String name; public int price; public Fruit() { } public Fruit(String name, int price) { this.name = name; this.price = price; } } @ApplicationScoped public class FruitProcessor { private static final double CONVERSION_RATE = 0.88; @Incoming("fruit-in") @Outgoing("fruit-out") @Broadcast public Fruit process(Fruit fruit) { fruit.price = fruit.price * CONVERSION_RATE; return fruit; } } Java file Java file
60. @systemcraftsman SerDe via Jackson <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-jackson</artifactId> </dependency> # Configure the Kafka source (we read from it) mp.messaging.incoming.fruit-in.connector=smallrye-kafka mp.messaging.incoming.fruit-in.topic=fruit-in mp.messaging.incoming.fruit-in.value.deserializer=com.acme. fruit.jackson.FruitDeserializer # Configure the Kafka sink (we write to it) mp.messaging.outgoing.fruit-out.connector=smallrye-kafka mp.messaging.outgoing.fruit-out.topic=fruit-out mp.messaging.outgoing.fruit-out.value.serializer=io.quarkus .kafka.client.serialization.ObjectMapperSerializer package com.acme.fruit.jackson; import io.quarkus.kafka.client.serialization.ObjectMapperDeseri alizer; public class FruitDeserializer extends ObjectMapperDeserializer<Fruit> { public FruitDeserializer() { super(Fruit.class); } } Java file pom.xml application.properties
61. @systemcraftsman SerDe via JSON-B <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-jsonb</artifactId> </dependency> # Configure the Kafka source (we read from it) mp.messaging.incoming.fruit-in.connector=smallrye-kafka mp.messaging.incoming.fruit-in.topic=fruit-in mp.messaging.incoming.fruit-in.value.deserializer=com.acme. fruit.jackson.FruitDeserializer # Configure the Kafka sink (we write to it) mp.messaging.outgoing.fruit-out.connector=smallrye-kafka mp.messaging.outgoing.fruit-out.topic=fruit-out mp.messaging.outgoing.fruit-out.value.serializer=io.quarkus .kafka.client.serialization.JsonbSerializer package com.acme.fruit.jsonb; import io.quarkus.kafka.client.serialization.JsonbDeserializer; public class FruitDeserializer extends JsonbDeserializer<Fruit> { public FruitDeserializer() { super(Fruit.class); } } application.properties pom.xml Java file
62. @systemcraftsman Avro io.apicurio.registry.utils.serde.AvroKafkaSerializer io.apicurio.registry.utils.serde.AvroKafkaDeserializer <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-avro</artifactId> </dependency> <dependency> <groupId>io.apicurio</groupId> <artifactId>apicurio-registry-utils-serde</artifactId> </dependency> # Configure the Kafka source (we read from it) mp.messaging.incoming.fruit-in.connector=smallrye-kafka mp.messaging.incoming.fruit-in.topic=fruit-in mp.messaging.incoming.fruit-in.value.deserializer=io.apicur io.registry.utils.serde.AvroKafkaDeserializer # Configure the Kafka sink (we write to it) mp.messaging.outgoing.fruit-out.connector=smallrye-kafka mp.messaging.outgoing.fruit-out.topic=fruit-out mp.messaging.outgoing.fruit-out.value.serializer=io.apicuri o.registry.utils.serde.AvroKafkaSerializer application.properties pom.xml
63. @systemcraftsman Talk is Cheap, Show me the Demo!
64. @systemcraftsman Resources • MicroProfile Reactive Messaging Spec 2.0 • Quarkus Apache Kafka Reference Guide • SmallRye Reactive Messaging • Twitter: @systemcraftsman • Email: aykut@systemcraftsman.com • Demo: https://github.com/RedHatTraining/AD482-ToT-CoffeeShop
65. @systemcraftsman Thank You! www.SystemCraftsman.com