Upgrade to Pro — share decks privately, control downloads, hide ads and more …

MicroProfile Reactive Messaging with Quarkus and Apache Kafka

MicroProfile Reactive Messaging with Quarkus and Apache Kafka

JakartaOne Turkish, Aug 21st 2021

Follow me on twitter (@systemcraftsman) or subscribe to https://www.systemcraftsman.com/join/ to get any updates from me.

E3afc47cd48ab5a032c7e2078c0d3ce7?s=128

Aykut Bulgu

August 21, 2021
Tweet

Transcript

  1. @systemcraftsman MicroProfile Reactive Messaging with Quarkus and Apache Kafka Aykut

    M. Bulgu Trainer & Instructor | Technology Consultant www.SystemCraftsman.com
  2. @systemcraftsman Me as Code #oc apply -f aykutbulgu.yaml apiVersion: redhat/v3.2

    kind: Services Content Architect metadata: name: Aykut Bulgu namespace: Red Hat Global Learning Services annotations: twitter: @systemcraftsman email: - aykut@systemcraftsman.com - abulgu@redhat.com organizer: Software Craftsmanship Turkey founder: System Craftsman labels: married: yes children: 1 interests: - tech (mostly kafka) - aikido - gamification - stoicism spec: replicas: 1 containers: - image: aykut:latest
  3. @systemcraftsman The Specification

  4. @systemcraftsman MicroProfile Open-source community specification for Cloud-Native Java applications. A

    community of individuals, organizations, and vendors collaborating.
  5. @systemcraftsman MicroProfile 4.0

  6. @systemcraftsman Why this specification?

  7. @systemcraftsman Monoliths https://speakerdeck.com/mbogoevici/data-strategies-for-microservice-architectures?slide=4

  8. @systemcraftsman (Micro)Services & Data Problem https://speakerdeck.com/mbogoevici/data-strategies-for-microservice-architectures?slide=5

  9. @systemcraftsman Request-Driven Architecture Strong Coupling

  10. @systemcraftsman Request-Driven Architecture Latency Availability Performance No fault tolerance Tightly

    coupled
  11. @systemcraftsman Event-Driven Architecture and Messaging

  12. @systemcraftsman Reactive Systems

  13. @systemcraftsman What is wrong with JMS and Message Driven Beans?

    Heavyweight (mostly because of XA) Incompatible with popular brokers like Kafka No support for asynchronous IO
  14. @systemcraftsman Concepts

  15. @systemcraftsman Concepts

  16. @systemcraftsman Overall Architecture

  17. @systemcraftsman Concepts Internal Channels: In-memory messaging that creates a reactive

    chain of processing of beans External Channels: Channels are to be connected to external systems like Kafka or AMQP broker
  18. @systemcraftsman Concepts Internal Channels: In-memory messaging that creates a reactive

    chain of processing of beans External Channels: Channels are to be connected to external systems like Kafka or AMQP broker External channels Internal channels
  19. @systemcraftsman Concepts org.eclipse.microprofile.reactive.messaging.Message

  20. @systemcraftsman Concepts org.eclipse.microprofile.reactive.messaging.Incoming

  21. @systemcraftsman Concepts org.eclipse.microprofile.reactive.messaging.Acknowledgment

  22. @systemcraftsman Concepts org.eclipse.microprofile.reactive.messaging.Outgoing

  23. @systemcraftsman Concepts org.eclipse.microprofile.reactive.messaging.Emitter

  24. @systemcraftsman Concepts org.eclipse.microprofile.reactive.messaging.connector.IncomingConnectorFactory org.eclipse.microprofile.reactive.messaging.connector.OutgoingConnectorFactory mp.messaging.[incoming | outgoing].[channel-name].connector=[connector-name] mp.messaging.[incoming | outgoing].[channel-name].[attribute]=[value]

    application.properties Java
  25. @systemcraftsman The Implementation

  26. @systemcraftsman Implementations Lightbend Alpakka SmallRye Reactive Messaging Open Liberty 19.0.0.9+

    usage (via SmallRye Reactive Messaging)
  27. @systemcraftsman SmallRye Reactive Messaging

  28. @systemcraftsman Configuring for Kafka kafka.bootstrap.servers=localhost:9092 #mp.messaging.incoming.prices.bootstrap.servers=localhost:9092 mp.messaging.incoming.prices.connector=smallrye-kafka mp.messaging.outgoing.prices.connector=smallrye-kafka application.properties

  29. @systemcraftsman Receiving Messages from Kafka

  30. @systemcraftsman Configuration kafka.bootstrap.servers=localhost:9092 mp.messaging.incoming.prices.connector=smallrye-kafka mp.messaging.incoming.prices.topic=prices application.properties

  31. @systemcraftsman Using @Incoming import org.eclipse.microprofile.reactive.messaging.Incoming; import javax.enterprise.context.ApplicationScoped; @ApplicationScoped public class

    PriceConsumer { @Incoming("prices") public void consume(double price) { // process your price. } } Java file
  32. @systemcraftsman Using @Incoming @Incoming("prices") public CompletionStage<Void> consume(Message<Double> msg) { //

    access record metadata var metadata = msg.getMetadata(IncomingKafkaRecordMetadata.class).orElseThrow(); // process the message payload. double price = msg.getPayload(); // Acknowledge the incoming message (commit the offset) return msg.ack(); } Java file
  33. @systemcraftsman Using @Channel import io.smallrye.mutiny.Multi; import io.smallrye.reactive.messaging.annotations.Channel; ...some imports omitted...

    import org.jboss.resteasy.annotations.SseElementType; @Path("/prices") public class PriceResource { @Inject @Channel("prices") Multi<Double> prices; @GET @Path("/prices") @Produces(MediaType.SERVER_SENT_EVENTS) @SseElementType("text/plain") public Multi<Double> stream() { return prices; } } Java file
  34. @systemcraftsman Using @Channel @Inject @Channel("prices") Multi<Double> streamOfPayloads; @Inject @Channel("prices") Multi<Message<Double>>

    streamOfMessages; @Inject @Channel("prices") Publisher<Double> publisherOfPayloads; @Inject @Channel("prices") Publisher<Message<Double>> publisherOfMessages; Java file
  35. @systemcraftsman Acknowledgment Strategies Acknowledgment.Strategy.PRE_PROCESSING Acknowledgment.Strategy.POST_PROCESSING Acknowledgment.Strategy.NONE Acknowledgment.Strategy.MANUAL

  36. @systemcraftsman Acknowledgment Strategies @Incoming("prices") @Acknowledgment(Acknowledgment.Strategy.PRE_PROCESSING) public void process(double price) {

    // process price } @Incoming("prices") public CompletionStage<Void> process(Message<Double> msg) { // process price return msg.ack(); } Java file Java file
  37. @systemcraftsman Commit Strategies throttled keeps track of received messages and

    commits an offset of the latest acked message in sequence (meaning, all previous messages were also acked). This is the default if enable.auto.commit is not explicitly set to true. latest commits the record offset received by the Kafka consumer as soon as the associated message is acknowledged (if the offset is higher than the previously committed offset). ignore performs no commit. This strategy is the default strategy when the consumer is explicitly configured with enable.auto.commit to true.
  38. @systemcraftsman Error Handling fail fail the application, no more records

    will be processed (default strategy). The offset of the record that has not been processed correctly is not committed. ignore the failure is logged, but the processing continue. The offset of the record that has not been processed correctly is committed. dead-letter-queue the offset of the record that has not been processed correctly is committed, but the record is written to a Kafka dead letter topic.
  39. @systemcraftsman For High Throughput Use the throttled policy Or set

    enable.auto.commit to true and use @Acknowledgment(Acknowledgment.Strategy.NONE).
  40. @systemcraftsman Consumer Groups Single consumer thread inside a consumer group

    Multiple consumer threads inside a consumer group Multiple consumer applications inside a consumer group Pub/Sub: Multiple consumer groups subscribed to a topic 4 Patterns for consuming:
  41. @systemcraftsman Consumer Groups Single consumer thread inside a consumer group

    quarkus.application.name or kafka.group.id
  42. @systemcraftsman Consumer Groups Multiple consumer threads inside a consumer group

    mp.messaging.incoming.[channel-name].partitions
  43. @systemcraftsman Consumer Groups Multiple consumer applications inside a consumer group

    mp.messaging.incoming.[channel-name].group.id
  44. @systemcraftsman Consumer Groups Pub/Sub: Multiple consumer groups subscribed to a

    topic mp.messaging.incoming.somechannel.group.id=someid mp.messaging.incoming.somechannel.group.id=someotherid
  45. @systemcraftsman Sending Messages to Kafka

  46. @systemcraftsman Configuration kafka.bootstrap.servers=localhost:9092 mp.messaging.outgoing.prices.connector=smallrye-kafka mp.messaging.outgoing.prices.topic=prices application.properties

  47. @systemcraftsman Configuration kafka.bootstrap.servers=localhost:9092 mp.messaging.incoming.prices.connector=smallrye-kafka mp.messaging.incoming.prices.topic=prices mp.messaging.outgoing.prices-out.connector=smallrye-kafka mp.messaging.outgoing.prices-out.topic=prices application.properties

  48. @systemcraftsman Using @Outgoing import io.smallrye.mutiny.Multi; import org.eclipse.microprofile.reactive.messaging.Outgoing; import javax.enterprise.context.ApplicationScoped; import

    java.time.Duration; import java.util.Random; @ApplicationScoped public class KafkaPriceProducer { private final Random random = new Random(); @Outgoing("prices") public Multi<Double> generate() { // Build an infinite stream of random prices // It emits a price every second return Multi.createFrom().ticks().every(Duration.ofSeconds(1)) .map(x -> random.nextDouble()); } } Java file
  49. @systemcraftsman Using @Outgoing @Outgoing("prices") public Multi<Message<Double>> generate() { return Multi.createFrom().ticks().every(Duration.ofSeconds(1))

    .map(x -> Message.of(random.nextDouble()) .addMetadata(OutgoingKafkaRecordMetadata.<String>builder() .withKey("my-key") .withTopic("my-key-prices") .withHeaders(new RecordHeaders().add("my-header", "value".getBytes())) .build())); } Java file
  50. @systemcraftsman Using @Outgoing @Outgoing("prices") T generate(); // T excluding void

    @Outgoing("prices") Message<T> generate(); @Outgoing("prices") Uni<T> generate(); @Outgoing("prices") Uni<Message<T>> generate(); @Outgoing("prices") CompletionStage<T> generate(); @Outgoing("prices") CompletionStage<Message<T>> generate(); Java file
  51. @systemcraftsman Using @Emitter import org.eclipse.microprofile.reactive.messaging.Channel; import org.eclipse.microprofile.reactive.messaging.Emitter; ...some imports omitted...

    @Path("/prices") public class PriceResource { @Inject @Channel("prices") Emitter<Double> priceEmitter; @POST @Consumes(MediaType.TEXT_PLAIN) public void addPrice(Double price) { CompletionStage<Void> ack = priceEmitter.send(price); } } Java file
  52. @systemcraftsman Using @Emitter import java.util.concurrent.CompletableFuture; import org.eclipse.microprofile.reactive.messaging.Channel; import org.eclipse.microprofile.reactive.messaging.Emitter; ...some

    imports omitted... @Path("/prices") public class PriceResource { @Inject @Channel("prices") Emitter<Double> priceEmitter; @POST @Consumes(MediaType.TEXT_PLAIN) public void addPrice(Double price) { priceEmitter.send(Message.of(price) .withAck(() -> { // Called when the message is acked return CompletableFuture.completedFuture(null); }) .withNack(throwable -> { // Called when the message is nacked return CompletableFuture.completedFuture(null); })); } } Java file
  53. @systemcraftsman Write Acknowledgement mp.messaging.outgoing.[channel-name].acks=[0 | 1 | all] mp.messaging.outgoing.[channel-name].waitForWriteCompletion=[true* |

    false]
  54. @systemcraftsman Retries mp.messaging.outgoing.[channel-name].retries=[value] mp.messaging.outgoing.[channel-name].retry.backoff.ms=[value] mp.messaging.outgoing.[channel-name].delivery.timeout.ms=[value] mp.messaging.outgoing.[channel-name].max.in.flight.requests.per.connection=[value]

  55. @systemcraftsman Processing Messages

  56. @systemcraftsman Processing Messages import org.eclipse.microprofile.reactive.messaging.Incoming; import org.eclipse.microprofile.reactive.messaging.Outgoing; import javax.enterprise.context.ApplicationScoped; @ApplicationScoped

    public class PriceProcessor { private static final double CONVERSION_RATE = 0.88; @Incoming("price-in") @Outgoing("price-out") public double process(double price) { return price * CONVERSION_RATE; } } Java file
  57. @systemcraftsman Processing Messages import javax.enterprise.context.ApplicationScoped; import org.eclipse.microprofile.reactive.messaging.Incoming; import org.eclipse.microprofile.reactive.messaging.Outgoing; import

    io.smallrye.mutiny.Multi; @ApplicationScoped public class PriceProcessor { private static final double CONVERSION_RATE = 0.88; @Incoming("price-in") @Outgoing("price-out") public Multi<Double> process(Multi<Integer> prices) { return prices.filter(p -> p > 100).map(p -> p * CONVERSION_RATE); } } Java file
  58. @systemcraftsman Serialization/Deserialization (SerDe)

  59. @systemcraftsman JSON public class Fruit { public String name; public

    int price; public Fruit() { } public Fruit(String name, int price) { this.name = name; this.price = price; } } @ApplicationScoped public class FruitProcessor { private static final double CONVERSION_RATE = 0.88; @Incoming("fruit-in") @Outgoing("fruit-out") @Broadcast public Fruit process(Fruit fruit) { fruit.price = fruit.price * CONVERSION_RATE; return fruit; } } Java file Java file
  60. @systemcraftsman SerDe via Jackson <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-jackson</artifactId> </dependency> # Configure

    the Kafka source (we read from it) mp.messaging.incoming.fruit-in.connector=smallrye-kafka mp.messaging.incoming.fruit-in.topic=fruit-in mp.messaging.incoming.fruit-in.value.deserializer=com.acme. fruit.jackson.FruitDeserializer # Configure the Kafka sink (we write to it) mp.messaging.outgoing.fruit-out.connector=smallrye-kafka mp.messaging.outgoing.fruit-out.topic=fruit-out mp.messaging.outgoing.fruit-out.value.serializer=io.quarkus .kafka.client.serialization.ObjectMapperSerializer package com.acme.fruit.jackson; import io.quarkus.kafka.client.serialization.ObjectMapperDeseri alizer; public class FruitDeserializer extends ObjectMapperDeserializer<Fruit> { public FruitDeserializer() { super(Fruit.class); } } Java file pom.xml application.properties
  61. @systemcraftsman SerDe via JSON-B <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-jsonb</artifactId> </dependency> # Configure

    the Kafka source (we read from it) mp.messaging.incoming.fruit-in.connector=smallrye-kafka mp.messaging.incoming.fruit-in.topic=fruit-in mp.messaging.incoming.fruit-in.value.deserializer=com.acme. fruit.jackson.FruitDeserializer # Configure the Kafka sink (we write to it) mp.messaging.outgoing.fruit-out.connector=smallrye-kafka mp.messaging.outgoing.fruit-out.topic=fruit-out mp.messaging.outgoing.fruit-out.value.serializer=io.quarkus .kafka.client.serialization.JsonbSerializer package com.acme.fruit.jsonb; import io.quarkus.kafka.client.serialization.JsonbDeserializer; public class FruitDeserializer extends JsonbDeserializer<Fruit> { public FruitDeserializer() { super(Fruit.class); } } application.properties pom.xml Java file
  62. @systemcraftsman Avro io.apicurio.registry.utils.serde.AvroKafkaSerializer io.apicurio.registry.utils.serde.AvroKafkaDeserializer <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-avro</artifactId> </dependency> <dependency> <groupId>io.apicurio</groupId>

    <artifactId>apicurio-registry-utils-serde</artifactId> </dependency> # Configure the Kafka source (we read from it) mp.messaging.incoming.fruit-in.connector=smallrye-kafka mp.messaging.incoming.fruit-in.topic=fruit-in mp.messaging.incoming.fruit-in.value.deserializer=io.apicur io.registry.utils.serde.AvroKafkaDeserializer # Configure the Kafka sink (we write to it) mp.messaging.outgoing.fruit-out.connector=smallrye-kafka mp.messaging.outgoing.fruit-out.topic=fruit-out mp.messaging.outgoing.fruit-out.value.serializer=io.apicuri o.registry.utils.serde.AvroKafkaSerializer application.properties pom.xml
  63. @systemcraftsman Talk is Cheap, Show me the Demo!

  64. @systemcraftsman Resources • MicroProfile Reactive Messaging Spec 2.0 • Quarkus

    Apache Kafka Reference Guide • SmallRye Reactive Messaging • Twitter: @systemcraftsman • Email: aykut@systemcraftsman.com • Demo: https://github.com/RedHatTraining/AD482-ToT-CoffeeShop
  65. @systemcraftsman Thank You! www.SystemCraftsman.com