Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

MicroProfile Reactive Messaging with Quarkus an...

MicroProfile Reactive Messaging with Quarkus and Apache Kafka

JakartaOne Turkish, Aug 21st 2021

Follow me on twitter (@systemcraftsman) or subscribe to https://www.systemcraftsman.com/join/ to get any updates from me.

Aykut Bulgu

August 21, 2021
Tweet

More Decks by Aykut Bulgu

Other Decks in Programming

Transcript

  1. @systemcraftsman MicroProfile Reactive Messaging with Quarkus and Apache Kafka Aykut

    M. Bulgu Trainer & Instructor | Technology Consultant www.SystemCraftsman.com
  2. @systemcraftsman Me as Code #oc apply -f aykutbulgu.yaml apiVersion: redhat/v3.2

    kind: Services Content Architect metadata: name: Aykut Bulgu namespace: Red Hat Global Learning Services annotations: twitter: @systemcraftsman email: - [email protected] - [email protected] organizer: Software Craftsmanship Turkey founder: System Craftsman labels: married: yes children: 1 interests: - tech (mostly kafka) - aikido - gamification - stoicism spec: replicas: 1 containers: - image: aykut:latest
  3. @systemcraftsman MicroProfile Open-source community specification for Cloud-Native Java applications. A

    community of individuals, organizations, and vendors collaborating.
  4. @systemcraftsman What is wrong with JMS and Message Driven Beans?

    Heavyweight (mostly because of XA) Incompatible with popular brokers like Kafka No support for asynchronous IO
  5. @systemcraftsman Concepts Internal Channels: In-memory messaging that creates a reactive

    chain of processing of beans External Channels: Channels are to be connected to external systems like Kafka or AMQP broker
  6. @systemcraftsman Concepts Internal Channels: In-memory messaging that creates a reactive

    chain of processing of beans External Channels: Channels are to be connected to external systems like Kafka or AMQP broker External channels Internal channels
  7. @systemcraftsman Using @Incoming @Incoming("prices") public CompletionStage<Void> consume(Message<Double> msg) { //

    access record metadata var metadata = msg.getMetadata(IncomingKafkaRecordMetadata.class).orElseThrow(); // process the message payload. double price = msg.getPayload(); // Acknowledge the incoming message (commit the offset) return msg.ack(); } Java file
  8. @systemcraftsman Using @Channel import io.smallrye.mutiny.Multi; import io.smallrye.reactive.messaging.annotations.Channel; ...some imports omitted...

    import org.jboss.resteasy.annotations.SseElementType; @Path("/prices") public class PriceResource { @Inject @Channel("prices") Multi<Double> prices; @GET @Path("/prices") @Produces(MediaType.SERVER_SENT_EVENTS) @SseElementType("text/plain") public Multi<Double> stream() { return prices; } } Java file
  9. @systemcraftsman Using @Channel @Inject @Channel("prices") Multi<Double> streamOfPayloads; @Inject @Channel("prices") Multi<Message<Double>>

    streamOfMessages; @Inject @Channel("prices") Publisher<Double> publisherOfPayloads; @Inject @Channel("prices") Publisher<Message<Double>> publisherOfMessages; Java file
  10. @systemcraftsman Acknowledgment Strategies @Incoming("prices") @Acknowledgment(Acknowledgment.Strategy.PRE_PROCESSING) public void process(double price) {

    // process price } @Incoming("prices") public CompletionStage<Void> process(Message<Double> msg) { // process price return msg.ack(); } Java file Java file
  11. @systemcraftsman Commit Strategies throttled keeps track of received messages and

    commits an offset of the latest acked message in sequence (meaning, all previous messages were also acked). This is the default if enable.auto.commit is not explicitly set to true. latest commits the record offset received by the Kafka consumer as soon as the associated message is acknowledged (if the offset is higher than the previously committed offset). ignore performs no commit. This strategy is the default strategy when the consumer is explicitly configured with enable.auto.commit to true.
  12. @systemcraftsman Error Handling fail fail the application, no more records

    will be processed (default strategy). The offset of the record that has not been processed correctly is not committed. ignore the failure is logged, but the processing continue. The offset of the record that has not been processed correctly is committed. dead-letter-queue the offset of the record that has not been processed correctly is committed, but the record is written to a Kafka dead letter topic.
  13. @systemcraftsman For High Throughput Use the throttled policy Or set

    enable.auto.commit to true and use @Acknowledgment(Acknowledgment.Strategy.NONE).
  14. @systemcraftsman Consumer Groups Single consumer thread inside a consumer group

    Multiple consumer threads inside a consumer group Multiple consumer applications inside a consumer group Pub/Sub: Multiple consumer groups subscribed to a topic 4 Patterns for consuming:
  15. @systemcraftsman Consumer Groups Pub/Sub: Multiple consumer groups subscribed to a

    topic mp.messaging.incoming.somechannel.group.id=someid mp.messaging.incoming.somechannel.group.id=someotherid
  16. @systemcraftsman Using @Outgoing import io.smallrye.mutiny.Multi; import org.eclipse.microprofile.reactive.messaging.Outgoing; import javax.enterprise.context.ApplicationScoped; import

    java.time.Duration; import java.util.Random; @ApplicationScoped public class KafkaPriceProducer { private final Random random = new Random(); @Outgoing("prices") public Multi<Double> generate() { // Build an infinite stream of random prices // It emits a price every second return Multi.createFrom().ticks().every(Duration.ofSeconds(1)) .map(x -> random.nextDouble()); } } Java file
  17. @systemcraftsman Using @Outgoing @Outgoing("prices") public Multi<Message<Double>> generate() { return Multi.createFrom().ticks().every(Duration.ofSeconds(1))

    .map(x -> Message.of(random.nextDouble()) .addMetadata(OutgoingKafkaRecordMetadata.<String>builder() .withKey("my-key") .withTopic("my-key-prices") .withHeaders(new RecordHeaders().add("my-header", "value".getBytes())) .build())); } Java file
  18. @systemcraftsman Using @Outgoing @Outgoing("prices") T generate(); // T excluding void

    @Outgoing("prices") Message<T> generate(); @Outgoing("prices") Uni<T> generate(); @Outgoing("prices") Uni<Message<T>> generate(); @Outgoing("prices") CompletionStage<T> generate(); @Outgoing("prices") CompletionStage<Message<T>> generate(); Java file
  19. @systemcraftsman Using @Emitter import org.eclipse.microprofile.reactive.messaging.Channel; import org.eclipse.microprofile.reactive.messaging.Emitter; ...some imports omitted...

    @Path("/prices") public class PriceResource { @Inject @Channel("prices") Emitter<Double> priceEmitter; @POST @Consumes(MediaType.TEXT_PLAIN) public void addPrice(Double price) { CompletionStage<Void> ack = priceEmitter.send(price); } } Java file
  20. @systemcraftsman Using @Emitter import java.util.concurrent.CompletableFuture; import org.eclipse.microprofile.reactive.messaging.Channel; import org.eclipse.microprofile.reactive.messaging.Emitter; ...some

    imports omitted... @Path("/prices") public class PriceResource { @Inject @Channel("prices") Emitter<Double> priceEmitter; @POST @Consumes(MediaType.TEXT_PLAIN) public void addPrice(Double price) { priceEmitter.send(Message.of(price) .withAck(() -> { // Called when the message is acked return CompletableFuture.completedFuture(null); }) .withNack(throwable -> { // Called when the message is nacked return CompletableFuture.completedFuture(null); })); } } Java file
  21. @systemcraftsman Processing Messages import org.eclipse.microprofile.reactive.messaging.Incoming; import org.eclipse.microprofile.reactive.messaging.Outgoing; import javax.enterprise.context.ApplicationScoped; @ApplicationScoped

    public class PriceProcessor { private static final double CONVERSION_RATE = 0.88; @Incoming("price-in") @Outgoing("price-out") public double process(double price) { return price * CONVERSION_RATE; } } Java file
  22. @systemcraftsman Processing Messages import javax.enterprise.context.ApplicationScoped; import org.eclipse.microprofile.reactive.messaging.Incoming; import org.eclipse.microprofile.reactive.messaging.Outgoing; import

    io.smallrye.mutiny.Multi; @ApplicationScoped public class PriceProcessor { private static final double CONVERSION_RATE = 0.88; @Incoming("price-in") @Outgoing("price-out") public Multi<Double> process(Multi<Integer> prices) { return prices.filter(p -> p > 100).map(p -> p * CONVERSION_RATE); } } Java file
  23. @systemcraftsman JSON public class Fruit { public String name; public

    int price; public Fruit() { } public Fruit(String name, int price) { this.name = name; this.price = price; } } @ApplicationScoped public class FruitProcessor { private static final double CONVERSION_RATE = 0.88; @Incoming("fruit-in") @Outgoing("fruit-out") @Broadcast public Fruit process(Fruit fruit) { fruit.price = fruit.price * CONVERSION_RATE; return fruit; } } Java file Java file
  24. @systemcraftsman SerDe via Jackson <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-jackson</artifactId> </dependency> # Configure

    the Kafka source (we read from it) mp.messaging.incoming.fruit-in.connector=smallrye-kafka mp.messaging.incoming.fruit-in.topic=fruit-in mp.messaging.incoming.fruit-in.value.deserializer=com.acme. fruit.jackson.FruitDeserializer # Configure the Kafka sink (we write to it) mp.messaging.outgoing.fruit-out.connector=smallrye-kafka mp.messaging.outgoing.fruit-out.topic=fruit-out mp.messaging.outgoing.fruit-out.value.serializer=io.quarkus .kafka.client.serialization.ObjectMapperSerializer package com.acme.fruit.jackson; import io.quarkus.kafka.client.serialization.ObjectMapperDeseri alizer; public class FruitDeserializer extends ObjectMapperDeserializer<Fruit> { public FruitDeserializer() { super(Fruit.class); } } Java file pom.xml application.properties
  25. @systemcraftsman SerDe via JSON-B <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-jsonb</artifactId> </dependency> # Configure

    the Kafka source (we read from it) mp.messaging.incoming.fruit-in.connector=smallrye-kafka mp.messaging.incoming.fruit-in.topic=fruit-in mp.messaging.incoming.fruit-in.value.deserializer=com.acme. fruit.jackson.FruitDeserializer # Configure the Kafka sink (we write to it) mp.messaging.outgoing.fruit-out.connector=smallrye-kafka mp.messaging.outgoing.fruit-out.topic=fruit-out mp.messaging.outgoing.fruit-out.value.serializer=io.quarkus .kafka.client.serialization.JsonbSerializer package com.acme.fruit.jsonb; import io.quarkus.kafka.client.serialization.JsonbDeserializer; public class FruitDeserializer extends JsonbDeserializer<Fruit> { public FruitDeserializer() { super(Fruit.class); } } application.properties pom.xml Java file
  26. @systemcraftsman Avro io.apicurio.registry.utils.serde.AvroKafkaSerializer io.apicurio.registry.utils.serde.AvroKafkaDeserializer <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-avro</artifactId> </dependency> <dependency> <groupId>io.apicurio</groupId>

    <artifactId>apicurio-registry-utils-serde</artifactId> </dependency> # Configure the Kafka source (we read from it) mp.messaging.incoming.fruit-in.connector=smallrye-kafka mp.messaging.incoming.fruit-in.topic=fruit-in mp.messaging.incoming.fruit-in.value.deserializer=io.apicur io.registry.utils.serde.AvroKafkaDeserializer # Configure the Kafka sink (we write to it) mp.messaging.outgoing.fruit-out.connector=smallrye-kafka mp.messaging.outgoing.fruit-out.topic=fruit-out mp.messaging.outgoing.fruit-out.value.serializer=io.apicuri o.registry.utils.serde.AvroKafkaSerializer application.properties pom.xml
  27. @systemcraftsman Resources • MicroProfile Reactive Messaging Spec 2.0 • Quarkus

    Apache Kafka Reference Guide • SmallRye Reactive Messaging • Twitter: @systemcraftsman • Email: [email protected] • Demo: https://github.com/RedHatTraining/AD482-ToT-CoffeeShop