Upgrade to Pro — share decks privately, control downloads, hide ads and more …

MicroProfile Reactive Messaging with Quarkus and Apache Kafka

MicroProfile Reactive Messaging with Quarkus and Apache Kafka

JakartaOne Turkish, Aug 21st 2021

Follow me on twitter (@systemcraftsman) or subscribe to https://www.systemcraftsman.com/join/ to get any updates from me.

Aykut Bulgu

August 21, 2021
Tweet

More Decks by Aykut Bulgu

Other Decks in Programming

Transcript

  1. @systemcraftsman
    MicroProfile Reactive Messaging
    with
    Quarkus and Apache Kafka
    Aykut M. Bulgu
    Trainer & Instructor | Technology Consultant
    www.SystemCraftsman.com

    View full-size slide

  2. @systemcraftsman
    Me as Code #oc apply -f aykutbulgu.yaml
    apiVersion: redhat/v3.2
    kind: Services Content Architect
    metadata:
    name: Aykut Bulgu
    namespace: Red Hat Global Learning Services
    annotations:
    twitter: @systemcraftsman
    email:
    - [email protected]
    - [email protected]
    organizer: Software Craftsmanship Turkey
    founder: System Craftsman
    labels:
    married: yes
    children: 1
    interests:
    - tech (mostly kafka)
    - aikido
    - gamification
    - stoicism
    spec:
    replicas: 1
    containers:
    - image: aykut:latest

    View full-size slide

  3. @systemcraftsman
    The Specification

    View full-size slide

  4. @systemcraftsman
    MicroProfile
    Open-source community specification for
    Cloud-Native Java applications.
    A community of individuals, organizations, and
    vendors collaborating.

    View full-size slide

  5. @systemcraftsman
    MicroProfile 4.0

    View full-size slide

  6. @systemcraftsman
    Why this specification?

    View full-size slide

  7. @systemcraftsman
    Monoliths
    https://speakerdeck.com/mbogoevici/data-strategies-for-microservice-architectures?slide=4

    View full-size slide

  8. @systemcraftsman
    (Micro)Services & Data Problem
    https://speakerdeck.com/mbogoevici/data-strategies-for-microservice-architectures?slide=5

    View full-size slide

  9. @systemcraftsman
    Request-Driven Architecture
    Strong Coupling

    View full-size slide

  10. @systemcraftsman
    Request-Driven Architecture
    Latency
    Availability
    Performance
    No fault tolerance
    Tightly coupled

    View full-size slide

  11. @systemcraftsman
    Event-Driven Architecture and Messaging

    View full-size slide

  12. @systemcraftsman
    Reactive Systems

    View full-size slide

  13. @systemcraftsman
    What is wrong with JMS and Message Driven Beans?
    Heavyweight (mostly
    because of XA)
    Incompatible with popular
    brokers like Kafka
    No support for
    asynchronous IO

    View full-size slide

  14. @systemcraftsman
    Concepts

    View full-size slide

  15. @systemcraftsman
    Concepts

    View full-size slide

  16. @systemcraftsman
    Overall Architecture

    View full-size slide

  17. @systemcraftsman
    Concepts
    Internal Channels: In-memory messaging that creates a reactive chain of processing of beans
    External Channels: Channels are to be connected to external systems like Kafka or AMQP broker

    View full-size slide

  18. @systemcraftsman
    Concepts
    Internal Channels: In-memory messaging that creates a reactive chain of processing of beans
    External Channels: Channels are to be connected to external systems like Kafka or AMQP broker
    External
    channels
    Internal
    channels

    View full-size slide

  19. @systemcraftsman
    Concepts
    org.eclipse.microprofile.reactive.messaging.Message

    View full-size slide

  20. @systemcraftsman
    Concepts
    org.eclipse.microprofile.reactive.messaging.Incoming

    View full-size slide

  21. @systemcraftsman
    Concepts
    org.eclipse.microprofile.reactive.messaging.Acknowledgment

    View full-size slide

  22. @systemcraftsman
    Concepts
    org.eclipse.microprofile.reactive.messaging.Outgoing

    View full-size slide

  23. @systemcraftsman
    Concepts
    org.eclipse.microprofile.reactive.messaging.Emitter

    View full-size slide

  24. @systemcraftsman
    Concepts
    org.eclipse.microprofile.reactive.messaging.connector.IncomingConnectorFactory
    org.eclipse.microprofile.reactive.messaging.connector.OutgoingConnectorFactory
    mp.messaging.[incoming | outgoing].[channel-name].connector=[connector-name]
    mp.messaging.[incoming | outgoing].[channel-name].[attribute]=[value] application.properties
    Java

    View full-size slide

  25. @systemcraftsman
    The Implementation

    View full-size slide

  26. @systemcraftsman
    Implementations
    Lightbend Alpakka
    SmallRye Reactive Messaging
    Open Liberty 19.0.0.9+ usage (via SmallRye Reactive Messaging)

    View full-size slide

  27. @systemcraftsman
    SmallRye Reactive Messaging

    View full-size slide

  28. @systemcraftsman
    Configuring for Kafka
    kafka.bootstrap.servers=localhost:9092
    #mp.messaging.incoming.prices.bootstrap.servers=localhost:9092
    mp.messaging.incoming.prices.connector=smallrye-kafka
    mp.messaging.outgoing.prices.connector=smallrye-kafka application.properties

    View full-size slide

  29. @systemcraftsman
    Receiving Messages from Kafka

    View full-size slide

  30. @systemcraftsman
    Configuration
    kafka.bootstrap.servers=localhost:9092
    mp.messaging.incoming.prices.connector=smallrye-kafka
    mp.messaging.incoming.prices.topic=prices application.properties

    View full-size slide

  31. @systemcraftsman
    Using @Incoming
    import org.eclipse.microprofile.reactive.messaging.Incoming;
    import javax.enterprise.context.ApplicationScoped;
    @ApplicationScoped
    public class PriceConsumer {
    @Incoming("prices")
    public void consume(double price) {
    // process your price.
    }
    }
    Java file

    View full-size slide

  32. @systemcraftsman
    Using @Incoming
    @Incoming("prices")
    public CompletionStage consume(Message msg) {
    // access record metadata
    var metadata =
    msg.getMetadata(IncomingKafkaRecordMetadata.class).orElseThrow();
    // process the message payload.
    double price = msg.getPayload();
    // Acknowledge the incoming message (commit the offset)
    return msg.ack();
    }
    Java file

    View full-size slide

  33. @systemcraftsman
    Using @Channel
    import io.smallrye.mutiny.Multi;
    import io.smallrye.reactive.messaging.annotations.Channel;
    ...some imports omitted...
    import org.jboss.resteasy.annotations.SseElementType;
    @Path("/prices")
    public class PriceResource {
    @Inject
    @Channel("prices")
    Multi prices;
    @GET
    @Path("/prices")
    @Produces(MediaType.SERVER_SENT_EVENTS)
    @SseElementType("text/plain")
    public Multi stream() {
    return prices;
    }
    } Java file

    View full-size slide

  34. @systemcraftsman
    Using @Channel
    @Inject @Channel("prices") Multi streamOfPayloads;
    @Inject @Channel("prices") Multi> streamOfMessages;
    @Inject @Channel("prices") Publisher publisherOfPayloads;
    @Inject @Channel("prices") Publisher> publisherOfMessages;
    Java file

    View full-size slide

  35. @systemcraftsman
    Acknowledgment Strategies
    Acknowledgment.Strategy.PRE_PROCESSING
    Acknowledgment.Strategy.POST_PROCESSING
    Acknowledgment.Strategy.NONE
    Acknowledgment.Strategy.MANUAL

    View full-size slide

  36. @systemcraftsman
    Acknowledgment Strategies
    @Incoming("prices")
    @Acknowledgment(Acknowledgment.Strategy.PRE_PROCESSING)
    public void process(double price) {
    // process price
    }
    @Incoming("prices")
    public CompletionStage process(Message msg) {
    // process price
    return msg.ack();
    } Java file
    Java file

    View full-size slide

  37. @systemcraftsman
    Commit Strategies
    throttled
    keeps track of received messages and commits an offset of the latest acked
    message in sequence (meaning, all previous messages were also acked). This is
    the default if enable.auto.commit is not explicitly set to true.
    latest
    commits the record offset received by the Kafka consumer as soon as the
    associated message is acknowledged (if the offset is higher than the previously
    committed offset).
    ignore
    performs no commit. This strategy is the default strategy when the consumer
    is explicitly configured with enable.auto.commit to true.

    View full-size slide

  38. @systemcraftsman
    Error Handling
    fail
    fail the application, no more records will be processed (default
    strategy). The offset of the record that has not been processed
    correctly is not committed.
    ignore
    the failure is logged, but the processing continue. The offset of
    the record that has not been processed correctly is committed.
    dead-letter-queue
    the offset of the record that has not been processed correctly is
    committed, but the record is written to a Kafka dead letter topic.

    View full-size slide

  39. @systemcraftsman
    For High Throughput
    Use the throttled policy
    Or set enable.auto.commit to true and use
    @Acknowledgment(Acknowledgment.Strategy.NONE).

    View full-size slide

  40. @systemcraftsman
    Consumer Groups
    Single consumer thread inside a consumer group
    Multiple consumer threads inside a consumer group
    Multiple consumer applications inside a consumer group
    Pub/Sub: Multiple consumer groups subscribed to a topic
    4 Patterns for consuming:

    View full-size slide

  41. @systemcraftsman
    Consumer Groups
    Single consumer thread inside a consumer group
    quarkus.application.name
    or
    kafka.group.id

    View full-size slide

  42. @systemcraftsman
    Consumer Groups
    Multiple consumer threads inside a consumer group
    mp.messaging.incoming.[channel-name].partitions

    View full-size slide

  43. @systemcraftsman
    Consumer Groups
    Multiple consumer applications inside a consumer group
    mp.messaging.incoming.[channel-name].group.id

    View full-size slide

  44. @systemcraftsman
    Consumer Groups
    Pub/Sub: Multiple consumer groups subscribed to a topic
    mp.messaging.incoming.somechannel.group.id=someid
    mp.messaging.incoming.somechannel.group.id=someotherid

    View full-size slide

  45. @systemcraftsman
    Sending Messages to Kafka

    View full-size slide

  46. @systemcraftsman
    Configuration
    kafka.bootstrap.servers=localhost:9092
    mp.messaging.outgoing.prices.connector=smallrye-kafka
    mp.messaging.outgoing.prices.topic=prices application.properties

    View full-size slide

  47. @systemcraftsman
    Configuration
    kafka.bootstrap.servers=localhost:9092
    mp.messaging.incoming.prices.connector=smallrye-kafka
    mp.messaging.incoming.prices.topic=prices
    mp.messaging.outgoing.prices-out.connector=smallrye-kafka
    mp.messaging.outgoing.prices-out.topic=prices application.properties

    View full-size slide

  48. @systemcraftsman
    Using @Outgoing
    import io.smallrye.mutiny.Multi;
    import org.eclipse.microprofile.reactive.messaging.Outgoing;
    import javax.enterprise.context.ApplicationScoped;
    import java.time.Duration;
    import java.util.Random;
    @ApplicationScoped
    public class KafkaPriceProducer {
    private final Random random = new Random();
    @Outgoing("prices")
    public Multi generate() {
    // Build an infinite stream of random prices
    // It emits a price every second
    return Multi.createFrom().ticks().every(Duration.ofSeconds(1))
    .map(x -> random.nextDouble());
    }
    }
    Java file

    View full-size slide

  49. @systemcraftsman
    Using @Outgoing
    @Outgoing("prices")
    public Multi> generate() {
    return Multi.createFrom().ticks().every(Duration.ofSeconds(1))
    .map(x -> Message.of(random.nextDouble())
    .addMetadata(OutgoingKafkaRecordMetadata.builder()
    .withKey("my-key")
    .withTopic("my-key-prices")
    .withHeaders(new RecordHeaders().add("my-header",
    "value".getBytes()))
    .build()));
    }
    Java file

    View full-size slide

  50. @systemcraftsman
    Using @Outgoing
    @Outgoing("prices") T generate(); // T excluding void
    @Outgoing("prices") Message generate();
    @Outgoing("prices") Uni generate();
    @Outgoing("prices") Uni> generate();
    @Outgoing("prices") CompletionStage generate();
    @Outgoing("prices") CompletionStage> generate();
    Java file

    View full-size slide

  51. @systemcraftsman
    Using @Emitter
    import org.eclipse.microprofile.reactive.messaging.Channel;
    import org.eclipse.microprofile.reactive.messaging.Emitter;
    ...some imports omitted...
    @Path("/prices")
    public class PriceResource {
    @Inject
    @Channel("prices")
    Emitter priceEmitter;
    @POST
    @Consumes(MediaType.TEXT_PLAIN)
    public void addPrice(Double price) {
    CompletionStage ack = priceEmitter.send(price);
    }
    } Java file

    View full-size slide

  52. @systemcraftsman
    Using @Emitter
    import java.util.concurrent.CompletableFuture;
    import org.eclipse.microprofile.reactive.messaging.Channel;
    import org.eclipse.microprofile.reactive.messaging.Emitter;
    ...some imports omitted...
    @Path("/prices")
    public class PriceResource {
    @Inject @Channel("prices") Emitter priceEmitter;
    @POST
    @Consumes(MediaType.TEXT_PLAIN)
    public void addPrice(Double price) {
    priceEmitter.send(Message.of(price)
    .withAck(() -> {
    // Called when the message is acked
    return CompletableFuture.completedFuture(null);
    })
    .withNack(throwable -> {
    // Called when the message is nacked
    return CompletableFuture.completedFuture(null);
    }));
    }
    } Java file

    View full-size slide

  53. @systemcraftsman
    Write Acknowledgement
    mp.messaging.outgoing.[channel-name].acks=[0 | 1 | all]
    mp.messaging.outgoing.[channel-name].waitForWriteCompletion=[true* | false]

    View full-size slide

  54. @systemcraftsman
    Retries
    mp.messaging.outgoing.[channel-name].retries=[value]
    mp.messaging.outgoing.[channel-name].retry.backoff.ms=[value]
    mp.messaging.outgoing.[channel-name].delivery.timeout.ms=[value]
    mp.messaging.outgoing.[channel-name].max.in.flight.requests.per.connection=[value]

    View full-size slide

  55. @systemcraftsman
    Processing Messages

    View full-size slide

  56. @systemcraftsman
    Processing Messages
    import org.eclipse.microprofile.reactive.messaging.Incoming;
    import org.eclipse.microprofile.reactive.messaging.Outgoing;
    import javax.enterprise.context.ApplicationScoped;
    @ApplicationScoped
    public class PriceProcessor {
    private static final double CONVERSION_RATE = 0.88;
    @Incoming("price-in")
    @Outgoing("price-out")
    public double process(double price) {
    return price * CONVERSION_RATE;
    }
    }
    Java file

    View full-size slide

  57. @systemcraftsman
    Processing Messages
    import javax.enterprise.context.ApplicationScoped;
    import org.eclipse.microprofile.reactive.messaging.Incoming;
    import org.eclipse.microprofile.reactive.messaging.Outgoing;
    import io.smallrye.mutiny.Multi;
    @ApplicationScoped
    public class PriceProcessor {
    private static final double CONVERSION_RATE = 0.88;
    @Incoming("price-in")
    @Outgoing("price-out")
    public Multi process(Multi prices) {
    return prices.filter(p -> p > 100).map(p -> p * CONVERSION_RATE);
    }
    }
    Java file

    View full-size slide

  58. @systemcraftsman
    Serialization/Deserialization
    (SerDe)

    View full-size slide

  59. @systemcraftsman
    JSON
    public class Fruit {
    public String name;
    public int price;
    public Fruit() {
    }
    public Fruit(String name, int price) {
    this.name = name;
    this.price = price;
    }
    }
    @ApplicationScoped
    public class FruitProcessor {
    private static final double CONVERSION_RATE = 0.88;
    @Incoming("fruit-in")
    @Outgoing("fruit-out")
    @Broadcast
    public Fruit process(Fruit fruit) {
    fruit.price = fruit.price * CONVERSION_RATE;
    return fruit;
    }
    } Java file
    Java file

    View full-size slide

  60. @systemcraftsman
    SerDe via Jackson

    io.quarkus
    quarkus-jackson
    # Configure the Kafka source (we read from it)
    mp.messaging.incoming.fruit-in.connector=smallrye-kafka
    mp.messaging.incoming.fruit-in.topic=fruit-in
    mp.messaging.incoming.fruit-in.value.deserializer=com.acme.
    fruit.jackson.FruitDeserializer
    # Configure the Kafka sink (we write to it)
    mp.messaging.outgoing.fruit-out.connector=smallrye-kafka
    mp.messaging.outgoing.fruit-out.topic=fruit-out
    mp.messaging.outgoing.fruit-out.value.serializer=io.quarkus
    .kafka.client.serialization.ObjectMapperSerializer
    package com.acme.fruit.jackson;
    import
    io.quarkus.kafka.client.serialization.ObjectMapperDeseri
    alizer;
    public class FruitDeserializer extends
    ObjectMapperDeserializer {
    public FruitDeserializer() {
    super(Fruit.class);
    }
    } Java file
    pom.xml
    application.properties

    View full-size slide

  61. @systemcraftsman
    SerDe via JSON-B

    io.quarkus
    quarkus-jsonb
    # Configure the Kafka source (we read from it)
    mp.messaging.incoming.fruit-in.connector=smallrye-kafka
    mp.messaging.incoming.fruit-in.topic=fruit-in
    mp.messaging.incoming.fruit-in.value.deserializer=com.acme.
    fruit.jackson.FruitDeserializer
    # Configure the Kafka sink (we write to it)
    mp.messaging.outgoing.fruit-out.connector=smallrye-kafka
    mp.messaging.outgoing.fruit-out.topic=fruit-out
    mp.messaging.outgoing.fruit-out.value.serializer=io.quarkus
    .kafka.client.serialization.JsonbSerializer
    package com.acme.fruit.jsonb;
    import
    io.quarkus.kafka.client.serialization.JsonbDeserializer;
    public class FruitDeserializer extends
    JsonbDeserializer {
    public FruitDeserializer() {
    super(Fruit.class);
    }
    }
    application.properties
    pom.xml
    Java file

    View full-size slide

  62. @systemcraftsman
    Avro
    io.apicurio.registry.utils.serde.AvroKafkaSerializer
    io.apicurio.registry.utils.serde.AvroKafkaDeserializer

    io.quarkus
    quarkus-avro


    io.apicurio
    apicurio-registry-utils-serde

    # Configure the Kafka source (we read from it)
    mp.messaging.incoming.fruit-in.connector=smallrye-kafka
    mp.messaging.incoming.fruit-in.topic=fruit-in
    mp.messaging.incoming.fruit-in.value.deserializer=io.apicur
    io.registry.utils.serde.AvroKafkaDeserializer
    # Configure the Kafka sink (we write to it)
    mp.messaging.outgoing.fruit-out.connector=smallrye-kafka
    mp.messaging.outgoing.fruit-out.topic=fruit-out
    mp.messaging.outgoing.fruit-out.value.serializer=io.apicuri
    o.registry.utils.serde.AvroKafkaSerializer
    application.properties
    pom.xml

    View full-size slide

  63. @systemcraftsman
    Talk is Cheap,
    Show me the Demo!

    View full-size slide

  64. @systemcraftsman
    Resources
    ● MicroProfile Reactive Messaging Spec 2.0
    ● Quarkus Apache Kafka Reference Guide
    ● SmallRye Reactive Messaging
    ● Twitter: @systemcraftsman
    ● Email: [email protected]
    ● Demo: https://github.com/RedHatTraining/AD482-ToT-CoffeeShop

    View full-size slide

  65. @systemcraftsman
    Thank You!
    www.SystemCraftsman.com

    View full-size slide