Slide 1

Slide 1 text

1 Event Streaming using Apache Kafka and Camel Kevin Dubois Sr Principal Developer Advocate Red Hat @kevindubois

Slide 2

Slide 2 text

@kevindubois Kevin Dubois ★ Sr Principal Developer Advocate at Red Hat ★ Based in Belgium 󰎐 ★ Java Champion ★ 🗣 Speak English, Dutch, French, Italian ★ Open Source Contributor (Quarkus, Camel, Knative, ..) youtube.com/@thekevindubois linkedin.com/in/kevindubois github.com/kdubois @kevindubois

Slide 3

Slide 3 text

Disparate Systems & Enterprise Integration

Slide 4

Slide 4 text

What problem needs to be resolved? Sometimes it’s a ‘MICRO-problem’, for instance ... ● Sometimes the focus is on data/protocol transformation. ● This is an integration problem. “I need this information in this specific format and protocol !” 4

Slide 5

Slide 5 text

some other times you have a ‘MACRO-problem’, for instance: ● Here the focus is on interconnecting sources and destinations efficiently. ➔ This also... is an integration problem. “I want an easy and standard way to interconnect my app” 5

Slide 6

Slide 6 text

6 “Good” Integration Bespoke/Custom made Integration Integration with reusable & standard components ● As a quick win, often a problem is initially resolved in a custom manner. ● In the long run this results to be counterproductive.

Slide 7

Slide 7 text

Apache Camel 7

Slide 8

Slide 8 text

🛞Mainly, it’s about not reinventing the wheel 🛞 8

Slide 9

Slide 9 text

Apache Camel Swiss knife of integration Solve integration problem by applying best practices out of the box. Even with microservice architectures. Patterns 300+ Components Lightweight Runtimes Data Formats Packed with 300+ components such as databases, message queues, APIs. Quarkus, Standalone, Spring Boot,Application Servers, and natively on Cloud. Translate messages in multiple formats, and industry standard formats from finance, telco, health-care, and more 9

Slide 10

Slide 10 text

Simple, self-explained Domain Specific Language. Available in Java, YAML, XML DSLs Reactive Routing Engine Backlog Tracer/Tracer Comprehensive Tooling Back pressure model. Smooth flow control. Better thread management. Capturing a trace inside and between Camel route. For better observability Support autocomplete, correction on multiple IDE with LSP Server. Graphical data mapper. from(“kafka:topic”) .to(“grpc:endpoint”) 10 Apache Camel Swiss knife of integration

Slide 11

Slide 11 text

Camel Concepts and Constructs

Slide 12

Slide 12 text

Camel Route 12 connect a source endpoint to a destination endpoint Consumer receives message from the Source Endpoint Producer(s) sends message to the Target Endpoint Processors handle EIPs, routing, transformation, mediation, enrichment, validation, interception Camel route describes the step-by-step movement of a Message from a source endpoint, through arbitrary types of decision-making routines (such as filters and routers) to a destination endpoint (if any) 0, 1 or more Target endpoints 1 Source endpoint -- Camel context holds 1 to N routes Hello Camel from ${routeId} A Camel DSL wires endpoints and processors together to form routes

Slide 13

Slide 13 text

Example: Process and route XML orders 13 Customer Purchase Split Orders Route each order to its service Electronics Others from("file:work/orders/input") .split(xpath("//orders")) .choice() .when(xpath("/order:order/order:type = 'E'")) .to("jms:queue:electronic/us") .otherwise() .to(“jms:queue:other/us”); XML Source endpoint Splitter EIP Target endpoint CBR EIP Target endpoint

Slide 14

Slide 14 text

Camel Components 14 encapsulate APIs to enable it for routes. Endpoint URI component :resource[?options] Core direct :result timer :name?period=1000 file:directoryName[?options] bean:Class?method=myMethod 100+ extra (dynamically loaded) jms:queue:order kafka :myTopic sql:select * from orders where…[?options] smtp://mycompany.mailserver:30[?options]

Slide 15

Slide 15 text

Enterprise Integration Patterns 15 Apache Camel Implementations Splitter .split() Filter .filter() .end() -- optional Content-based Router .choice() .when() .otherwise() Message Translator .setBody() .transform() .bean() .to() -- xslt, jslt, .. And more at https://camel.apache.org/components/4.4.x/eips/enterprise-integration-patterns.html

Slide 16

Slide 16 text

Conditional processing and data extraction 16 with Expression languages Simple language XPath JSONPath when routing, filtering and transforming for basic expressions and conditions for navigating and extracting parts of XML documents for navigating and extracting parts of JSON structures .choice().when(simple("${body} contains 'Camel'")) .filter(simple("${header.type'} == 'gold'")) .setBody(simple("The today is ${date:now:yyyyMMdd} and it is a great day.")) .choice().when(jsonpath("$.orderType == 'Online'")) .filter(jsonpath("$.isActive == true")) .setBody(jsonpath("$.customer.name")) .choice().when(xpath("/customer/type = 'Premium'")) .filter(xpath("/person[@name='James']")) .setBody(xpath("/customer/name/text()")) And more at https://camel.apache.org/components/4.4.x/languages/index.html

Slide 17

Slide 17 text

Data Format Transformations 17 Convert messages to and from various data formats (e.g., JSON, XML, CSV) to Java objects, facilitating the exchange of data between components that expect different formats Advanced Transformations (De)Serialization Perform complex data transformations that go beyond simple format conversions or basic manipulations. Data Type Converters Automatic or explicit conversion of message bodies and headers from one type to another. Built-in converters significantly reduce the need for custom conversion logic Convert message content from one format to another .mashal().jaxb() – XML to Pojo .unmarshal().jaxb() – Pojo to XML .convertBodyTo(String.class) .transform() XML➡XML – XSLT JSON➡JSON – JSLT XML⬅➡JSON – XSLT|XJ Custom processor

Slide 18

Slide 18 text

Custom processing using Beans 18 ▸ Extend routing logic with custom Java code ▸ For complex integrations, custom transformations and business logic implementations ▸ Binding annotations - body, exchange, header, variable, … ▸ Bind or lookup beans in Registry public class MyBean { public String process(String body) { return body.toUpperCase(); } public boolean isGoldCustomer(Exchange exchange) { // ... } } .bean(MyBean.class, "process") .bean(OrderService.class) -- calls a method annotated with @Handler .filter().method(MyBean.class, "isGoldCustomer") .bean("foo") -- call a bean from Registry

Slide 19

Slide 19 text

Make applications externally configurable 19 and define placeholders instead of the actual values db.host = 127.0.0.1 db.port = 8080 db.user = ibek db.pwd = 12345 file.path = file.json in properties file, environment variables, configmaps, secrets, etc. Placeholder functions - env, sys (jvm properties), bean, service Kubernetes placeholder functions - configmap, secret .to("file:foo?fileName={{file.path:/some/path}}") -- with default value .to("file:foo?bufferSize={{?myBufferSize}}") -- optional .log(“What {{configmap:myconfig/drink}} do you want?”) .from("file:{{bean:foo.bar}}") .to("{{env:HOME}}") .to("{{sys:file.separator}}")

Slide 20

Slide 20 text

20 ▸ World's most popular java framework ▸ Focus on speed, simplicity and productivity ▸ For stand-alone Spring applications ▸ Opinionated with simplified build configuration ▸ Many out of the box solutions Camel for Spring Boot ▸ Latest Camel v4 (lighter, faster) ▸ Container-based environments ▸ Camel context with automatic detection of Camel routes ▸ Auto-configuration of Camel components ▸ Starters for many Camel pre-built components ▸ Highly advanced integrations ▸ Custom flows with full control ▸ Long-term support commitment Spring Boot ▸ Integration swiss knife ▸ Super lightweight ▸ Connectors ▸ Most popular Integration Framework Camel Camel for Spring Boot APPLICATION FOUNDATIONS

Slide 21

Slide 21 text

21 ▸ Framework addressing wide range of distributed application architectures ▸ Containers First ▸ Serverless characteristics ▸ Unified configuration ▸ Dev mode with live reload ▸ Kubernetes-native ▸ Hundreds of best-of-breed standard libraries Camel for Quarkus ▸ Latest Camel v4 (lighter, faster) ▸ Container-based environments ▸ Flash boot time ▸ Minimal memory footprint ▸ Standalone & OpenShift ▸ Highly advanced integrations ▸ Custom flows with full control ▸ Long-term support commitment Quarkus ▸ Integration swiss knife ▸ Super lightweight ▸ Connectors ▸ Most popular Integration Framework Camel Camel for Quarkus APPLICATION FOUNDATIONS

Slide 22

Slide 22 text

Supersonic Subatomic Java Quarkus + Native (via GraalVM) 12 MB Quarkus + JVM (via OpenJDK) 73 MB Traditional Cloud-Native Stack 136 MB Quarkus + Native (via GraalVM) 0.016 Seconds Quarkus + JVM (via OpenJDK) 0.943 Seconds Traditional Cloud-Native Stack 4.3 Seconds 22

Slide 23

Slide 23 text

23 A cohesive platform for optimized developer joy: ▸ Zero config, live reload in the blink of an eye ▸ Based on standards, but not limited ▸ Unified configuration ▸ Streamlined code for the 80% common usages, flexible for the 20% ▸ No hassle native executable generation Developer Joy

Slide 24

Slide 24 text

Red Hat build of Apache Camel Connect data streams between disparate systems using Camel as Kafka source and sink connectors Versatile application development toolkit for Enterprise integration that simplifies and standardizes the practices of connecting diverse systems. Build flexible streaming data pipelines interconnected with Camel integrations to decouple the systems Optimize data integrity and processing efficiency with streaming data integration patterns based on Apache Camel for Streaming Data Integration

Slide 25

Slide 25 text

Events EVENT-DRIVEN ARCHITECTURE 25

Slide 26

Slide 26 text

Why Event-Driven Architecture? EVENT-DRIVEN ARCHITECTURE 26 Mirrors the real world The real world is event-driven. Systems generate and respond to events in everyday life, e.g., the human central nervous system. Reduced coupling Traditional RPC-style service architecture results in tightly-bound services. Changes to the application flow typically require service code changes. EDA allows new functionality to be added by adding services that consume existing event streams. Encapsulation Microservices concepts have grown in popularity due to the ability for service teams to develop services in isolation. EDA means that service designers need not be aware of how events are consumed. Fine-grained scaling Services can be independently scaled up and down to meet the event volume. Near real-time latency Customers increasing expect a near real-time experience. Polling on APIs is a delicate trade-off between responsiveness and load. EDA allow apps to react in near real-time without compromise.

Slide 27

Slide 27 text

EVENT-DRIVEN ARCHITECTURE What is an event? 27 Async form of Remote Procedure Call, contains instructions telling recipient what to do, may cause a change of state. Query Command Similar to commands, queries expect a response returning the results, but does not cause any change in state. Event Immutable state and value of a particular entity, which occurred during operation among services.

Slide 28

Slide 28 text

EVENT-DRIVEN ARCHITECTURE Types of events 28 Events stored durably until read by all registered consumers. Traditional store-and-forward brokers. Replayable Consumable Events stored durably for specific period of time or storage capacity. Consumers can move back and forth of the stream. Volatile The event needs to be disseminated to all consumers online at time of publication. Not persisted.

Slide 29

Slide 29 text

EVENT-DRIVEN ARCHITECTURE What is Apache Kafka? 29 Apache Kafka is a distributed system designed for streams. It is built to be an horizontally-scalable, fault-tolerant, commit log, and allows distributed data streams and stream processing applications.

Slide 30

Slide 30 text

Why Kafka? MyPlane Insert Update MyPlane [Insert] MyOtherPlane [Insert] MyPlane [Update]

Slide 31

Slide 31 text

EVENT-DRIVEN ARCHITECTURE Running on OpenShift/Kubernetes 31 Source: ● Strimzi provides: ○ Container images for Apache Kafka, Connect, Zookeeper and MirrorMaker ○ Kubernetes Operators for managing/configuring Apache Kafka clusters, topics and users ○ Kafka Consumer, Producer and Admin clients, Kafka Streams ○ 100% Open source project licensed under Apache License 2.0 ○ Part of the Cloud Native Computing Foundation (CNCF)

Slide 32

Slide 32 text

Connecting data streams between disparate systems 32

Slide 33

Slide 33 text

Connect data streams between disparate systems Camel as Kafka source and sink connectors source source kamelet source pipe Kafka cluster channel sink pipe sink kamelet sink action kamelets action kamelets 2. Kamelets are reusable Camel route templates 3. Action kamelets validate, cleanse, and transform data File system 1. Stream data out and into various systems and services Database API and more with over 100 components IBM MQ Custom

Slide 34

Slide 34 text

Build flexible streaming data pipelines interconnected with Camel integrations to decouple the systems Data stream processing Raw data Processed data Data ingestion Routing to data storage (search index, analytics cluster, …) {;} Protobuf Json / Avro Schema Registry (Apicurio, Azure, …) Validation sources sources Avro deserialization # consumer group [message-offset] destinations SASL-SSL

Slide 35

Slide 35 text

Optimize data integrity and processing efficiency with streaming data integration patterns Data stream processing Raw data Processed data Data ingestion Routing to data storage Error handling - Dead letter pattern - Message redelivery sources sources Exactly-once Filtering Batch processing Aggregation Data transformation destinations Idempotent filtering Content-based routing Throttler Enricher Splitter (search index, analytics cluster, …)

Slide 36

Slide 36 text

Demo

Slide 37

Slide 37 text

Demo

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

rest("/getresults") .get() .bean("vote", "orderedList()") .marshal().json() .convertBodyTo(String.class);

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

rest("/favstack") .post() .consumes("application/json") .produces("application/json") .to("kafka:{{kafka.topic.name}}");

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

from("kafka:{{kafka.topic.name}}") .unmarshal().json(JsonLibrary.Jackson, Response.class) .transacted() .to("bean:vote?method=updateCounter(${body.getShortname})");

Slide 46

Slide 46 text

Quarkus Dev UI

Slide 47

Slide 47 text

mvn clean package -> also generates kubernetes/knative manifests when you add the quarkus-kubernetes extension mvn clean package -Dnative -Dquarkus.kubernetes.deploy -> deploys native binary directly to kubernetes/openshift

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

CAMEL DEV TOOLS 49 for simple creation and testing of Camel routes ▸ VS Code extensions ・ Language Support for Apache Camel by Red Hat ・ Debug Adapter for Apache Camel by Red Hat ・ Kaoto - Lowcode visual designer for Camel integrations ▸ Camel CLI for rapid integration prototyping ▸ Camel REST DSL OpenApi Maven Plugin generates REST DSL source code from OpenAPI specification ▸ CXF SOAP Development Tools Java to WSDL, WSDL to Java, WSDL Validator ▸ Development support

Slide 50

Slide 50 text

Rapid Integration Prototyping 50 using Camel CLI ▸ Immediately start with writing integration code ▸ Automatic dependency resolution for all supported components and EIPs ▸ Export functionality to a full Camel java-based project when done with prototyping ▸ Enables running integration designs from UI tools such as Kaoto Iterate and experiment hot reload Log & watch Debug Manage Monitor camel export --runtime=quarkus --gav=com.foo:acme:1.0-SNAPSHOT create commit CI/CD camel init routesA.xml camel run * --dev

Slide 51

Slide 51 text

HawtIO diagnostic console 51 for troubleshooting and debugging Camel routes ▸ React + PatternFly v4 ▸ New plugin mechanism ▸ Standalone -> Hawtio v4 ▸ OpenShift -> Hawtio Online v2 ▸ Camel plugin ・ Contexts/Routes/Endpoints ・ Route diagram ・ Trace/Debug ・ Source ・ Simple statistics ▸ Other plugins - Connect, JMX, OAuth (Keycloak), Artemis ▸ Quarkus (JVM only) and SpringBoot

Slide 52

Slide 52 text

Kaoto in Visual Studio Code 52 Visual Designer for Camel ▸ Run with Camel CLI ▸ Debug with Debug Adapter ▸ Language support for syntax highlighting and autocompletion

Slide 53

Slide 53 text

Camel 4 Quickstarts 53 ▸ Message bridge between AMQ and IBM MQ with connection pooling and XA transactions ▸ ETL between multiple JDBC data sources ▸ JPA Idempotent repository ▸ MQTT and ElasticSearch ▸ Monitoring with Micrometer, Prometheus and Grafana ▸ AMQP and Salesforce ▸ and many more in development Quarkus Quickstarts Spring Boot Quickstarts

Slide 54

Slide 54 text

Free Developer e-Books & tutorials! developers.redhat.com/eventtutorials

Slide 55

Slide 55 text

redhat.com/en/products/application-foundations

Slide 56

Slide 56 text

Thank You youtube.com/@thekevindubois linkedin.com/in/kevindubois github.com/kdubois @kevindubois.com @[email protected] Slides