Slide 1

Slide 1 text

with the Red Hat Build of Apache Camel and Red Hat AMQ Connecting Disparate Systems Zineb Bendhiba Senior Software Engineer @ZinebBendhiba Kevin Dubois Principal Developer Advocate @KevinDubois

Slide 2

Slide 2 text

Zineb Bendhiba ● Senior Software Engineer at Red Hat ● Apache Camel Committer and PMC Linkedin: zbendhiba Twitter: @ZinebBendhiba Mastodon: @[email protected] Github: zbendhiba

Slide 3

Slide 3 text

Kevin Dubois ★ Principal Developer Advocate at Red Hat ★ Java Champion ★ Based in Belgium 󰎐 ★ 🗣 English, Dutch, French, Italian ★ Open Source Contributor (Quarkus, Podman, Camel,..) @[email protected] youtube.com/@thekevindubois linkedin.com/in/kevindubois github.com/kdubois @kevindubois.com

Slide 4

Slide 4 text

Disparate Systems & Enterprise Integration

Slide 5

Slide 5 text

What problem needs to be resolved? Sometimes it’s a ‘MICRO-problem’, for instance ... ● Sometimes the focus is on data/protocol transformation. ● This is an integration problem. “I need this information in this specific format and protocol !” 5

Slide 6

Slide 6 text

some other times you have a ‘MACRO-problem’, for instance: ● Here the focus is on interconnecting sources and destinations efficiently. ➔ This also... is an integration problem. “I want an easy and standard way to interconnect my app” 6

Slide 7

Slide 7 text

7 “Good” Integration Bespoke/Custom made Integration Integration with reusable & standard components ● As a quick win, often a problem is initially resolved in a custom manner. ● In the long run this results to be counterproductive.

Slide 8

Slide 8 text

8 Kubernetes & Cluster Services Install | Over-the-air updates | Networking | Ingress | Storage | Monitoring | Log forwarding | Registry | Authorization | Containers | VMs | Operators | Helm Linux (container host operating system) Physical Virtual Private cloud Public cloud Edge Integrated DevOps Services Service Mesh | Serverless | Builds | Pipelines | GitOps |Tracing | Log Management | Cost Management | Migration Tools Advanced Management & Security Multicluster Management | Cluster Security| Global Registry | Cluster Data Management Red Hat OpenShift on IBM Cloud Red Hat OpenShift Service on AWS Azure Red Hat OpenShift OpenShift Dedicated Self-Managed Platforms OpenShift Cloud Services Cloud-native app dev platform

Slide 9

Slide 9 text

9 9 Application servers Data integration and transformation Runtimes and frameworks Single sign-on API management Migration toolkits Enterprise integration In Memory Cache Service composition Change data capture Real-time messaging and event streaming The Red Hat Build of Camel and Red Hat AMQ are part of Red Hat Application Foundations; a comprehensive set of components and frameworks to modernize and accelerate application development

Slide 10

Slide 10 text

Apache Camel 10

Slide 11

Slide 11 text

🛞Mainly, it’s about not reinventing the wheel 🛞 11

Slide 12

Slide 12 text

Apache Camel Swiss knife of integration Solve integration problem by applying best practices out of the box. Even with microservice architectures. Patterns 300+ Components Lightweight Runtimes Data Formats Packed with 300+ components such as databases, message queues, APIs. Quarkus, Standalone, Spring Boot,Application Servers, and natively on Cloud. Translate messages in multiple formats, and industry standard formats from finance, telco, health-care, and more 12

Slide 13

Slide 13 text

Simple, self-explained Domain Specific Language. Available in Java, YAML, XML DSLs Reactive Routing Engine Backlog Tracer/Tracer Comprehensive Tooling Back pressure model. Smooth flow control. Better thread management. Capturing a trace inside and between Camel route. For better observability Support autocomplete, correction on multiple IDE with LSP Server. Graphical data mapper. from(“kafka:topic”) .to(“grpc:endpoint”) 13 Apache Camel Swiss knife of integration

Slide 14

Slide 14 text

Camel Concepts and Constructs

Slide 15

Slide 15 text

Camel Route 15 connect a source endpoint to a destination endpoint Consumer receives message from the Source Endpoint Producer(s) sends message to the Target Endpoint Processors handle EIPs, routing, transformation, mediation, enrichment, validation, interception Camel route describes the step-by-step movement of a Message from a source endpoint, through arbitrary types of decision-making routines (such as filters and routers) to a destination endpoint (if any) 0, 1 or more Target endpoints 1 Source endpoint -- Camel context holds 1 to N routes Hello Camel from ${routeId} A Camel DSL wires endpoints and processors together to form routes

Slide 16

Slide 16 text

Example: Process and route XML orders 16 Customer Purchase Split Orders Route each order to its service Electronics Others from("file:work/orders/input") .split(xpath("//orders")) .choice() .when(xpath("/order:order/order:type = 'E'")) .to("jms:queue:electronic/us") .otherwise() .to(“jms:queue:other/us”); XML Source endpoint Splitter EIP Target endpoint CBR EIP Target endpoint

Slide 17

Slide 17 text

Camel Components 17 encapsulate APIs to enable it for routes. Endpoint URI component:resource[?options] Core direct:result timer:name?period=1000 file:directoryName[?options] bean:Class?method=myMethod 100+ extra (dynamically loaded) jms:queue:order kafka:myTopic sql:select * from orders where…[?options] smtp://mycompany.mailserver:30[?options]

Slide 18

Slide 18 text

Enterprise Integration Patterns 18 Apache Camel Implementations Splitter .split() Filter .filter() .end() -- optional Content-based Router .choice() .when() .otherwise() Message Translator .setBody() .transform() .bean() .to() -- xslt, jslt, .. And more at https://camel.apache.org/components/4.4.x/eips/enterprise-integration-patterns.html

Slide 19

Slide 19 text

Conditional processing and data extraction 19 with Expression languages Simple language XPath JSONPath when routing, filtering and transforming for basic expressions and conditions for navigating and extracting parts of XML documents for navigating and extracting parts of JSON structures .choice().when(simple("${body} contains 'Camel'")) .filter(simple("${header.type'} == 'gold'")) .setBody(simple("The today is ${date:now:yyyyMMdd} and it is a great day.")) .choice().when(jsonpath("$.orderType == 'Online'")) .filter(jsonpath("$.isActive == true")) .setBody(jsonpath("$.customer.name")) .choice().when(xpath("/customer/type = 'Premium'")) .filter(xpath("/person[@name='James']")) .setBody(xpath("/customer/name/text()")) And more at https://camel.apache.org/components/4.4.x/languages/index.html

Slide 20

Slide 20 text

Data Format Transformations 20 Convert messages to and from various data formats (e.g., JSON, XML, CSV) to Java objects, facilitating the exchange of data between components that expect different formats Advanced Transformations (De)Serialization Perform complex data transformations that go beyond simple format conversions or basic manipulations. Data Type Converters Automatic or explicit conversion of message bodies and headers from one type to another. Built-in converters significantly reduce the need for custom conversion logic Convert message content from one format to another .mashal().jaxb() – XML to Pojo .unmarshal().jaxb() – Pojo to XML .convertBodyTo(String.class) .transform() XML➡XML – XSLT JSON➡JSON – JSLT XML⬅➡JSON – XSLT|XJ Custom processor

Slide 21

Slide 21 text

Custom processing using Beans 21 ▸ Extend routing logic with custom Java code ▸ For complex integrations, custom transformations and business logic implementations ▸ Binding annotations - body, exchange, header, variable, … ▸ Bind or lookup beans in Registry public class MyBean { public String process(String body) { return body.toUpperCase(); } public boolean isGoldCustomer(Exchange exchange) { // ... } } .bean(MyBean.class, "process") .bean(OrderService.class) -- calls a method annotated with @Handler .filter().method(MyBean.class, "isGoldCustomer") .bean("foo") -- call a bean from Registry

Slide 22

Slide 22 text

Make applications externally configurable 22 and define placeholders instead of the actual values db.host = 127.0.0.1 db.port = 8080 db.user = ibek db.pwd = 12345 file.path = file.json in properties file, environment variables, configmaps, secrets, etc. Placeholder functions - env, sys (jvm properties), bean, service Kubernetes placeholder functions - configmap, secret .to("file:foo?fileName={{file.path:/some/path}}") -- with default value .to("file:foo?bufferSize={{?myBufferSize}}") -- optional .log(“What {{configmap:myconfig/drink}} do you want?”) .from("file:{{bean:foo.bar}}") .to("{{env:HOME}}") .to("{{sys:file.separator}}")

Slide 23

Slide 23 text

23 ▸ World's most popular java framework ▸ Focus on speed, simplicity and productivity ▸ For stand-alone Spring applications ▸ Opinionated with simplified build configuration ▸ Many out of the box solutions Camel for Spring Boot ▸ Latest Camel v4 (lighter, faster) ▸ Container-based environments ▸ Camel context with automatic detection of Camel routes ▸ Auto-configuration of Camel components ▸ Starters for many Camel pre-built components ▸ Highly advanced integrations ▸ Custom flows with full control ▸ Long-term support commitment Spring Boot ▸ Integration swiss knife ▸ Super lightweight ▸ Connectors ▸ Most popular Integration Framework Camel Camel for Spring Boot APPLICATION FOUNDATIONS

Slide 24

Slide 24 text

24 ▸ Framework addressing wide range of distributed application architectures ▸ Containers First ▸ Serverless characteristics ▸ Unified configuration ▸ Dev mode with live reload ▸ Kubernetes-native ▸ Hundreds of best-of-breed standard libraries Camel for Quarkus ▸ Latest Camel v4 (lighter, faster) ▸ Container-based environments ▸ Flash boot time ▸ Minimal memory footprint ▸ Standalone & OpenShift ▸ Highly advanced integrations ▸ Custom flows with full control ▸ Long-term support commitment Quarkus ▸ Integration swiss knife ▸ Super lightweight ▸ Connectors ▸ Most popular Integration Framework Camel Camel for Quarkus APPLICATION FOUNDATIONS

Slide 25

Slide 25 text

Supersonic Subatomic Java Quarkus + Native (via GraalVM) 12 MB Quarkus + JVM (via OpenJDK) 73 MB Traditional Cloud-Native Stack 136 MB Quarkus + Native (via GraalVM) 0.016 Seconds Quarkus + JVM (via OpenJDK) 0.943 Seconds Traditional Cloud-Native Stack 4.3 Seconds 25

Slide 26

Slide 26 text

26 A cohesive platform for optimized developer joy: ▸ Zero config, live reload in the blink of an eye ▸ Based on standards, but not limited ▸ Unified configuration ▸ Streamlined code for the 80% common usages, flexible for the 20% ▸ No hassle native executable generation Developer Joy

Slide 27

Slide 27 text

Red Hat build of Apache Camel Connect data streams between disparate systems using Camel as Kafka source and sink connectors Versatile application development toolkit for Enterprise integration that simplifies and standardizes the practices of connecting diverse systems. Build flexible streaming data pipelines interconnected with Camel integrations to decouple the systems Optimize data integrity and processing efficiency with streaming data integration patterns based on Apache Camel for Streaming Data Integration

Slide 28

Slide 28 text

Connecting data streams between disparate systems 28

Slide 29

Slide 29 text

CONFIDENTIAL Designator 29 29 Operational Efficiency with Red Hat AMQ streams 29 Kafka optimized for OpenShift ▸ Strimzi.io provides Kube-native fit for Kafka ▸ AMQ streams is the productized and supported version of strimzi ▸ Addresses management and operational complexity of enterprise Kafka architecture via OpenShift Operator ▸ Runs on-prem and across multiple clouds INTRODUCTION

Slide 30

Slide 30 text

Connect data streams between disparate systems Camel as Kafka source and sink connectors source source kamelet source pipe Kafka cluster channel sink pipe sink kamelet sink action kamelets action kamelets 2. Kamelets are reusable Camel route templates 3. Action kamelets validate, cleanse, and transform data File system 1. Stream data out and into various systems and services Database API and more with over 100 components IBM MQ Custom

Slide 31

Slide 31 text

Build flexible streaming data pipelines interconnected with Camel integrations to decouple the systems Data stream processing Raw data Processed data Data ingestion Routing to data storage (search index, analytics cluster, …) {;} Protobuf Json / Avro Schema Registry (Apicurio, Azure, …) Validation sources sources Avro deserialization # consumer group [message-offset] destinations SASL-SSL

Slide 32

Slide 32 text

Optimize data integrity and processing efficiency with streaming data integration patterns Data stream processing Raw data Processed data Data ingestion Routing to data storage Error handling - Dead letter pattern - Message redelivery sources sources Exactly-once Filtering Batch processing Aggregation Data transformation destinations Idempotent filtering Content-based routing Throttler Enricher Splitter (search index, analytics cluster, …)

Slide 33

Slide 33 text

Demo

Slide 34

Slide 34 text

Demo

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

rest("/getresults") .get() .bean("vote", "orderedList()") .marshal().json() .convertBodyTo(String.class);

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

rest("/favstack") .post() .consumes("application/json") .produces("application/json") .to("kafka:{{kafka.topic.name}}");

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

from("kafka:{{kafka.topic.name}}") .unmarshal().json(JsonLibrary.Jackson, Response.class) .transacted() .to("bean:vote?method=updateCounter(${body.getShortname})");

Slide 43

Slide 43 text

Quarkus Dev UI

Slide 44

Slide 44 text

mvn clean package -> also generates kubernetes/knative manifests when you add the quarkus-kubernetes extension mvn clean package -Dnative -Dquarkus.kubernetes.deploy -> deploys native binary directly to kubernetes/openshift

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

Red Hat build of Apache Camel - Roadmap Highlights 46 ▸ Migration guides, solution pattern and utilities for Red Hat Fuse / Camel 2 modernization ▸ (tech-preview) HawtIO diagnostic console for Camel ▸ (tech-preview) Kaoto visual designer in VS Code ▸ Camel 4.x lifecycle extended to Oct 2028 For more details, please refer to the release notes What’s new in Camel 4 in the next 12 months: ▸ RHBAC 4.4 ▸ Camel K operator ▸ Camel health dashboard ▸ Visual data mapping in Kaoto For complete information and additional details, please refer to the customer-facing roadmap deck Coming soon…

Slide 47

Slide 47 text

Camel with Kamelets Use Kamelets where you need, as sources, sinks, or mid-flow actions Source Sink Action KMLT Incoming Data Outgoing Data KMLT KMLT Camel pipe ... ... more Camel code Kamelets are Kamel route snippets KMLT

Slide 48

Slide 48 text

bind Kamelets to form a running integration unit Kafka client Kafka app Kafka app Kafka client Camel Pipes

Slide 49

Slide 49 text

Camel K operator 49 for managed Camel experience ▸ Automate operational tasks ・ Self-healing ・ Red Hat Camel version upgrades ▸ Simplify setup tasks ・ Config maps / Secrets ・ Tuning - jvm, quarkus, camel, components, .. ・ Enable Camel in Serverless ・ Camel in offline env. - disconnected mode ・ Enable multitenancy / HA deployments ・ Adhoc sourceless integrations ▸ Camel-specific integration dashboard / console ・ Monitoring Camel integrations - entry point for what's running in the cluster ▸ Personas ・ Platform engineer ・ Non-java developers (AI/python developers, Serverless developers) for adhoc integrations

Slide 50

Slide 50 text

CAMEL DEV TOOLS 50 for simple creation and testing of Camel routes ▸ VS Code extensions ・ Language Support for Apache Camel by Red Hat ・ Debug Adapter for Apache Camel by Red Hat ・ Kaoto - Lowcode visual designer for Camel integrations ▸ Camel CLI for rapid integration prototyping ▸ Camel REST DSL OpenApi Maven Plugin generates REST DSL source code from OpenAPI specification ▸ CXF SOAP Development Tools Java to WSDL, WSDL to Java, WSDL Validator ▸ Development support

Slide 51

Slide 51 text

Rapid Integration Prototyping 51 using Camel CLI ▸ Immediately start with writing integration code ▸ Automatic dependency resolution for all supported components and EIPs ▸ Export functionality to a full Camel java-based project when done with prototyping ▸ Enables running integration designs from UI tools such as Kaoto Iterate and experiment hot reload Log & watch Debug Manage Monitor camel export --runtime=quarkus --gav=com.foo:acme:1.0-SNAPSHOT create commit CI/CD camel init routesA.xml camel run * --dev

Slide 52

Slide 52 text

▸ Some components (S3) contribute to the readiness status of application - if a connection is not established, the pod becomes “not ready” 52 Camel on OpenShift Camel K Operator Camel Service Camel Pipe Connector Camel Service Source to Image Synthetic integration Operator-managed Camel integration services Sourceless integration

Slide 53

Slide 53 text

HawtIO diagnostic console 53 for troubleshooting and debugging Camel routes ▸ React + PatternFly v4 ▸ New plugin mechanism ▸ Standalone -> Hawtio v4 ▸ OpenShift -> Hawtio Online v2 ▸ Camel plugin ・ Contexts/Routes/Endpoints ・ Route diagram ・ Trace/Debug ・ Source ・ Simple statistics ▸ Other plugins - Connect, JMX, OAuth (Keycloak), Artemis ▸ Quarkus (JVM only) and SpringBoot

Slide 54

Slide 54 text

Kaoto in Visual Studio Code 54 Visual Designer for Camel ▸ Run with Camel CLI ▸ Debug with Debug Adapter ▸ Language support for syntax highlighting and autocompletion

Slide 55

Slide 55 text

Demo 55

Slide 56

Slide 56 text

Camel Reference Architecture and Patterns 56

Slide 57

Slide 57 text

Camel 4 Quickstarts 57 ▸ Message bridge between AMQ and IBM MQ with connection pooling and XA transactions ▸ ETL between multiple JDBC data sources ▸ JPA Idempotent repository ▸ MQTT and ElasticSearch ▸ Monitoring with Micrometer, Prometheus and Grafana ▸ AMQP and Salesforce ▸ and many more in development Quarkus Quickstarts Spring Boot Quickstarts

Slide 58

Slide 58 text

Red Hat Summit and AnsibleFest 2024 Modern Application Development Roadshow ▸ A single-day experience for you ▸ Tailored hands-on labs and learning ▸ Explore modern tooling, techniques, and architectures red.ht/MAD

Slide 59

Slide 59 text

Free Developer e-Books & tutorials! developers.redhat.com/eventtutorials

Slide 60

Slide 60 text

linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHat Thank you

Slide 61

Slide 61 text

linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHat Thank you