Red Hat Summit 2024 - Connecting Disparate Systems with Red Hat Build of Camel and AMQ Streams

Transcript

1. with the Red Hat Build of Apache Camel and Red Hat AMQ Connecting Disparate Systems Zineb Bendhiba Senior Software Engineer @ZinebBendhiba Kevin Dubois Principal Developer Advocate @KevinDubois
2. Zineb Bendhiba • Senior Software Engineer at Red Hat • Apache Camel Committer and PMC Linkedin: zbendhiba Twitter: @ZinebBendhiba Mastodon: @zbendhiba@mastodon.social Github: zbendhiba
3. Kevin Dubois ★ Principal Developer Advocate at Red Hat ★ Java Champion ★ Based in Belgium 󰎐 ★ 🗣 English, Dutch, French, Italian ★ Open Source Contributor (Quarkus, Podman, Camel,..) @kevindubois@mastodon.social youtube.com/@thekevindubois linkedin.com/in/kevindubois github.com/kdubois @kevindubois.com
4. Disparate Systems & Enterprise Integration
5. What problem needs to be resolved? Sometimes it’s a ‘MICRO-problem’, for instance ... • Sometimes the focus is on data/protocol transformation. • This is an integration problem. “I need this information in this specific format and protocol !” 5
6. some other times you have a ‘MACRO-problem’, for instance: • Here the focus is on interconnecting sources and destinations efficiently. ➔ This also... is an integration problem. “I want an easy and standard way to interconnect my app” 6
7. 7 “Good” Integration Bespoke/Custom made Integration Integration with reusable & standard components • As a quick win, often a problem is initially resolved in a custom manner. • In the long run this results to be counterproductive.
8. 8 Kubernetes & Cluster Services Install | Over-the-air updates | Networking | Ingress | Storage | Monitoring | Log forwarding | Registry | Authorization | Containers | VMs | Operators | Helm Linux (container host operating system) Physical Virtual Private cloud Public cloud Edge Integrated DevOps Services Service Mesh | Serverless | Builds | Pipelines | GitOps |Tracing | Log Management | Cost Management | Migration Tools Advanced Management & Security Multicluster Management | Cluster Security| Global Registry | Cluster Data Management Red Hat OpenShift on IBM Cloud Red Hat OpenShift Service on AWS Azure Red Hat OpenShift OpenShift Dedicated Self-Managed Platforms OpenShift Cloud Services Cloud-native app dev platform
9. 9 9 Application servers Data integration and transformation Runtimes and frameworks Single sign-on API management Migration toolkits Enterprise integration In Memory Cache Service composition Change data capture Real-time messaging and event streaming The Red Hat Build of Camel and Red Hat AMQ are part of Red Hat Application Foundations; a comprehensive set of components and frameworks to modernize and accelerate application development
10. Apache Camel 10
11. 🛞Mainly, it’s about not reinventing the wheel 🛞 11
12. Apache Camel Swiss knife of integration Solve integration problem by applying best practices out of the box. Even with microservice architectures. Patterns 300+ Components Lightweight Runtimes Data Formats Packed with 300+ components such as databases, message queues, APIs. Quarkus, Standalone, Spring Boot,Application Servers, and natively on Cloud. Translate messages in multiple formats, and industry standard formats from finance, telco, health-care, and more 12
13. Simple, self-explained Domain Specific Language. Available in Java, YAML, XML DSLs Reactive Routing Engine Backlog Tracer/Tracer Comprehensive Tooling Back pressure model. Smooth flow control. Better thread management. Capturing a trace inside and between Camel route. For better observability Support autocomplete, correction on multiple IDE with LSP Server. Graphical data mapper. from(“kafka:topic”) .to(“grpc:endpoint”) 13 Apache Camel Swiss knife of integration
14. Camel Concepts and Constructs
15. Camel Route 15 connect a source endpoint to a destination endpoint Consumer receives message from the Source Endpoint Producer(s) sends message to the Target Endpoint Processors handle EIPs, routing, transformation, mediation, enrichment, validation, interception Camel route describes the step-by-step movement of a Message from a source endpoint, through arbitrary types of decision-making routines (such as filters and routers) to a destination endpoint (if any) 0, 1 or more Target endpoints 1 Source endpoint <camel> -- Camel context holds 1 to N routes <route> <from uri="timer:sec?period=1000"/> <setBody> <simple>Hello Camel from ${routeId}</simple> </setBody> <log message="${body}"/> </route> </camel> <XML DSL> A Camel DSL wires endpoints and processors together to form routes
16. Example: Process and route XML orders 16 Customer Purchase Split Orders Route each order to its service Electronics Others from("file:work/orders/input") .split(xpath("//orders")) .choice() .when(xpath("/order:order/order:type = 'E'")) .to("jms:queue:electronic/us") .otherwise() .to(“jms:queue:other/us”); XML Source endpoint Splitter EIP Target endpoint CBR EIP <Java DSL> Target endpoint
17. Camel Components 17 encapsulate APIs to enable it for routes. Endpoint URI component:resource[?options] Core direct:result timer:name?period=1000 file:directoryName[?options] bean:Class?method=myMethod 100+ extra (dynamically loaded) jms:queue:order kafka:myTopic sql:select * from orders where…[?options] smtp://mycompany.mailserver:30[?options]
18. Enterprise Integration Patterns 18 Apache Camel Implementations Splitter .split(<expression>) Filter .filter(<expression>) <step(s)> .end() -- optional Content-based Router .choice() .when(<expression>) <steps> .otherwise() <steps> Message Translator .setBody(<expression>) .transform(<expression>) .bean(<customBean>) .to() -- xslt, jslt, .. And more at https://camel.apache.org/components/4.4.x/eips/enterprise-integration-patterns.html
19. Conditional processing and data extraction 19 with Expression languages Simple language XPath JSONPath when routing, filtering and transforming for basic expressions and conditions for navigating and extracting parts of XML documents for navigating and extracting parts of JSON structures .choice().when(simple("${body} contains 'Camel'")) .filter(simple("${header.type'} == 'gold'")) .setBody(simple("The today is ${date:now:yyyyMMdd} and it is a great day.")) .choice().when(jsonpath("$.orderType == 'Online'")) .filter(jsonpath("$.isActive == true")) .setBody(jsonpath("$.customer.name")) .choice().when(xpath("/customer/type = 'Premium'")) .filter(xpath("/person[@name='James']")) .setBody(xpath("/customer/name/text()")) And more at https://camel.apache.org/components/4.4.x/languages/index.html
20. Data Format Transformations 20 Convert messages to and from various data formats (e.g., JSON, XML, CSV) to Java objects, facilitating the exchange of data between components that expect different formats Advanced Transformations (De)Serialization Perform complex data transformations that go beyond simple format conversions or basic manipulations. Data Type Converters Automatic or explicit conversion of message bodies and headers from one type to another. Built-in converters significantly reduce the need for custom conversion logic Convert message content from one format to another .mashal().jaxb() – XML to Pojo .unmarshal().jaxb() – Pojo to XML .convertBodyTo(String.class) .transform(<dataType>) XML➡XML – XSLT JSON➡JSON – JSLT XML⬅➡JSON – XSLT|XJ Custom processor
21. Custom processing using Beans 21 ▸ Extend routing logic with custom Java code ▸ For complex integrations, custom transformations and business logic implementations ▸ Binding annotations - body, exchange, header, variable, … ▸ Bind or lookup beans in Registry public class MyBean { public String process(String body) { return body.toUpperCase(); } public boolean isGoldCustomer(Exchange exchange) { // ... } } .bean(MyBean.class, "process") .bean(OrderService.class) -- calls a method annotated with @Handler .filter().method(MyBean.class, "isGoldCustomer") .bean("foo") -- call a bean from Registry
22. Make applications externally configurable 22 and define placeholders instead of the actual values db.host = 127.0.0.1 db.port = 8080 db.user = ibek db.pwd = 12345 file.path = file.json in properties file, environment variables, configmaps, secrets, etc. Placeholder functions - env, sys (jvm properties), bean, service Kubernetes placeholder functions - configmap, secret .to("file:foo?fileName={{file.path:/some/path}}") -- with default value .to("file:foo?bufferSize={{?myBufferSize}}") -- optional .log(“What {{configmap:myconfig/drink}} do you want?”) .from("file:{{bean:foo.bar}}") .to("{{env:HOME}}") .to("{{sys:file.separator}}")
23. 23 ▸ World's most popular java framework ▸ Focus on speed, simplicity and productivity ▸ For stand-alone Spring applications ▸ Opinionated with simplified build configuration ▸ Many out of the box solutions Camel for Spring Boot ▸ Latest Camel v4 (lighter, faster) ▸ Container-based environments ▸ Camel context with automatic detection of Camel routes ▸ Auto-configuration of Camel components ▸ Starters for many Camel pre-built components ▸ Highly advanced integrations ▸ Custom flows with full control ▸ Long-term support commitment Spring Boot ▸ Integration swiss knife ▸ Super lightweight ▸ Connectors ▸ Most popular Integration Framework Camel Camel for Spring Boot APPLICATION FOUNDATIONS
24. 24 ▸ Framework addressing wide range of distributed application architectures ▸ Containers First ▸ Serverless characteristics ▸ Unified configuration ▸ Dev mode with live reload ▸ Kubernetes-native ▸ Hundreds of best-of-breed standard libraries Camel for Quarkus ▸ Latest Camel v4 (lighter, faster) ▸ Container-based environments ▸ Flash boot time ▸ Minimal memory footprint ▸ Standalone & OpenShift ▸ Highly advanced integrations ▸ Custom flows with full control ▸ Long-term support commitment Quarkus ▸ Integration swiss knife ▸ Super lightweight ▸ Connectors ▸ Most popular Integration Framework Camel Camel for Quarkus APPLICATION FOUNDATIONS
25. Supersonic Subatomic Java Quarkus + Native (via GraalVM) 12 MB Quarkus + JVM (via OpenJDK) 73 MB Traditional Cloud-Native Stack 136 MB Quarkus + Native (via GraalVM) 0.016 Seconds Quarkus + JVM (via OpenJDK) 0.943 Seconds Traditional Cloud-Native Stack 4.3 Seconds 25
26. 26 A cohesive platform for optimized developer joy: ▸ Zero config, live reload in the blink of an eye ▸ Based on standards, but not limited ▸ Unified configuration ▸ Streamlined code for the 80% common usages, flexible for the 20% ▸ No hassle native executable generation Developer Joy
27. Red Hat build of Apache Camel Connect data streams between disparate systems using Camel as Kafka source and sink connectors Versatile application development toolkit for Enterprise integration that simplifies and standardizes the practices of connecting diverse systems. Build flexible streaming data pipelines interconnected with Camel integrations to decouple the systems Optimize data integrity and processing efficiency with streaming data integration patterns based on Apache Camel for Streaming Data Integration
28. Connecting data streams between disparate systems 28
29. CONFIDENTIAL Designator 29 29 Operational Efficiency with Red Hat AMQ streams 29 Kafka optimized for OpenShift ▸ Strimzi.io provides Kube-native fit for Kafka ▸ AMQ streams is the productized and supported version of strimzi ▸ Addresses management and operational complexity of enterprise Kafka architecture via OpenShift Operator ▸ Runs on-prem and across multiple clouds INTRODUCTION
30. Connect data streams between disparate systems Camel as Kafka source and sink connectors source source kamelet source pipe Kafka cluster channel sink pipe sink kamelet sink action kamelets action kamelets 2. Kamelets are reusable Camel route templates 3. Action kamelets validate, cleanse, and transform data File system 1. Stream data out and into various systems and services Database API and more with over 100 components IBM MQ Custom
31. Build flexible streaming data pipelines interconnected with Camel integrations to decouple the systems Data stream processing Raw data Processed data Data ingestion Routing to data storage (search index, analytics cluster, …) {;} Protobuf Json / Avro Schema Registry (Apicurio, Azure, …) Validation sources sources Avro deserialization # consumer group [message-offset] destinations SASL-SSL
32. Optimize data integrity and processing efficiency with streaming data integration patterns Data stream processing Raw data Processed data Data ingestion Routing to data storage Error handling - Dead letter pattern - Message redelivery sources sources Exactly-once Filtering Batch processing Aggregation Data transformation destinations Idempotent filtering Content-based routing Throttler Enricher Splitter (search index, analytics cluster, …)
33. Demo
34. Demo
36. rest("/getresults") .get() .bean("vote", "orderedList()") .marshal().json() .convertBodyTo(String.class);
39. rest("/favstack") .post() .consumes("application/json") .produces("application/json") .to("kafka:{{kafka.topic.name}}");
42. from("kafka:{{kafka.topic.name}}") .unmarshal().json(JsonLibrary.Jackson, Response.class) .transacted() .to("bean:vote?method=updateCounter(${body.getShortname})");
43. Quarkus Dev UI
44. mvn clean package -> also generates kubernetes/knative manifests when you add the quarkus-kubernetes extension mvn clean package -Dnative -Dquarkus.kubernetes.deploy -> deploys native binary directly to kubernetes/openshift
46. Red Hat build of Apache Camel - Roadmap Highlights 46 ▸ Migration guides, solution pattern and utilities for Red Hat Fuse / Camel 2 modernization ▸ (tech-preview) HawtIO diagnostic console for Camel ▸ (tech-preview) Kaoto visual designer in VS Code ▸ Camel 4.x lifecycle extended to Oct 2028 For more details, please refer to the release notes What’s new in Camel 4 in the next 12 months: ▸ RHBAC 4.4 ▸ Camel K operator ▸ Camel health dashboard ▸ Visual data mapping in Kaoto For complete information and additional details, please refer to the customer-facing roadmap deck Coming soon…
47. Camel with Kamelets Use Kamelets where you need, as sources, sinks, or mid-flow actions Source Sink Action KMLT Incoming Data Outgoing Data KMLT KMLT Camel pipe ... ... more Camel code Kamelets are Kamel route snippets KMLT
48. bind Kamelets to form a running integration unit Kafka client Kafka app Kafka app Kafka client Camel Pipes
49. Camel K operator 49 for managed Camel experience ▸ Automate operational tasks ･ Self-healing ･ Red Hat Camel version upgrades ▸ Simplify setup tasks ･ Config maps / Secrets ･ Tuning - jvm, quarkus, camel, components, .. ･ Enable Camel in Serverless ･ Camel in offline env. - disconnected mode ･ Enable multitenancy / HA deployments ･ Adhoc sourceless integrations ▸ Camel-specific integration dashboard / console ･ Monitoring Camel integrations - entry point for what's running in the cluster ▸ Personas ･ Platform engineer ･ Non-java developers (AI/python developers, Serverless developers) for adhoc integrations
50. CAMEL DEV TOOLS 50 for simple creation and testing of Camel routes ▸ VS Code extensions ･ Language Support for Apache Camel by Red Hat ･ Debug Adapter for Apache Camel by Red Hat ･ Kaoto - Lowcode visual designer for Camel integrations ▸ Camel CLI for rapid integration prototyping ▸ Camel REST DSL OpenApi Maven Plugin generates REST DSL source code from OpenAPI specification ▸ CXF SOAP Development Tools Java to WSDL, WSDL to Java, WSDL Validator ▸ Development support
51. Rapid Integration Prototyping 51 using Camel CLI ▸ Immediately start with writing integration code ▸ Automatic dependency resolution for all supported components and EIPs ▸ Export functionality to a full Camel java-based project when done with prototyping ▸ Enables running integration designs from UI tools such as Kaoto Iterate and experiment hot reload Log & watch Debug Manage Monitor camel export --runtime=quarkus --gav=com.foo:acme:1.0-SNAPSHOT create commit CI/CD camel init routesA.xml camel run * --dev
52. ▸ Some components (S3) contribute to the readiness status of application - if a connection is not established, the pod becomes “not ready” 52 Camel on OpenShift Camel K Operator Camel Service Camel Pipe Connector Camel Service Source to Image Synthetic integration Operator-managed Camel integration services Sourceless integration
53. HawtIO diagnostic console 53 for troubleshooting and debugging Camel routes ▸ React + PatternFly v4 ▸ New plugin mechanism ▸ Standalone -> Hawtio v4 ▸ OpenShift -> Hawtio Online v2 ▸ Camel plugin ･ Contexts/Routes/Endpoints ･ Route diagram ･ Trace/Debug ･ Source ･ Simple statistics ▸ Other plugins - Connect, JMX, OAuth (Keycloak), Artemis ▸ Quarkus (JVM only) and SpringBoot
54. Kaoto in Visual Studio Code 54 Visual Designer for Camel ▸ Run with Camel CLI ▸ Debug with Debug Adapter ▸ Language support for syntax highlighting and autocompletion
55. Demo 55
56. Camel Reference Architecture and Patterns 56
57. Camel 4 Quickstarts 57 ▸ Message bridge between AMQ and IBM MQ with connection pooling and XA transactions ▸ ETL between multiple JDBC data sources ▸ JPA Idempotent repository ▸ MQTT and ElasticSearch ▸ Monitoring with Micrometer, Prometheus and Grafana ▸ AMQP and Salesforce ▸ and many more in development Quarkus Quickstarts Spring Boot Quickstarts
58. Red Hat Summit and AnsibleFest 2024 Modern Application Development Roadshow ▸ A single-day experience for you ▸ Tailored hands-on labs and learning ▸ Explore modern tooling, techniques, and architectures red.ht/MAD
59. Free Developer e-Books & tutorials! developers.redhat.com/eventtutorials
60. linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHat Thank you
61. linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHat Thank you