DevNation Stockholm - Getting started with Event Streaming using Apache Kafka and Camel

Transcript

1. 1 Event Streaming using Apache Kafka and Camel Kevin Dubois Sr Principal Developer Advocate Red Hat @kevindubois
2. @kevindubois Kevin Dubois ★ Sr Principal Developer Advocate at Red Hat ★ Based in Belgium 󰎐 ★ Java Champion ★ 🗣 Speak English, Dutch, French, Italian ★ Open Source Contributor (Quarkus, Camel, Knative, ..) youtube.com/@thekevindubois linkedin.com/in/kevindubois github.com/kdubois @kevindubois
3. Disparate Systems & Enterprise Integration
4. What problem needs to be resolved? Sometimes it’s a ‘MICRO-problem’, for instance ... • Sometimes the focus is on data/protocol transformation. • This is an integration problem. “I need this information in this specific format and protocol !” 4
5. some other times you have a ‘MACRO-problem’, for instance: • Here the focus is on interconnecting sources and destinations efficiently. ➔ This also... is an integration problem. “I want an easy and standard way to interconnect my app” 5
6. 6 “Good” Integration Bespoke/Custom made Integration Integration with reusable & standard components • As a quick win, often a problem is initially resolved in a custom manner. • In the long run this results to be counterproductive.
7. Apache Camel 7
8. 🛞Mainly, it’s about not reinventing the wheel 🛞 8
9. Apache Camel Swiss knife of integration Solve integration problem by applying best practices out of the box. Even with microservice architectures. Patterns 300+ Components Lightweight Runtimes Data Formats Packed with 300+ components such as databases, message queues, APIs. Quarkus, Standalone, Spring Boot,Application Servers, and natively on Cloud. Translate messages in multiple formats, and industry standard formats from finance, telco, health-care, and more 9
10. Simple, self-explained Domain Specific Language. Available in Java, YAML, XML DSLs Reactive Routing Engine Backlog Tracer/Tracer Comprehensive Tooling Back pressure model. Smooth flow control. Better thread management. Capturing a trace inside and between Camel route. For better observability Support autocomplete, correction on multiple IDE with LSP Server. Graphical data mapper. from(“kafka:topic”) .to(“grpc:endpoint”) 10 Apache Camel Swiss knife of integration
11. Camel Concepts and Constructs
12. Camel Route 12 connect a source endpoint to a destination endpoint Consumer receives message from the Source Endpoint Producer(s) sends message to the Target Endpoint Processors handle EIPs, routing, transformation, mediation, enrichment, validation, interception Camel route describes the step-by-step movement of a Message from a source endpoint, through arbitrary types of decision-making routines (such as filters and routers) to a destination endpoint (if any) 0, 1 or more Target endpoints 1 Source endpoint <camel> -- Camel context holds 1 to N routes <route> <from uri="timer:sec?period=1000"/> <setBody> <simple>Hello Camel from ${routeId}</simple> </setBody> <log message="${body}"/> </route> </camel> <XML DSL> A Camel DSL wires endpoints and processors together to form routes
13. Example: Process and route XML orders 13 Customer Purchase Split Orders Route each order to its service Electronics Others from("file:work/orders/input") .split(xpath("//orders")) .choice() .when(xpath("/order:order/order:type = 'E'")) .to("jms:queue:electronic/us") .otherwise() .to(“jms:queue:other/us”); XML Source endpoint Splitter EIP Target endpoint CBR EIP <Java DSL> Target endpoint
14. Camel Components 14 encapsulate APIs to enable it for routes. Endpoint URI component :resource[?options] Core direct :result timer :name?period=1000 file:directoryName[?options] bean:Class?method=myMethod 100+ extra (dynamically loaded) jms:queue:order kafka :myTopic sql:select * from orders where…[?options] smtp://mycompany.mailserver:30[?options]
15. Enterprise Integration Patterns 15 Apache Camel Implementations Splitter .split(<expression>) Filter .filter(<expression>) <step(s)> .end() -- optional Content-based Router .choice() .when(<expression>) <steps> .otherwise() <steps> Message Translator .setBody(<expression>) .transform(<expression>) .bean(<customBean>) .to() -- xslt, jslt, .. And more at https://camel.apache.org/components/4.4.x/eips/enterprise-integration-patterns.html
16. Conditional processing and data extraction 16 with Expression languages Simple language XPath JSONPath when routing, filtering and transforming for basic expressions and conditions for navigating and extracting parts of XML documents for navigating and extracting parts of JSON structures .choice().when(simple("${body} contains 'Camel'")) .filter(simple("${header.type'} == 'gold'")) .setBody(simple("The today is ${date:now:yyyyMMdd} and it is a great day.")) .choice().when(jsonpath("$.orderType == 'Online'")) .filter(jsonpath("$.isActive == true")) .setBody(jsonpath("$.customer.name")) .choice().when(xpath("/customer/type = 'Premium'")) .filter(xpath("/person[@name='James']")) .setBody(xpath("/customer/name/text()")) And more at https://camel.apache.org/components/4.4.x/languages/index.html
17. Data Format Transformations 17 Convert messages to and from various data formats (e.g., JSON, XML, CSV) to Java objects, facilitating the exchange of data between components that expect different formats Advanced Transformations (De)Serialization Perform complex data transformations that go beyond simple format conversions or basic manipulations. Data Type Converters Automatic or explicit conversion of message bodies and headers from one type to another. Built-in converters significantly reduce the need for custom conversion logic Convert message content from one format to another .mashal().jaxb() – XML to Pojo .unmarshal().jaxb() – Pojo to XML .convertBodyTo(String.class) .transform(<dataType>) XML➡XML – XSLT JSON➡JSON – JSLT XML⬅➡JSON – XSLT|XJ Custom processor
18. Custom processing using Beans 18 ▸ Extend routing logic with custom Java code ▸ For complex integrations, custom transformations and business logic implementations ▸ Binding annotations - body, exchange, header, variable, … ▸ Bind or lookup beans in Registry public class MyBean { public String process(String body) { return body.toUpperCase(); } public boolean isGoldCustomer(Exchange exchange) { // ... } } .bean(MyBean.class, "process") .bean(OrderService.class) -- calls a method annotated with @Handler .filter().method(MyBean.class, "isGoldCustomer") .bean("foo") -- call a bean from Registry
19. Make applications externally configurable 19 and define placeholders instead of the actual values db.host = 127.0.0.1 db.port = 8080 db.user = ibek db.pwd = 12345 file.path = file.json in properties file, environment variables, configmaps, secrets, etc. Placeholder functions - env, sys (jvm properties), bean, service Kubernetes placeholder functions - configmap, secret .to("file:foo?fileName={{file.path:/some/path}}") -- with default value .to("file:foo?bufferSize={{?myBufferSize}}") -- optional .log(“What {{configmap:myconfig/drink}} do you want?”) .from("file:{{bean:foo.bar}}") .to("{{env:HOME}}") .to("{{sys:file.separator}}")
20. 20 ▸ World's most popular java framework ▸ Focus on speed, simplicity and productivity ▸ For stand-alone Spring applications ▸ Opinionated with simplified build configuration ▸ Many out of the box solutions Camel for Spring Boot ▸ Latest Camel v4 (lighter, faster) ▸ Container-based environments ▸ Camel context with automatic detection of Camel routes ▸ Auto-configuration of Camel components ▸ Starters for many Camel pre-built components ▸ Highly advanced integrations ▸ Custom flows with full control ▸ Long-term support commitment Spring Boot ▸ Integration swiss knife ▸ Super lightweight ▸ Connectors ▸ Most popular Integration Framework Camel Camel for Spring Boot APPLICATION FOUNDATIONS
21. 21 ▸ Framework addressing wide range of distributed application architectures ▸ Containers First ▸ Serverless characteristics ▸ Unified configuration ▸ Dev mode with live reload ▸ Kubernetes-native ▸ Hundreds of best-of-breed standard libraries Camel for Quarkus ▸ Latest Camel v4 (lighter, faster) ▸ Container-based environments ▸ Flash boot time ▸ Minimal memory footprint ▸ Standalone & OpenShift ▸ Highly advanced integrations ▸ Custom flows with full control ▸ Long-term support commitment Quarkus ▸ Integration swiss knife ▸ Super lightweight ▸ Connectors ▸ Most popular Integration Framework Camel Camel for Quarkus APPLICATION FOUNDATIONS
22. Supersonic Subatomic Java Quarkus + Native (via GraalVM) 12 MB Quarkus + JVM (via OpenJDK) 73 MB Traditional Cloud-Native Stack 136 MB Quarkus + Native (via GraalVM) 0.016 Seconds Quarkus + JVM (via OpenJDK) 0.943 Seconds Traditional Cloud-Native Stack 4.3 Seconds 22
23. 23 A cohesive platform for optimized developer joy: ▸ Zero config, live reload in the blink of an eye ▸ Based on standards, but not limited ▸ Unified configuration ▸ Streamlined code for the 80% common usages, flexible for the 20% ▸ No hassle native executable generation Developer Joy
24. Red Hat build of Apache Camel Connect data streams between disparate systems using Camel as Kafka source and sink connectors Versatile application development toolkit for Enterprise integration that simplifies and standardizes the practices of connecting diverse systems. Build flexible streaming data pipelines interconnected with Camel integrations to decouple the systems Optimize data integrity and processing efficiency with streaming data integration patterns based on Apache Camel for Streaming Data Integration
25. Events EVENT-DRIVEN ARCHITECTURE 25
26. Why Event-Driven Architecture? EVENT-DRIVEN ARCHITECTURE 26 Mirrors the real world The real world is event-driven. Systems generate and respond to events in everyday life, e.g., the human central nervous system. Reduced coupling Traditional RPC-style service architecture results in tightly-bound services. Changes to the application flow typically require service code changes. EDA allows new functionality to be added by adding services that consume existing event streams. Encapsulation Microservices concepts have grown in popularity due to the ability for service teams to develop services in isolation. EDA means that service designers need not be aware of how events are consumed. Fine-grained scaling Services can be independently scaled up and down to meet the event volume. Near real-time latency Customers increasing expect a near real-time experience. Polling on APIs is a delicate trade-off between responsiveness and load. EDA allow apps to react in near real-time without compromise.
27. EVENT-DRIVEN ARCHITECTURE What is an event? 27 Async form of Remote Procedure Call, contains instructions telling recipient what to do, may cause a change of state. Query Command Similar to commands, queries expect a response returning the results, but does not cause any change in state. Event Immutable state and value of a particular entity, which occurred during operation among services.
28. EVENT-DRIVEN ARCHITECTURE Types of events 28 Events stored durably until read by all registered consumers. Traditional store-and-forward brokers. Replayable Consumable Events stored durably for specific period of time or storage capacity. Consumers can move back and forth of the stream. Volatile The event needs to be disseminated to all consumers online at time of publication. Not persisted.
29. EVENT-DRIVEN ARCHITECTURE What is Apache Kafka? 29 Apache Kafka is a distributed system designed for streams. It is built to be an horizontally-scalable, fault-tolerant, commit log, and allows distributed data streams and stream processing applications.
30. Why Kafka? MyPlane Insert Update MyPlane [Insert] MyOtherPlane [Insert] MyPlane [Update]
31. EVENT-DRIVEN ARCHITECTURE Running on OpenShift/Kubernetes 31 Source: • Strimzi provides: ◦ Container images for Apache Kafka, Connect, Zookeeper and MirrorMaker ◦ Kubernetes Operators for managing/configuring Apache Kafka clusters, topics and users ◦ Kafka Consumer, Producer and Admin clients, Kafka Streams ◦ 100% Open source project licensed under Apache License 2.0 ◦ Part of the Cloud Native Computing Foundation (CNCF)
32. Connecting data streams between disparate systems 32
33. Connect data streams between disparate systems Camel as Kafka source and sink connectors source source kamelet source pipe Kafka cluster channel sink pipe sink kamelet sink action kamelets action kamelets 2. Kamelets are reusable Camel route templates 3. Action kamelets validate, cleanse, and transform data File system 1. Stream data out and into various systems and services Database API and more with over 100 components IBM MQ Custom
34. Build flexible streaming data pipelines interconnected with Camel integrations to decouple the systems Data stream processing Raw data Processed data Data ingestion Routing to data storage (search index, analytics cluster, …) {;} Protobuf Json / Avro Schema Registry (Apicurio, Azure, …) Validation sources sources Avro deserialization # consumer group [message-offset] destinations SASL-SSL
35. Optimize data integrity and processing efficiency with streaming data integration patterns Data stream processing Raw data Processed data Data ingestion Routing to data storage Error handling - Dead letter pattern - Message redelivery sources sources Exactly-once Filtering Batch processing Aggregation Data transformation destinations Idempotent filtering Content-based routing Throttler Enricher Splitter (search index, analytics cluster, …)
36. Demo
37. Demo
39. rest("/getresults") .get() .bean("vote", "orderedList()") .marshal().json() .convertBodyTo(String.class);
42. rest("/favstack") .post() .consumes("application/json") .produces("application/json") .to("kafka:{{kafka.topic.name}}");
45. from("kafka:{{kafka.topic.name}}") .unmarshal().json(JsonLibrary.Jackson, Response.class) .transacted() .to("bean:vote?method=updateCounter(${body.getShortname})");
46. Quarkus Dev UI
47. mvn clean package -> also generates kubernetes/knative manifests when you add the quarkus-kubernetes extension mvn clean package -Dnative -Dquarkus.kubernetes.deploy -> deploys native binary directly to kubernetes/openshift
49. CAMEL DEV TOOLS 49 for simple creation and testing of Camel routes ▸ VS Code extensions ･ Language Support for Apache Camel by Red Hat ･ Debug Adapter for Apache Camel by Red Hat ･ Kaoto - Lowcode visual designer for Camel integrations ▸ Camel CLI for rapid integration prototyping ▸ Camel REST DSL OpenApi Maven Plugin generates REST DSL source code from OpenAPI specification ▸ CXF SOAP Development Tools Java to WSDL, WSDL to Java, WSDL Validator ▸ Development support
50. Rapid Integration Prototyping 50 using Camel CLI ▸ Immediately start with writing integration code ▸ Automatic dependency resolution for all supported components and EIPs ▸ Export functionality to a full Camel java-based project when done with prototyping ▸ Enables running integration designs from UI tools such as Kaoto Iterate and experiment hot reload Log & watch Debug Manage Monitor camel export --runtime=quarkus --gav=com.foo:acme:1.0-SNAPSHOT create commit CI/CD camel init routesA.xml camel run * --dev
51. HawtIO diagnostic console 51 for troubleshooting and debugging Camel routes ▸ React + PatternFly v4 ▸ New plugin mechanism ▸ Standalone -> Hawtio v4 ▸ OpenShift -> Hawtio Online v2 ▸ Camel plugin ･ Contexts/Routes/Endpoints ･ Route diagram ･ Trace/Debug ･ Source ･ Simple statistics ▸ Other plugins - Connect, JMX, OAuth (Keycloak), Artemis ▸ Quarkus (JVM only) and SpringBoot
52. Kaoto in Visual Studio Code 52 Visual Designer for Camel ▸ Run with Camel CLI ▸ Debug with Debug Adapter ▸ Language support for syntax highlighting and autocompletion
53. Camel 4 Quickstarts 53 ▸ Message bridge between AMQ and IBM MQ with connection pooling and XA transactions ▸ ETL between multiple JDBC data sources ▸ JPA Idempotent repository ▸ MQTT and ElasticSearch ▸ Monitoring with Micrometer, Prometheus and Grafana ▸ AMQP and Salesforce ▸ and many more in development Quarkus Quickstarts Spring Boot Quickstarts
54. Free Developer e-Books & tutorials! developers.redhat.com/eventtutorials
55. redhat.com/en/products/application-foundations
56. Thank You youtube.com/@thekevindubois linkedin.com/in/kevindubois github.com/kdubois @kevindubois.com @kevindubois@mastodon.social Slides