A Story about Serverless Camels Living in Containers

When you build an application you will undoubtedly need to integrate it with other applications and systems, and transform data from one format to another. The Apache Camel project’s goal is to simplify the way we do these kinds of integration, using the book “Enterprise Integration Patterns” as the gold standard.

Apache Camel has been around for a while, but it is more relevant now than it has ever been. In this session we’ll show you how Camel can leverage Quarkus’ fast startup time and developer experience to build modern, distributed and cloud native integration workloads.
As a bonus it can make use of technologies like serverless computing (eg. Knative) and data streaming (eg. Kafka) to remove bottlenecks, integrate faster and better than ever before, and save resources and money on top of it.

Note: Camels were not hurt in the process of making this talk.

Attendees should come away from this talk with the following:

Why you would want to use Apache Camel to create easy to understand, reusable and distributed integration components.
How the next iteration of Apache Camel (based on Quarkus) can leverage Kubernetes and/or Serverless computing and event driven architectures to deploy early and often.

Transcript

1. A Story about Serverless Camels Living in Containers Zineb Bendhiba, Senior Software Engineer, Red Hat Kevin Dubois, Principal Developer Advocate, Red Hat
2. Serverless Enterprise Integration Apache Camel Quarkus Kubernetes
3. Kevin Dubois ❖ Principal Developer Advocate at Red Hat ❖ International Speaker ❖ 18+ years professional software development experience ❖ Speak English, Dutch, French, Italian ❖ Open Source Contributor (Quarkus, Camel, Knative, ..) ❖ Open Source Community Member ➢ BeJUG, BEL CNCF chapter @kevindubois@mastodon.social youtube.com/@thekevindubois linkedin.com/in/kevindubois github.com/kdubois @kevindubois.com
4. Zineb Bendhiba • Senior Software Engineer at Red Hat • Apache Camel Committer and PMC • International Speaker • 15+ years professional software development experience • Speak English, French, Arabic, Moroccan Darija Linkedin: zbendhiba Twitter: @ZinebBendhiba Mastodon: @zbendhiba@mastodon.social Github: zbendhiba
5. Enterprise Integration
6. What problem needs to be resolved? Sometimes it’s a ‘MICRO-problem’, for instance ... • Sometimes the focus is on data/protocol transformation. • This is an integration problem. “I need this information in this specific format and protocol !”
7. … and, some other times you have a ‘MACRO-problem’, for instance: • Here the focus is on interconnecting sources and destinations efficiently. ➔ This also... is an integration problem. “I want an easy and standard way to interconnect my app”
8. 8 “Good” Integration Bespoke/Custom made Integration Integration with reusable & standard components • As a quick win, often a problem is initially resolved in a custom manner. • In the long run this results to be counterproductive.
9. 🛞Mainly, it’s about not reinventing the wheel 🛞
11. Apache Camel
12. Apache Camel Swiss knife of integration Solve integration problem by applying best practices out of the box. Even with microservice architectures. Patterns 300+ Components Lightweight Runtimes Data Formats Packed with 300+ components such as databases, message queues, APIs. Quarkus, Standalone, Spring Boot,Application Servers, and natively on Cloud. Translate messages in multiple formats, and industry standard formats from finance, telco, health-care, and more
13. Camel “Routes” Source Sink Action
14. https://camel.apache.org/components
15. Apache Camel Swiss knife of integration Simple, self-explained Domain Specific Language. Available in Java, YAML, XML DSLs Reactive Routing Engine Backlog Tracer/Tracer Comprehensive Tooling Back pressure model. Smooth flow control. Better thread management. Capturing a trace inside and between Camel route. For better observability Support autocomplete, correction on multiple IDE with LSP Server. Graphical data mapper. from(“kafka:topic”) .to(“grpc:endpoint”)
16. Apache Camel Community Source: https://www.apache.org/foundation/docs/FY2021AnnualReport.pdf ➢ The biggest and most active community for open source integration software ➢ 600+ unique contributors ➢ 3800+ pull requests (mostly from external contributors) ➢ 15+ years of development and one of the most active Apache projects
17. Serverless
19. Serverless “Serverless computing refers to the concept of building and running applications that do not require server management. It describes a finer-grained deployment model where applications, bundled as one or more functions are uploaded to a platform and then executed, scaled, and billed in response to the exact demand needed at the moment” -- CNCF Definition, https://www.cncf.io/blog/2018/02/14/cncf-takes-first-step-towards-serverless-computing/
20. Immutable revisions Deploy new features: performing canary, A/B or blue-green testing with gradual traffic rollout with no sweat and following best practices. No need to configure number of replicas, or idling. Scale to zero when not in use, auto scale to thousands during peak, with built-in reliability and fault-tolerance. Automatic scaling Ready for the Hybrid Cloud Truly portable serverless running anywhere Kubernetes runs, that is on-premises or on any public cloud. Leverage data locality and SaaS when needed. Event Driven Architectures Build loosely coupled & distributed apps connecting with a variety of built-in or third-party event sources or connectors powered by Operators. Any programming language Use any programming language or runtime of choice. From Java, Python, Go and JavaScript to Quarkus, SpringBoot or Node.js. Simplified developer experience to deploy applications/code on serverless containers abstracting infrastructure & focusing on what matters. Containers made easy prem aws azur e Knative is an Open Source, Cloud Agnostic Solution to build Serverless and Event Driven Applications Knative
21. Eventing - Key Terminology • Event • Source / Sink • Propagate an event from an event source to multiple event sinks by using: ◦ Broker ◦ Channell • Trigger and Filter (Broker) • Subscription (Channel) • CloudEvent (CNCF) Technical Overview decouple relationships between event producers and event consumers Application (Knative Service) Event Sink Event Source kn source kn source create kn source list kn source list-types Cloud Event
22. Serverless Operational Benefits Over provisioning Time in capacity planning IT cost of idle resources Under provisioning Lost business revenue Poor quality of service More applications Direct line between IT costs & business revenue NOT Serverless with Serverless
23. “Traditional” Java wasn’t designed for Containers & Serverless At the expense of startup speed Rich dynamic behavior built for mutable systems Designed to be long-running Yet containers are primarily immutable Designed for Throughput At the expense of footprint ➔ Java likes to uses as many resources as it can get
24. When you can’t scale fast enough…
25. Serverless Adoption 6.1% Java https://serverless.com/blog/2018-serverless-community-survey-huge-growth-usage/ 62.9% Node.js 20.8% Python 6.4% Go 6.1% Java 3.8% C#
26. What about our existing Java skills & Expertise?
27. 27 Supersonic. Subatomic. Java.
28. Quarkus is a Java Stack that : ★ Is based on Java standards ★ Moves as much as possible to build phase ★ Minimizes runtime dependencies ★ Maximizes dead code elimination ★ Enables Native Build with GraalVM / Mandrel ◦ (without having to know how to work with GraalVM!) ★ Brings Developer joy !
29. Supersonic Subatomic Java Quarkus + Native (via GraalVM) 12 MB Quarkus + JVM (via OpenJDK) 73 MB Traditional Cloud-Native Stack 136 MB Quarkus + Native (via GraalVM) 0.016 Seconds Quarkus + JVM (via OpenJDK) 0.943 Seconds Traditional Cloud-Native Stack 4.3 Seconds
30. A cohesive platform for optimized developer joy: • Quick code-build-verify turnaround ◦ Live reload in the blink of an eye ◦ Continuous Testing • No hassle native executable generation • Many developer-friendly features: Dev services; Hibernate Panache; simplified logging; IDE plugins; Kubernetes & Openshift Extensions, etc Quarkus brings Developer Joy
31. Apache Camel extensions for Quarkus (or just “Camel Quarkus”)
32. code.quarkus.io Camel extensions for Quarkus
33. Demo
34. Demo
36. rest("/getresults") .get() .bean("vote", "orderedList()") .marshal().json() .convertBodyTo(String.class);
39. rest("/favstack") .post() .consumes("application/json") .produces("application/json") .to("kafka:{{kafka.topic.name}}" );
42. from("kafka:{{kafka.topic.name}}" ) .unmarshal().json(JsonLibrary.Jackson, Response.class) .transacted() .to("bean:vote?method=updateCounter(${body.getShortname})" );
43. Quarkus Dev UI
44. @ZinebBendhiba @kevindubois #Camel #Quarkus mvn clean package -> also generates kubernetes/knative manifests when you add the quarkus-kubernetes extension mvn clean package -Dnative -Dquarkus.kubernetes.deploy -> deploys native binary directly to kubernetes/openshift
45. @ZinebBendhiba @kevindubois #Camel #Quarkus
46. • A platform directly running integrations on Openshift and Kubernetes with Serverless-ready capabilities • Architectured by Kubernetes CRDs and Operators • A community-driven project • Part of Apache Camel. Started on August 31st, 2018 What is Camel K? K
47. Camel K 47 What does the Camel K platform has to offer ? Cloud Native EIP Orchestration Auto adapt to the environment (Knative, Kafka, OpenShift, Kubernetes, Service Binding, etc) Seamless Integration Connect anything - as it leverages the Apache Camel set of components - the Cloud Native way Connectivity Auto scaling (including scaling to 0) sources and sinks for event-driven serverless applications Serverless Integration Quarkus based runtime Efficiency Live Reload, Projectless Developer Joy
48. 48 Quarkus + K = + Camel K Camel Operator Powered by Next Generation Building Blocks Camel K
49. 1 source code file > kamel run roll-die.yaml web service roll-die.yam l • 1 CLI command Create integration file Running on OpenShift 1 Execute CLI Tools 2 3 Camel Ｋ For Developers - from: uri: "platform-http:/roll-die" steps: - setBody: simple: "roll: ${random(1,6)}”
50. Analyze Integration Custom Resource Observe Act Serverless O↔N Camel K Operator Camel Developer > kamel run roll-die.js
51. Wrap up ★ Apache Camel = an integration toolkit with 350+ connectors ★ Camel Quarkus = developer joy + fast startup + easy integration ★ Scale from 0 and easy deployments with Knative ★ Also cool: Low/No Code with Camel K and Kamelets
52. @ZinebBendhiba @kevindubois #Camel #Quarkus Links
53. @ZinebBendhiba @kevindubois #Camel #Quarkus Feedback
54. Free Developer e-Books! developers.redhat.com/e-books
55. Start exploring in the OpenShift Sandbox. Learn containers, Kubernetes, and OpenShift in your browser. developers.redhat.com/developer-sandbox Try Red Hat's products and technologies without setup or configuration.
56. Thank you! @ZinebBendhiba @kevindubois