A Story about Serverless Camels Living in Containers

When you build an application you will undoubtedly need to integrate it with other applications and systems, and transform data from one format to another. The Apache Camel project’s goal is to simplify the way we do these kinds of integration, using the book “Enterprise Integration Patterns” as the gold standard.

Apache Camel has been around for a while, but it is more relevant now than it has ever been. In this session we’ll show you how Camel can leverage Quarkus’ fast startup time and developer experience to build modern, distributed and cloud native integration workloads. As a bonus it can make use of technologies like serverless computing (eg. Knative) and data streaming (eg. Kafka) to remove bottlenecks, integrate faster and better than ever before, and save resources and money on top of it.

Note: Camels were not hurt in the process of making this talk.

Attendees should come away from this talk with the following:

- Why you would want to use Apache Camel to create easy to understand, reusable and distributed integration components.
- How the next iteration of Apache Camel (based on Quarkus) can leverage Kubernetes and/or Serverless computing and event driven architectures to deploy early and often.

Transcript

1. @kevindubois Serverless Integration with Camel Quarkus Kevin Dubois Principal Developer Advocate, Red Hat Zineb Bendhiba Senior Software Engineer, Red Hat
2. @kevindubois Serverless Enterprise Integration Apache Camel Quarkus Kubernetes
3. @kevindubois Kevin Dubois • Principal Developer Advocate at Red Hat • Previously: (Lead) Software Engineer / Architect, mostly in USA • Speak English, Dutch, French, Italian • Passionate about improving Dev Experience with Open Source Linkedin: kevindubois Github: kdubois @kevindubois@mastodon.social
4. @kevindubois Zineb Bendhiba • Senior Software Engineer at Red Hat • Apache Camel Committer and PMC Linkedin: zbendhiba Twitter: @ZinebBendhiba Mastodon: @zbendhiba@mastodon.social Github: zbendhiba
5. @kevindubois Enterprise Integration
6. @kevindubois What problem needs to be resolved? Sometimes it’s a ‘MICRO-problem’, for instance ... • Sometimes the focus is on data/protocol transformation. • This is an integration problem. “I need this information in this specific format and protocol !”
7. @kevindubois … and, some other times you have a ‘MACRO-problem’, for instance: • Here the focus is on interconnecting sources and destinations efficiently. ➔ This also... is an integration problem. “I want an easy and standard way to interconnect my app”
8. @kevindubois 8 “Good” Integration Bespoke/Custom made Integration Integration with reusable & standard components • As a quick win, often a problem is initially resolved in a custom manner. • In the long run this results to be counterproductive.
9. @kevindubois 🛞Mainly, it’s about not reinventing the wheel 🛞
10. @kevindubois
11. @kevindubois Apache Camel
12. @kevindubois Apache Camel Swiss knife of integration Solve integration problem by applying best practices out of the box. Even with microservice architectures. Patterns 300+ Components Lightweight Runtimes Data Formats Packed with 300+ components such as databases, message queues, APIs. Quarkus, Standalone, Spring Boot,Application Servers, and natively on Cloud. Translate messages in multiple formats, and industry standard formats from finance, telco, health-care, and more
13. @kevindubois Camel “Routes” Source Sink Action
14. @kevindubois https://camel.apache.org/components
15. @kevindubois Apache Camel Swiss knife of integration Simple, self-explained Domain Specific Language. Available in Java, YAML, XML DSLs Reactive Routing Engine Backlog Tracer/Tracer Comprehensive Tooling Back pressure model. Smooth flow control. Better thread management. Capturing a trace inside and between Camel route. For better observability Support autocomplete, correction on multiple IDE with LSP Server. Graphical data mapper. from(“kafka:topic”) .to(“grpc:endpoint”)
16. @kevindubois Apache Camel Community Source: https://www.apache.org/foundation/docs/FY2021AnnualReport.pdf ➢ The biggest and most active community for open source integration software ➢ 600+ unique contributors ➢ 3800+ pull requests (mostly from external contributors) ➢ 15+ years of development and one of the most active Apache projects
17. @kevindubois Serverless
18. @kevindubois
19. @kevindubois Serverless “Serverless computing refers to the concept of building and running applications that do not require server management. It describes a finer-grained deployment model where applications, bundled as one or more functions are uploaded to a platform and then executed, scaled, and billed in response to the exact demand needed at the moment” -- CNCF Definition, https://www.cncf.io/blog/2018/02/14/cncf-takes-first-step-towards-serverless-computing/
20. @kevindubois Immutable revisions Deploy new features: performing canary, A/B or blue-green testing with gradual traffic rollout with no sweat and following best practices. No need to configure number of replicas, or idling. Scale to zero when not in use, auto scale to thousands during peak, with built-in reliability and fault-tolerance. Automatic scaling Ready for the Hybrid Cloud Truly portable serverless running anywhere Kubernetes runs, that is on-premises or on any public cloud. Leverage data locality and SaaS when needed. Event Driven Architectures Build loosely coupled & distributed apps connecting with a variety of built-in or third-party event sources or connectors powered by Operators. Any programming language Use any programming language or runtime of choice. From Java, Python, Go and JavaScript to Quarkus, SpringBoot or Node.js. Simplified developer experience to deploy applications/code on serverless containers abstracting infrastructure & focusing on what matters. Containers made easy prem aws azur e Knative is an Open Source, Cloud Agnostic Solution to build Serverless and Event Driven Applications Knative
21. @kevindubois Serverless Operational Benefits Over provisioning Time in capacity planning IT cost of idle resources Under provisioning Lost business revenue Poor quality of service More applications Direct line between IT costs & business revenue NOT Serverless with Serverless
22. @kevindubois “Traditional” Java wasn’t designed for Containers & Serverless At the expense of startup speed Rich dynamic behavior built for mutable systems Designed to be long-running Yet containers are primarily immutable Designed for Throughput At the expense of footprint ➔ Java likes to uses as many resources as it can get
23. @kevindubois When you can’t scale fast enough…
24. @kevindubois Serverless Adoption 6.1% Java https://serverless.com/blog/2018-serverless-community-survey-huge-growth-usage/ 62.9% Node.js 20.8% Python 6.4% Go 6.1% Java 3.8% C#
25. @kevindubois What about our existing Java skills & Expertise?
26. @kevindubois 26 Supersonic. Subatomic. Java.
27. @kevindubois Quarkus is a Java Stack that : ★ Is based on Java standards ★ Moves as much as possible to build phase ★ Minimizes runtime dependencies ★ Maximizes dead code elimination ★ Enables Native Build with GraalVM / Mandrel ◦ (without having to know how to work with GraalVM!) ★ Brings Developer joy !
28. @kevindubois Supersonic Subatomic Java Quarkus + Native (via GraalVM) 12 MB Quarkus + JVM (via OpenJDK) 73 MB Traditional Cloud-Native Stack 136 MB Quarkus + Native (via GraalVM) 0.016 Seconds Quarkus + JVM (via OpenJDK) 0.943 Seconds Traditional Cloud-Native Stack 4.3 Seconds
29. @kevindubois A cohesive platform for optimized developer joy: • Quick code-build-verify turnaround ◦ Live reload in the blink of an eye ◦ Continuous Testing • No hassle native executable generation • Many developer-friendly features: Dev services; Hibernate Panache; simplified logging; IDE plugins; Kubernetes & Openshift Extensions, etc Quarkus brings Developer Joy
30. @kevindubois Apache Camel extensions for Quarkus (or just “Camel Quarkus”)
31. @kevindubois code.quarkus.io Camel extensions for Quarkus
32. @kevindubois Demo
33. @kevindubois Demo
34. @kevindubois
35. @kevindubois rest("/getresults") .get() .bean("vote", "orderedList()") .marshal().json() .convertBodyTo(String.class);
36. @kevindubois
37. @kevindubois
38. @kevindubois rest("/favstack") .post() .consumes("application/json") .produces("application/json") .to("kafka:{{kafka.topic.name}}" );
39. @kevindubois
40. @kevindubois
41. @kevindubois from("kafka:{{kafka.topic.name}}" ) .unmarshal().json(JsonLibrary.Jackson, Response.class) .transacted() .to("bean:vote?method=updateCounter(${body.getShortname})" );
42. @kevindubois Quarkus Dev UI
43. @kevindubois @ZinebBendhiba @kevindubois #Camel #Quarkus mvn clean package -> also generates kubernetes/knative manifests when you add the quarkus-kubernetes extension mvn clean package -Dnative -Dquarkus.kubernetes.deploy -> deploys native binary directly to kubernetes/openshift
44. @kevindubois @ZinebBendhiba @kevindubois #Camel #Quarkus
45. @kevindubois
46. @kevindubois “@kevindubois my favorite IDE/editor is xyz”
47. @kevindubois fromF("twitter-search:// %s?", searchTerm) .process(new TweetInfoProcessor ()) .choice() .when(simple("${body} ~~ openshiftdevspaces" )) .setBody(simple("{\"shortname\":\"openshiftdevspaces \"}") ) .to("direct:sendToKafka" ) (..) .otherwise().log("no match"); from("direct:sendToKafka" ) .to("kafka:{{kafka.topic.name}}" );
48. @kevindubois
49. @kevindubois
50. @kevindubois • A platform directly running integrations on Openshift and Kubernetes with Serverless-ready capabilities • Architectured by Kubernetes CRDs and Operators • A community-driven project • Part of Apache Camel. Started on August 31st, 2018 What is Camel K? K 50
51. @kevindubois Camel K 51 What does the Camel K platform has to offer ? Cloud Native EIP Orchestration Auto adapt to the environment (Knative, Kafka, OpenShift, Kubernetes, Service Binding, etc) Seamless Integration Connect anything - as it leverages the Apache Camel set of components - the Cloud Native way Connectivity Auto scaling (including scaling to 0) sources and sinks for event-driven serverless applications Serverless Integration Quarkus based runtime Efficiency Live Reload, Projectless Developer Joy
52. @kevindubois 52 Quarkus + K = + Camel K Camel Operator Powered by Next Generation Building Blocks Camel K
53. @kevindubois 1 source code file > kamel run roll-die.yaml web service roll-die.yam l • 1 CLI command Create integration file Running on OpenShift 1 Execute CLI Tools 2 3 Camel Ｋ For Developers - from: uri: "platform-http:/roll-die " steps: - setBody: simple: "roll: ${random( 1,6)}”
54. @kevindubois Analyze Integration Custom Resource Observe Act Serverless O↔N Camel K Operator Camel Developer > kamel run roll-die.js 54
55. @kevindubois Wrap up ★ Apache Camel = an integration toolkit with 350+ connectors ★ Camel Quarkus = developer joy + fast startup + easy integration ★ Scale from 0 and easy deployments with Knative ★ Also cool: Low/No Code with Camel K and Kamelets
56. @kevindubois @ZinebBendhiba @kevindubois #Camel #Quarkus github.com/kdubois/CamelQuarkusVoter
57. @kevindubois Thank you!