The Lambda Sidecar Pattern for Event-Driven Kubernetes

In this session we are going to share a top to bottom (architecture → tooling) story on how we at Custom Ink are using real cloud-native & serverless solutions for our Kubernetes workloads during our migration from AMQP to EventBridge. Like many teams, we are slowly marching to a more event-driven architecture (EDA) and we cant wait to share how Lambda, SQS, & EventBridge are a key part of our journey. In our time together, we want to explore the following topics.
A product review on the current state of open-soruce solutions for containerized event handling.

Transcript

1. The Lambda Sidecar Pattern for Event-Driven Kubernetes (YMMV) 2022-12-07 by Ken Collins
2. Our Need to Change?
3. 🐰 What is AMQP? AKA RabbitMQ, how is it used at Custom Ink?
4. 🐰 AMQP@CustomInk -> Topics ♻ ⏳ ♻ ⏳ polling process polling process
5. • Service owns the data being published. • Other service(s) interested in data create/updates. • Service does not depend on receipt of data/event. Our AMQP Patterns 🪢
6. • Custom in-house software 11 years old. No active development in 4 years for supporting software like AMQP & EventMachine. • Infrastructure is not managed nor multi-region. Application integrations are not cloud-native. • EDA is part of our future-ready platform’s composability, SaaS integration, & real-time data needs. Our Need to Change?
7. Cloud-Native EDA w/Kubernetes ⁉ • Containerization has always promised us future opportunities. Can we start cashing in on that now? • Can this be done via simple & viable means? Are the integration capabilities compelling?
8. Cloud-Native EDA w/Kubernetes ⁉
9. Cloud-Native EDA w/Kubernetes ⁉
10. Deep Dive into this Architecture 🤿 Down the C4 Model
11. ⛵ Our Custom Kubernetes Tooling
12. GitOps Tooling Flow (Before)
13. GitOps Tooling Flow (Before)
14. Kubernetes for each Project 🦈
15. Typical Kubernetes Project
16. Typical Kubernetes Project
17. With The “Lambda Sidecar” Pattern
18. Kubernetes AWS Resources 🪸
19. For Each EventBridge Rule (simple)
20. Tooling EventBridge Interface
21. GitOps Tooling Flow (Before)
22. GitOps Tooling Flow (After)
23. AWS::Lambda::Function ImageConfig https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-lambda-function-imageconfig.html
24. Runtime Interface Client (RIC) https://docs.aws.amazon.com/lambda/latest/dg/runtimes-api.html
25. GitOps CloudFormation Snippet
26. GitOps Tooling Flow (After)
27. Diving Deeper
28. Secret Environment Variables? https://external-secrets.io/v0.7.0-rc1/
29. Hello Crypteia & LD_PRELOAD https://github.com/customink/crypteia
30. For Each EventBridge Rule (simple)
31. For Each EventBridge Rule (full)
32. Multiple EventBridge Rules (full)
33. Typical Kubernetes Project (Before)
34. Typical Kubernetes Project (After)
35. Questions & Answers ⛈