The Lambda Sidecar Pattern for Event-Driven Kubernetes

In this session we are going to share a top to bottom (architecture → tooling) story on how we at Custom Ink are using real cloud-native & serverless solutions for our Kubernetes workloads during our migration from AMQP to EventBridge. Like many teams, we are slowly marching to a more event-driven architecture (EDA) and we cant wait to share how Lambda, SQS, & EventBridge are a key part of our journey. In our time together, we want to explore the following topics.
A product review on the current state of open-soruce solutions for containerized event handling.

Transcript

1. The Lambda Sidecar
   Pattern for Event-Driven
   Kubernetes (YMMV)
   2022-12-07 by Ken Collins
   

   View Slide

2. Our Need to Change?
   

   View Slide

3. 🐰
   What is AMQP?
   AKA RabbitMQ, how is it used at Custom Ink?
   

   View Slide

4. 🐰 AMQP@CustomInk -> Topics
   ♻
   ⏳
   ♻
   ⏳
   polling process
   polling process
   

   View Slide

5. ● Service owns the data being published.
   ● Other service(s) interested in data create/updates.
   ● Service does not depend on receipt of data/event.
   Our AMQP Patterns 🪢
   

   View Slide

6. ● Custom in-house software 11 years old. No active
   development in 4 years for supporting software like
   AMQP & EventMachine.
   ● Infrastructure is not managed nor multi-region.
   Application integrations are not cloud-native.
   ● EDA is part of our future-ready platform’s composability,
   SaaS integration, & real-time data needs.
   Our Need to Change?
   

   View Slide

7. Cloud-Native EDA w/Kubernetes ⁉
   ● Containerization has always promised us future
   opportunities. Can we start cashing in on that now?
   ● Can this be done via simple & viable means? Are the
   integration capabilities compelling?
   

   View Slide

8. Cloud-Native EDA w/Kubernetes ⁉
   

   View Slide

9. Cloud-Native EDA w/Kubernetes ⁉
   

   View Slide

10. Deep Dive into
    this Architecture
    🤿
    Down the C4 Model
    

    View Slide

11. ⛵
    Our Custom
    Kubernetes Tooling
    

    View Slide

12. GitOps Tooling Flow (Before)
    

    View Slide

13. GitOps Tooling Flow (Before)
    

    View Slide

14. Kubernetes
    for each Project
    🦈
    

    View Slide

15. Typical Kubernetes Project
    

    View Slide

16. Typical Kubernetes Project
    

    View Slide

17. With The “Lambda Sidecar” Pattern
    

    View Slide

18. Kubernetes
    AWS Resources
    🪸
    

    View Slide

19. For Each EventBridge Rule (simple)
    

    View Slide

20. Tooling EventBridge Interface
    

    View Slide

21. GitOps Tooling Flow (Before)
    

    View Slide

22. GitOps Tooling Flow (After)
    

    View Slide

23. AWS::Lambda::Function ImageConfig
    https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-lambda-function-imageconfig.html
    

    View Slide

24. Runtime Interface Client (RIC)
    https://docs.aws.amazon.com/lambda/latest/dg/runtimes-api.html
    

    View Slide

25. GitOps CloudFormation Snippet
    

    View Slide

26. GitOps Tooling Flow (After)
    

    View Slide

27. Diving Deeper
    

    View Slide

28. Secret Environment Variables?
    https://external-secrets.io/v0.7.0-rc1/
    

    View Slide

29. Hello Crypteia & LD_PRELOAD
    https://github.com/customink/crypteia
    

    View Slide

30. For Each EventBridge Rule (simple)
    

    View Slide

31. For Each EventBridge Rule (full)
    

    View Slide

32. Multiple EventBridge Rules (full)
    

    View Slide

33. Typical Kubernetes Project (Before)
    

    View Slide

34. Typical Kubernetes Project (After)
    

    View Slide

35. Questions &
    Answers
    ⛈
    

    View Slide