Building on System Resilience with Chaos Engineering for Serverless Applications on AWS

Slide deck for Chaos Carnival 2021 on 10 February, 2021

Transcript

1. BUILDING ON SYSTEM RESILIENCE WITH CHAOS ENGINEERING FOR SERVERLESS APPLICATIONS ON AWS RENALDI GONDOSUBROTO
2. THE AGENDA 1. Chaos on the Cloud 2. Endeavours in the Cloud 3. The Architecture 4. Applying Chaos 5. What we Got Out of it
3. A Bit About Myself • 12x AWS, 3x Azure Certified and 2x Google Cloud Certified • Personal field of interest is in penetration testing and accessibility practices • On the side: Running meetups, hackathons, doing tech talks and VR tech enthusiast Renaldi Gondosubroto Founder and Developer Advocate @ GReS Studio @Renaldig @renaldigondosubroto
4. CHAOS ON THE CLOUD The way we work is changing due to Covid-19, including the bigger transition towards serverless applications.
5. THE MAIN PRINCIPLES OF THE CLOUD Resiliency Availability Adaptability
6. THE FIRST ENDEAVOR: DEFINING STABILITY  What is considered normal behavior?  How do you want to choose to measure success and what metrics would you like to use?  How does the normal behavior align with business objectives?
7. THE SECOND ENDEAVOR: THE ESTIMATION PROCESS  Use experience to consider how to solve the problem  Think where Chaos can be injected based on that
8. THE THIRD ENDEAVOR: EXECUTION  Create a flow diagram of how you plan to execute  Prepare containment methods  Notification
9. THE FOURTH ENDEAVOR: EVALUATE  Define metrics that measure success  How do you measure resiliency needed  Continuous evaluation
10. THE FIFTH ENDEAVOR: REITERATION  Improve  Understand the effects
11. WHAT TO WATCH OUT FOR Handling various responses Creating chaos to an acceptable level Failover plans
12. A SIMPLE SAMPLE ARCHITECTURE
13. CREATING YOUR OWN PLAYGROUND TO EXPERIMENT  Create a test account  Experiment with varying parameters
14. LATENCY INJECTION IN AWS Latency spikes are something one should consider early API Gateway’s hard limit of 29s for timeout
15. LATENCY INJECTION IN AWS CONT. Inject within the HTTP client library Test for function’s timeout and that it can degrade accordingly in the request’s timeout Depends on other services also used
16. TWO SAMPLES OF APPLYING CHAOS  DynamoDB  Lambda  Can be done anywhere in AWS, not only in those two
17. APPLYING CHAOS IN DYNAMODB
18. APPLYING CHAOS IN LAMBDA
19. CREATING A POST-INCIDENT STRATEGY  What happened?  How did this affect your service?  How can you control the chaos better next time?  What metrics need to be reevaluated?  What are you already doing now to fix it?
20. CREATING A POST-INCIDENT STRATEGY CONT.  Create a weekly inspection schedule for the chaos engineering flow  Focus on maintaining that every item is still working well  Discussions in an agile-scrum culture
21. A CASE STUDY  Building a resilient system for an IoT network  Considering client constraints of low bandwidth in areas  How to best utilize the cloud platform to facilitate it
22. A CASE STUDY: WHAT WE DID  Selection of testing within Lambda  Testing putting high pressure on bandwidth in locations  Schedule testing accordingly
23. THE NEED FOR AGILE IN CHAOS ENGINEERING  Agile is all about iterative development  Chaos engineering needs fast reactions  Prioritizing disaster recovery
24. WHAT WE GOT OUT OF IT AGILITY SATISFACTION CULTURE
25. THANK YOU RENALDI GONDOSUBROTO