Chaos engineering for serverless applications - AWS Resiliency and Chaos Engineering October 27 2020

Presented at AWS Resiliency and Chaos Engineering, October 27th, 2020.

Video available here: https://youtu.be/a4SBhCqus_g

https://twitter.com/gunnargrosch
https://demo.serverlesschaos.com
https://github.com/gunnargrosch/failure-lambda

Planning and performing chaos experiments on instance- and container-based workloads have been battle-tested by companies of all sizes and industries. However, serverless functions and managed services present different failure modes and levels of abstraction. In this session, we focus on applying chaos engineering principles to serverless, both for serverless functions and managed services. This covers how hypotheses are formed to fit serverless, what the experiments can achieve, and how to perform them practically.

Transcript

1. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Gunnar Grosch @gunnargrosch October 27-28, 2020 Chaos engineering for serverless applications AWS Resiliency and Chaos Engineering
2. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Motivations behind chaos engineering
3. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Motivations behind chaos engineering Downtime or issues Happy users Learning from incidents Monitoring and alerting Organizational confidence
4. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Serverless chaos experiments
5. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What is serverless? No infrastructure provisioning, no management Automatic scaling Pay for value Highly available and secure
6. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Serverless chaos experiments Errors Fallbacks Timeouts Events
7. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Serverless chaos experiments Inject errors into your code Remove downstream services Alter the concurrency of functions Restrict the capacity of tables Client Amazon Simple Storage Service (Amazon S3) Amazon API Gateway AWS Lambda Amazon DynamoDB AWS Lambda Amazon Simple Storage Service (Amazon S3)
8. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Serverless chaos experiments Security policy errors CORS configuration errors Service configuration errors Function disk space failure Client Amazon Simple Storage Service (Amazon S3) Amazon API Gateway AWS Lambda Amazon DynamoDB AWS Lambda Amazon Simple Storage Service (Amazon S3)
9. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Serverless chaos experiments Add latency to your functions • Cold starts • Runtime or code issues • Integration issues • Timeouts Client Amazon Simple Storage Service (Amazon S3) Amazon API Gateway AWS Lambda Amazon DynamoDB AWS Lambda Amazon Simple Storage Service (Amazon S3)
10. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Chaos engineering for serverless tools Chaos-lambda Python Failure-lambda NodeJS
11. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Failure-lambda NodeJS NPM package for NodeJS Lambdas Configuration using Parameter Store or AWS AppConfig Several failure modes • Latency • Status code • Exception • Disk space • Denylist const failureLambda = require('failure-lambda’) exports.handler = failureLambda(async (event, context) => { ... }) { "isEnabled": false, "failureMode": "latency", "rate": 1, "minLatency": 100, "maxLatency": 400, "exceptionMsg": "Exception message!", "statusCode": 404, "diskSpace": 100, “denylist": [ "s3.*.amazonaws.com", "dynamodb.*.amazonaws.com" ] }
12. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Serverless chaos demo
13. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Serverless chaos demo
14. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Serverless chaos demo Client Amazon S3 Amazon API Gateway AWS Lambda Amazon DynamoDB AWS Lambda AWS Lambda
15. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Client Amazon S3 Amazon API Gateway AWS Lambda Amazon DynamoDB AWS Lambda AWS Lambda Serverless chaos demo • What if my function takes an extra 300 ms for each invocation? • What if my function returns an error code? • What if I can’t get data from DynamoDB?
16. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Demo
17. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What’s next?
18. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Serverless chaos CI/CD demo
19. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Serverless chaos CI/CD demo • What if my function takes an extra 300 ms for each invocation? • What if my function returns an error code? • What if I can’t get data from DynamoDB? Default deploy
20. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Feature flag Serverless chaos CI/CD demo • What if my function takes an extra 300 ms for each invocation? • What if my function returns an error code? • What if I can’t get data from DynamoDB?
21. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Canary deploy Serverless chaos CI/CD demo • What if my function takes an extra 300 ms for each invocation? • What if my function returns an error code? • What if I can’t get data from DynamoDB?
22. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Demo
23. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Application configuration control
24. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Adding application configuration control AWS AppConfig allows us to create, manage, and quickly deploy config Features: • Validate failure configuration • Deploy configuration using gradual or non- gradual deploy strategy • Monitor deployed configuration with automatical rollback
25. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Serverless chaos AppConfig demo
26. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Demo
27. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Summary Use chaos engineering to find weaknesses and fix them Use chaos engineering to verify that your system behaves as expected Use chaos engineering to build confidence Chaos engineering should be done regularly It’s easy to get started.
28. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you! Gunnar Grosch @gunnargrosch