Continuous verification for serverless applications - Serverless WEEK October 30 2020

Continuous verification for serverless applications - Serverless WEEK October 30 2020

Presented at Serverless WEEK, October 30th, 2020.

@gunnargrosch
Serverless Chaos Demo
failure-lambda

The transition into more complex systems is accelerating and the granularity in serverless applications makes even smaller systems complex and highly distributed. Chaos engineering draws from the rich history of empirical experimentation to proactively discover vulnerabilities in these complex systems and help us verify whether or not the system behaves as expected under adverse conditions. Join as we raise the bar for how chaos engineering can be used with serverless applications and help us to continuously verify the output of the system through automation and the advantages established by CI/CD.

B2fefbb30aba7c25bbe0c8819791631a?s=128

Gunnar Grosch

October 30, 2020
Tweet

Transcript

  1. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Gunnar Grosch @gunnargrosch October 30, 2020 Continuous verification for serverless applications Serverless WEEK
  2. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. “Testing to ensure that you can meet your availability goals is the only way you can have confidence that you will meet those goals” Reliability Pillar AWS Well-Architected Framework
  3. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Verification of applications
  4. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Verification of applications Objectives Verify Measures Conditions
  5. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Verification of applications Objectives Verify Measures Conditions
  6. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Conditions for verification Errors in your code Security policy errors Service configuration errors Function disk space failure Downstream service issues Concurrency and throttling Latency Client Amazon Simple Storage Service (Amazon S3) Amazon API Gateway AWS Lambda Amazon DynamoDB AWS Lambda Amazon Simple Storage Service (Amazon S3)
  7. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Tools for adding conditions Chaos-lambda Python Failure-lambda NodeJS
  8. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Failure-lambda NodeJS NPM package for NodeJS Lambdas Configuration using Parameter Store or AWS AppConfig Several failure modes • Latency • Status code • Exception • Disk space • Denylist const failureLambda = require('failure-lambda’) exports.handler = failureLambda(async (event, context) => { ... }) { "isEnabled": false, "failureMode": "latency", "rate": 1, "minLatency": 100, "maxLatency": 400, "exceptionMsg": "Exception message!", "statusCode": 404, "diskSpace": 100, “denylist": [ "s3.*.amazonaws.com", "dynamodb.*.amazonaws.com" ] }
  9. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Verification of applications Objectives Verify Measures Conditions
  10. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Verification example Objective: My purchase API should respond in less than 400 ms Measure: 400 ms threshold Condition: 100-200 ms latency injection to function Verify: Pass or fail Client Amazon API Gateway Amazon DynamoDB AWS Lambda
  11. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Verification example Objective: I want 10 clicks per second Measure: 10 clicks threshold Condition: Inject downstream service failure Verify: Pass or fail Client Amazon API Gateway Amazon DynamoDB AWS Lambda
  12. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Verification demo
  13. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Verification demo
  14. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Verification demo Client Amazon S3 Amazon API Gateway AWS Lambda Amazon DynamoDB AWS Lambda AWS Lambda
  15. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Client Amazon S3 Amazon API Gateway AWS Lambda Amazon DynamoDB AWS Lambda AWS Lambda Verification demo • Condition: Add 100-400 ms latency for each invocation • Condition: Return error codes on some invocations • Condition: Intercept and deny connections to DynamoDB
  16. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Demo
  17. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Making verification continuous
  18. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Making verification continuous
  19. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Making verification continuous Objective: My purchase API should respond in less than 400 ms Measure: 400 ms threshold Condition: 100-200 ms latency injection to function Verify: Pass Default deploy
  20. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Making verification continuous Objective: My purchase API should respond in less than 400 ms Measure: 400 ms threshold Condition: 100-200 ms latency injection to function Verify: Fail Canary deploy
  21. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Making verification continuous Objective: My purchase API should respond in less than 400 ms Measure: 400 ms threshold Condition: 100-200 ms latency injection to function Verify: Pass Feature flag
  22. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Continuous verification demo
  23. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Continuous verification demo
  24. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Continuous verification demo
  25. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Continuous verification demo Objective: Users should frequently get new images Measure: 10 images loaded per minute Condition: Return error codes on invocations Verify: Pass or Fail Canary deploy
  26. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Demo
  27. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Key learnings Verification tells you why it’s important to explore a set of conditions It’s not about breaking things, it’s about learning and building confidence It’s easy to get started.
  28. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. “Testing to ensure that you can meet your availability goals is the only way you can have confidence that you will meet those goals” Reliability Pillar AWS Well-Architected Framework
  29. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Do you want more? Reliability pillar https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/welcome.html Serverless Chaos Demo app https://demo.serverlesschaos.com Failure-lambda https://github.com/gunnargrosch/failure-lambda Chaos-lambda https://github.com/adhorn/aws-lambda-chaos-injection/
  30. © 2020, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Thank you! Gunnar Grosch @gunnargrosch