Serverless chaos - Do we need to worry about disk space in serverless? July 31 2019

Slides used for YouTube video:
https://youtu.be/U8o6-jxkPns

Chaos Engineering on a Serverless AWS environment using disk failure injection. Looking at storage for serverless applications using AWS Lambda and tmp storage. Creating a chaos engineering experiment using disk space failure as an additional failure mode in Adrian's Failure Injection Layer.

Transcript

1. Serverless chaos: Disk space failure Do we need to worry about disk space in serverless?
2. Serverless Chaos Demo – Last video • Looked at the new failure modes in Adrian’s failure injection layer: • Status code injection • Exception injection • Tested the functionality for rate control of failure • Demoed the new Serverless Chaos demo app
3. Serverless Chaos Demo – Last video
4. Serverless Chaos Demo – Last video
5. AWS Lambda Storage • 512 MB /tmp directory storage https://docs.aws.amazon.com/lambda/latest/dg/limits.html • Ephemeral storage shared between multiple invocations • Used until recycled or another Lambda instance is deployed • We don’t control if new instances are started or existing ones are re-used • Yes, we need to think about storage in serverless
6. Disk function demo
7. Disk corruption injection • Using disk duplicator command as a subprocess • The dd command used in this example: dd if=/dev/zero of=/tmp/corrupt-diskspace-1234567890.tmp count=1024 bs=1024 • Is there a faster way of filling disk space?
8. Things to consider • Disabling corrupt_diskspace doesn’t clean /tmp • Cleanup /tmp within your functions if possible • Handle errors thrown if disk is full • Could other storage solution (S3 or DynamoDB) be used instead? • As always: Graceful degradation!
9. @serverlesschaos #chaosengineering