Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Serverless chaos - Containing the blast radius July 3 2019

Serverless chaos - Containing the blast radius July 3 2019

Slides used for YouTube video:
https://youtu.be/Cw-JmAJHG-g

Chaos Engineering on a Serverless AWS environment using latency injection.

Gunnar Grosch

July 03, 2019
Tweet

More Decks by Gunnar Grosch

Other Decks in Technology

Transcript

  1. Serverless Chaos Demo – Last video • We added the

    Chaos Injection layer to one of the functions. • Parameter in Parameter Store enables/disables and configures the latency. • Defined and called delay() in the function code.
  2. Containing the blast radius “Experimenting in production has the potential

    to cause unnecessary customer pain. While there must be an allowance for some short-term negative impact, it is the responsibility and obligation of the Chaos Engineer to ensure the fallout from experiments are minimized and contained.” principlesofchaos.org • Design the smallest possible experiment to effectively test the system. • By starting small, even if things go wrong, it shouldn’t cause an outage. • Understand how failure plays out, then scale it up as confidence in the system grows.
  3. To do • Fork Adrian Hornsby’s LatencyInjectionLayer repository. • Add

    support for separate parameters per function. • Make the layer easy deployable using Serverless Framework. • Update the Serverless Chaos Demo Large App with the new layer. • Add parameters for each function. • Try it out!
  4. Containing the blast radius “Experimenting in production has the potential

    to cause unnecessary customer pain. While there must be an allowance for some short-term negative impact, it is the responsibility and obligation of the Chaos Engineer to ensure the fallout from experiments are minimized and contained.” principlesofchaos.org • Design the smallest possible experiment to effectively test the system. • By starting small, even if things go wrong, it shouldn’t cause an outage. • Understand how failure plays out, then scale it up as confidence in the system grows.