Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Safe Lambda Deployments - San Francisco - 2019-06-13

99d4c0ed3c8027ea1c37b53d6441d042?s=47 sullis
June 13, 2019

Safe Lambda Deployments - San Francisco - 2019-06-13

AWS Lambda
safe deployments
Serverless Nights San Francisco
June 13, 2019



June 13, 2019


  1. Sean Sullivan June 13, 2019 San Francisco Serverless Nights SAFE

  2. software engineer Portland Oregon e-commerce systems Hudson’s Bay Company About

  3. None
  4. None
  5. saksfifthavenue.com saksoff5th.com lordandtaylor.com thebay.com

  6. saksfifthavenue.com

  7. legacy systems cloud native systems

  8. servers serverless

  9. Trying to build serverless applications and chase rings. That’s all

    I'm about #DubNation
  10. @StephenCurry30 LOL 1-800-SERVERLESS 1-800-SERVERLESS

  11. @Drake I probably relate more to the casual developer who

    writes JavaScript and is not super athletic
  12. @StephenCurry30 LET’S GO!!! 1-800-RAPTORS 1-800-RAPTORS

  13. @Drake deploying code is the most important part of the

  14. @Drake I try to make production deployments look easy, but

    the behind-the-scenes stuff is a challenge
  15. production deployments

  16. Losing game 1 to the Raptors was frustrating, but not

    as frustrating as the bad production deployments that I've seen
  17. Extended production outages are pretty devastating. That feeling never goes

    away ☹
  18. Zach Holman March 2016 https://zachholman.com/posts/deploying-software

  19. “Your deploys should be as boring, straightforward, and stress-free as

    possible” Zach Holman March 2016 https://zachholman.com/posts/deploying-software
  20. “You don't need release managers, you don't need special deploy

    days, you don't need all hands on deck for every single deploy.” Zach Holman March 2016 https://zachholman.com/posts/deploying-software
  21. “We love to deploy new code incrementally hundreds of times

    a day. And there's good reason for that: it's safer overall. Incremental deploys are easier to understand and fix” https://zachholman.com/talk/move-fast-break-nothing/ Zach Holman October 2014
  22. deployment process

  23. I never wanted to play basketball, I just wanted to

    write code and deploy to production
  24. autonomy

  25. None
  26. bureaucracy freedom and responsibility

  27. HBC 2019 microservice architecture multi-tenant services increased adoption of AWS

    Lambda safe deployments
  28. system plumbing business logic

  29. None
  30. HBC email types • order acknowledgement • order shipment partial

    • order shipment full • order cancel • ready for pickup in store
  31. Queue consumer Event queue Email Dashboard UI Email Sender Email

    Render External Email Provider Feature Flag service
  32. Email Render Product Detail service Delivery Date Estimation service request

    HTML Feature Flag service
  33. HBC email system multi-tenant design Cloudwatch Metrics Cloudwatch Alarms

  34. HBC email system code coverage threshold components deployed independently safe

  35. Move Fast and Break Things Move Slow and Don’t Break

    Things Move Fast and Don’t Break Things Move Purposefully and Fix Things
  36. manual testing automated testing

  37. safe AWS Lambda deployments?

  38. None
  39. Safe Lambda deployments validation hooks traffic shifting

  40. YAML

  41. None
  42. Traffic Shifting

  43. You can now shift incoming traffic between two AWS Lambda

    function versions based on pre-assigned weights. You can now also use AWS CodeDeploy to automatically manage the rollout of new function versions. This allows you to gradually shift traffic between two versions source: aws.amazon.com
  44. Traffic Shifting configurations AllAtOnce Canary10Percent30Minutes Canary10Percent5Minutes Canary10Percent10Minutes Canary10Percent15Minutes

  45. Linear10PercentEvery10Minutes Linear10PercentEvery1Minute Linear10PercentEvery2Minutes Linear10PercentEvery3Minutes Traffic Shifting configurations

  46. all-at-once traffic shifting gradual traffic shifting

  47. AWS command line tools

  48. $ brew update $ brew tap aws/tap $ brew install

    aws/tap/aws-sam-cli $ brew install aws-cli
  49. aws cloudformation package \ --template-file my-template.yaml \ --s3-bucket mycorp-codedeploy \

    --s3-prefix my-lambda-func/jars \ --output-template-file package/2325bd1e.yaml
  50. aws cloudformation deploy \ --template-file package/2325bd1e.yaml \ --stack-name mylambda-func—xyz \

    --parameter-overrides Stage=xyz \ --parameter-overrides DeploymentPref=Canary10Percent5Minutes \ --region us-east-1 \ --capabilities CAPABILITY_IAM \ --notification-arns arn:aws:sns:us- east-1:123456789:myteam-cf-notifications
  51. Traffic Hooks

  52. AWS traffic hooks PreTraffic hook PostTraffic hook

  53. AWS CodeDeploy UI https://console.aws.amazon.com/codesuite/codedeploy/deployments

  54. https://github.com/awslabs/serverless-application-model/pull/605

  55. additional validation logic in Pre Traffic hook implement Post Traffic

    hook utilize AWS Cloud9 IDE Future work at HBC
  56. Final thoughts

  57. traffic shifting

  58. small incremental releases

  59. engineering velocity is a competitive advantage

  60. I'd rather be hated for the code that I’ve deployed

    than loved for winning an NBA title
  61. THANK YOU!

  62. None
  63. Bonus slides

  64. To win MVP was a dream come true for sure,

    but I'd rather deploy code to production
  65. Always Be Deploying