Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Safe Lambda Deployments - Seattle - 2019-02-21

99d4c0ed3c8027ea1c37b53d6441d042?s=47 sullis
February 21, 2019

Safe Lambda Deployments - Seattle - 2019-02-21

Seattle Serverless Meetup
February 21, 2019

#awslambda #awscloud #serverless #seattle



February 21, 2019

More Decks by sullis

Other Decks in Technology


  1. SAFE LAMBDA DEPLOYMENTS Sean Sullivan February 21, 2019 Seattle Serverless

  2. software engineer Portland Oregon e-commerce systems Hudson’s Bay Company About

  3. None
  4. saksfifthavenue.com saksoff5th.com lordandtaylor.com thebay.com

  5. saksfifthavenue.com

  6. legacy systems cloud native systems

  7. “You go to production with the code you have, not

    the code you might want or wish to have at a later time.” Donald Rumsfeld December 2004 * fake quote
  8. servers serverless

  9. “Provisioning servers is a waste of time. It destroys your

    budget and your self-respect and everything that goes along with your self-esteem.” Kurt Cobain web developer 1993 * fake quote
  10. production deployments

  11. “My songs have always been frustrating themes, inspired by bad

    production deployments that I've had” Kurt Cobain web developer 1993 * fake quote
  12. Zach Holman March 2016 https://zachholman.com/posts/deploying-software

  13. “Your deploys should be as boring, straightforward, and stress-free as

    possible” Zach Holman March 2016 https://zachholman.com/posts/deploying-software
  14. “You don't need release managers, you don't need special deploy

    days, you don't need all hands on deck for every single deploy.” Zach Holman March 2016 https://zachholman.com/posts/deploying-software
  15. “We love to deploy new code incrementally hundreds of times

    a day. And there's good reason for that: it's safer overall. Incremental deploys are easier to understand and fix than one gigantic deploy once a year.” https://zachholman.com/talk/move-fast-break-nothing/ Zach Holman October 2014
  16. https://twitter.com/gwenshap/status/1059870433736937472 QCon San Francisco November 2018

  17. deployment process

  18. “I never wanted to sing, I just wanted to write

    code and deploy to production.” Kurt Cobain web developer 1993 * fake quote
  19. autonomy

  20. None
  21. bureaucracy freedom and responsibility

  22. HBC 2019 microservice architecture multi-tenant services increased adoption of AWS

    Lambda safe deployments
  23. system plumbing business logic

  24. None
  25. HBC email types • order acknowledgement • order shipment partial

    • order shipment full • order cancel • ready for pickup in store
  26. Queue consumer Event queue Email Dashboard UI Email Sender Email

    Render External Email Provider Feature Flag service
  27. Email Render Product Detail service Delivery Date Estimation service request

    HTML Feature Flag service
  28. HBC email system • multi-tenant design • Cloudwatch Metrics •

    Cloudwatch Alerts
  29. HBC email system • code coverage threshold • components deployed

    independently • safe deployments
  30. Mark Zuckerberg

  31. Move Fast and Break Things Move Slow and Don’t Break

    Things Move Fast and Don’t Break Things Move Purposefully and Fix Things
  32. manual testing automated testing

  33. safe AWS Lambda deployments?

  34. None
  35. Safe Lambda deployments validation hooks traffic shifting

  36. None
  37. Traffic Shifting

  38. You can now shift incoming traffic between two AWS Lambda

    function versions based on pre-assigned weights. You can now also use AWS CodeDeploy to automatically manage the rollout of new function versions. This allows you to gradually shift traffic between two versions source: aws.amazon.com
  39. Traffic Shifting configurations AllAtOnce Canary10Percent30Minutes Canary10Percent5Minutes Canary10Percent10Minutes Canary10Percent15Minutes

  40. Linear10PercentEvery10Minutes Linear10PercentEvery1Minute Linear10PercentEvery2Minutes Linear10PercentEvery3Minutes Traffic Shifting configurations

  41. all-at-once traffic shifting gradual traffic shifting

  42. AWS command line tools

  43. $ brew update $ brew tap aws/tap $ brew install

    aws/tap/aws-sam-cli $ brew install aws-cli
  44. Live demo

  45. aws cloudformation package \ --template-file my-template.yaml \ --s3-bucket mycorp-codedeploy \

    --s3-prefix my-lambda-func/jars \ --output-template-file package/2325bd1e.yaml
  46. aws cloudformation deploy \ --template-file package/2325bd1e.yaml \ --stack-name mylambda-func—xyz \

    --parameter-overrides Stage=xyz \ --region us-east-1 \ --capabilities CAPABILITY_IAM \ --notification-arns arn:aws:sns:us- east-1:123456789:myteam-cf-notifications
  47. aws cloudformation deploy \ --template-file package/2325bd1e.yaml \ --stack-name mylambda-func—xyz \

    --parameter-overrides Stage=xyz \ --parameter-overrides DeploymentPref=Canary10Percent5Minutes \ --region us-east-1 \ --capabilities CAPABILITY_IAM \ --notification-arns arn:aws:sns:us- east-1:123456789:myteam-cf-notifications
  48. Traffic Hooks

  49. AWS traffic hooks • PreTraffic hook • PostTraffic hook

  50. AWS CodeDeploy UI https://console.aws.amazon.com/codesuite/codedeploy/deployments

  51. https://github.com/awslabs/serverless-application-model/pull/605

  52. AWS re:Invent 2018

  53. Clare Liguori November 2018 https://www.youtube.com/watch?v=01ewawuL-IY

  54. Clare Liguori November 2018 https://www.youtube.com/watch?v=01ewawuL-IY

  55. additional validation logic in Pre Traffic hook implement Post Traffic

    hook utilize AWS Cloud9 IDE Future work at HBC
  56. Final thoughts

  57. traffic shifting

  58. small incremental releases

  59. engineering velocity is a competitive advantage

  60. “I'd rather be hated for the code that I’ve deployed,

    than loved for the songs I've written” Kurt Cobain web developer 1993 * fake quote
  61. Thank you!

  62. None