Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Safe Lambda Deployments - Seattle - 2019-02-21

sullis
February 21, 2019

Safe Lambda Deployments - Seattle - 2019-02-21

Seattle Serverless Meetup
February 21, 2019

#awslambda #awscloud #serverless #seattle

sullis

February 21, 2019
Tweet

More Decks by sullis

Other Decks in Technology

Transcript

  1. SAFE
    LAMBDA
    DEPLOYMENTS
    Sean Sullivan
    February 21, 2019
    Seattle Serverless

    View Slide

  2. software engineer
    Portland Oregon
    e-commerce systems
    Hudson’s Bay Company
    About me

    View Slide

  3. View Slide

  4. saksfifthavenue.com
    saksoff5th.com
    lordandtaylor.com
    thebay.com

    View Slide

  5. saksfifthavenue.com

    View Slide

  6. legacy systems
    cloud native systems

    View Slide

  7. “You go to production with
    the code you have, not the
    code you might want or wish
    to have at a later time.”
    Donald Rumsfeld
    December 2004
    * fake quote

    View Slide

  8. servers
    serverless

    View Slide

  9. “Provisioning servers
    is a waste of time. It destroys your
    budget and your self-respect and
    everything that goes along with your
    self-esteem.”
    Kurt Cobain
    web developer
    1993
    * fake quote

    View Slide

  10. production
    deployments

    View Slide

  11. “My songs have always been
    frustrating themes,
    inspired by
    bad production deployments
    that I've had”
    Kurt Cobain
    web developer
    1993
    * fake quote

    View Slide

  12. Zach Holman
    March 2016
    https://zachholman.com/posts/deploying-software

    View Slide

  13. “Your deploys should be as
    boring, straightforward, and
    stress-free as possible”
    Zach Holman
    March 2016
    https://zachholman.com/posts/deploying-software

    View Slide

  14. “You don't need release managers,
    you don't need special deploy
    days, you don't need all hands on
    deck for every single deploy.”
    Zach Holman
    March 2016
    https://zachholman.com/posts/deploying-software

    View Slide

  15. “We love to deploy new code
    incrementally hundreds of times a day.
    And there's good reason for that: it's
    safer overall. Incremental deploys are
    easier to understand and fix than one
    gigantic deploy once a year.”
    https://zachholman.com/talk/move-fast-break-nothing/
    Zach Holman
    October 2014

    View Slide

  16. https://twitter.com/gwenshap/status/1059870433736937472
    QCon San Francisco
    November 2018

    View Slide

  17. deployment process

    View Slide

  18. “I never wanted to sing,
    I just wanted to write code
    and deploy to production.”
    Kurt Cobain
    web developer
    1993
    * fake quote

    View Slide

  19. autonomy

    View Slide

  20. View Slide

  21. bureaucracy
    freedom
    and
    responsibility

    View Slide

  22. HBC 2019
    microservice architecture
    multi-tenant services
    increased adoption of AWS Lambda
    safe deployments

    View Slide

  23. system
    plumbing
    business logic

    View Slide

  24. View Slide

  25. HBC email types
    • order acknowledgement
    • order shipment partial
    • order shipment full
    • order cancel
    • ready for pickup in store

    View Slide

  26. Queue
    consumer
    Event
    queue
    Email
    Dashboard UI
    Email
    Sender
    Email
    Render
    External
    Email
    Provider
    Feature
    Flag
    service

    View Slide

  27. Email
    Render
    Product
    Detail
    service
    Delivery
    Date
    Estimation
    service
    request
    HTML
    Feature
    Flag
    service

    View Slide

  28. HBC email system
    • multi-tenant design
    • Cloudwatch Metrics
    • Cloudwatch Alerts

    View Slide

  29. HBC email system
    • code coverage threshold
    • components deployed independently
    • safe deployments

    View Slide

  30. Mark Zuckerberg

    View Slide

  31. Move Fast and Break Things
    Move Slow and Don’t Break Things
    Move Fast and Don’t Break Things
    Move Purposefully and Fix Things

    View Slide

  32. manual
    testing
    automated
    testing

    View Slide

  33. safe
    AWS Lambda
    deployments?

    View Slide

  34. View Slide

  35. Safe Lambda deployments
    validation hooks
    traffic shifting

    View Slide

  36. View Slide

  37. Traffic
    Shifting

    View Slide

  38. You can now shift
    incoming traffic between
    two AWS Lambda
    function versions based on
    pre-assigned weights.
    You can now also
    use AWS
    CodeDeploy to
    automatically manage
    the rollout of new
    function versions.
    This allows you to
    gradually shift traffic
    between two versions source: aws.amazon.com

    View Slide

  39. Traffic Shifting configurations
    AllAtOnce
    Canary10Percent30Minutes
    Canary10Percent5Minutes
    Canary10Percent10Minutes
    Canary10Percent15Minutes

    View Slide

  40. Linear10PercentEvery10Minutes
    Linear10PercentEvery1Minute
    Linear10PercentEvery2Minutes
    Linear10PercentEvery3Minutes
    Traffic Shifting configurations

    View Slide

  41. all-at-once
    traffic shifting
    gradual
    traffic shifting

    View Slide

  42. AWS
    command line tools

    View Slide

  43. $ brew update
    $ brew tap aws/tap
    $ brew install aws/tap/aws-sam-cli
    $ brew install aws-cli

    View Slide

  44. Live demo

    View Slide

  45. aws cloudformation package \
    --template-file my-template.yaml \
    --s3-bucket mycorp-codedeploy \
    --s3-prefix my-lambda-func/jars \
    --output-template-file package/2325bd1e.yaml

    View Slide

  46. aws cloudformation deploy \
    --template-file package/2325bd1e.yaml \
    --stack-name mylambda-func—xyz \
    --parameter-overrides Stage=xyz \
    --region us-east-1 \
    --capabilities CAPABILITY_IAM \
    --notification-arns arn:aws:sns:us-
    east-1:123456789:myteam-cf-notifications

    View Slide

  47. aws cloudformation deploy \
    --template-file package/2325bd1e.yaml \
    --stack-name mylambda-func—xyz \
    --parameter-overrides Stage=xyz \
    --parameter-overrides
    DeploymentPref=Canary10Percent5Minutes \
    --region us-east-1 \
    --capabilities CAPABILITY_IAM \
    --notification-arns arn:aws:sns:us-
    east-1:123456789:myteam-cf-notifications

    View Slide

  48. Traffic
    Hooks

    View Slide

  49. AWS traffic hooks
    • PreTraffic hook
    • PostTraffic hook

    View Slide

  50. AWS CodeDeploy UI
    https://console.aws.amazon.com/codesuite/codedeploy/deployments

    View Slide

  51. https://github.com/awslabs/serverless-application-model/pull/605

    View Slide

  52. AWS re:Invent 2018

    View Slide

  53. Clare Liguori
    November 2018
    https://www.youtube.com/watch?v=01ewawuL-IY

    View Slide

  54. Clare Liguori
    November 2018
    https://www.youtube.com/watch?v=01ewawuL-IY

    View Slide

  55. additional validation logic in
    Pre Traffic hook
    implement Post Traffic hook
    utilize AWS Cloud9 IDE
    Future work at HBC

    View Slide

  56. Final thoughts

    View Slide

  57. traffic shifting

    View Slide

  58. small incremental releases

    View Slide

  59. engineering velocity is a
    competitive advantage

    View Slide

  60. “I'd rather be hated for
    the code that I’ve deployed,
    than loved for
    the songs I've written”
    Kurt Cobain
    web developer
    1993
    * fake quote

    View Slide

  61. Thank you!

    View Slide

  62. View Slide