Safe Lambda Deployments - Seattle - 2019-02-21

Seattle Serverless Meetup
February 21, 2019

#awslambda #awscloud #serverless #seattle

Transcript

1. SAFE
   LAMBDA
   DEPLOYMENTS
   Sean Sullivan
   February 21, 2019
   Seattle Serverless
   

   View Slide

2. software engineer
   Portland Oregon
   e-commerce systems
   Hudson’s Bay Company
   About me
   

   View Slide

3. View Slide

4. saksfifthavenue.com
   saksoff5th.com
   lordandtaylor.com
   thebay.com
   

   View Slide

5. saksfifthavenue.com
   

   View Slide

6. legacy systems
   cloud native systems
   

   View Slide

7. “You go to production with
   the code you have, not the
   code you might want or wish
   to have at a later time.”
   Donald Rumsfeld
   December 2004
   * fake quote
   

   View Slide

8. servers
   serverless
   

   View Slide

9. “Provisioning servers
   is a waste of time. It destroys your
   budget and your self-respect and
   everything that goes along with your
   self-esteem.”
   Kurt Cobain
   web developer
   1993
   * fake quote
   

   View Slide

10. production
    deployments
    

    View Slide

11. “My songs have always been
    frustrating themes,
    inspired by
    bad production deployments
    that I've had”
    Kurt Cobain
    web developer
    1993
    * fake quote
    

    View Slide

12. Zach Holman
    March 2016
    https://zachholman.com/posts/deploying-software
    

    View Slide

13. “Your deploys should be as
    boring, straightforward, and
    stress-free as possible”
    Zach Holman
    March 2016
    https://zachholman.com/posts/deploying-software
    

    View Slide

14. “You don't need release managers,
    you don't need special deploy
    days, you don't need all hands on
    deck for every single deploy.”
    Zach Holman
    March 2016
    https://zachholman.com/posts/deploying-software
    

    View Slide

15. “We love to deploy new code
    incrementally hundreds of times a day.
    And there's good reason for that: it's
    safer overall. Incremental deploys are
    easier to understand and fix than one
    gigantic deploy once a year.”
    https://zachholman.com/talk/move-fast-break-nothing/
    Zach Holman
    October 2014
    

    View Slide

16. https://twitter.com/gwenshap/status/1059870433736937472
    QCon San Francisco
    November 2018
    

    View Slide

17. deployment process
    

    View Slide

18. “I never wanted to sing,
    I just wanted to write code
    and deploy to production.”
    Kurt Cobain
    web developer
    1993
    * fake quote
    

    View Slide

19. autonomy
    

    View Slide

20. View Slide

21. bureaucracy
    freedom
    and
    responsibility
    

    View Slide

22. HBC 2019
    microservice architecture
    multi-tenant services
    increased adoption of AWS Lambda
    safe deployments
    

    View Slide

23. system
    plumbing
    business logic
    

    View Slide

24. View Slide

25. HBC email types
    • order acknowledgement
    • order shipment partial
    • order shipment full
    • order cancel
    • ready for pickup in store
    

    View Slide

26. Queue
    consumer
    Event
    queue
    Email
    Dashboard UI
    Email
    Sender
    Email
    Render
    External
    Email
    Provider
    Feature
    Flag
    service
    

    View Slide

27. Email
    Render
    Product
    Detail
    service
    Delivery
    Date
    Estimation
    service
    request
    HTML
    Feature
    Flag
    service
    

    View Slide

28. HBC email system
    • multi-tenant design
    • Cloudwatch Metrics
    • Cloudwatch Alerts
    

    View Slide

29. HBC email system
    • code coverage threshold
    • components deployed independently
    • safe deployments
    

    View Slide

30. Mark Zuckerberg
    

    View Slide

31. Move Fast and Break Things
    Move Slow and Don’t Break Things
    Move Fast and Don’t Break Things
    Move Purposefully and Fix Things
    

    View Slide

32. manual
    testing
    automated
    testing
    

    View Slide

33. safe
    AWS Lambda
    deployments?
    

    View Slide

34. View Slide

35. Safe Lambda deployments
    validation hooks
    traffic shifting
    

    View Slide

36. View Slide

37. Traffic
    Shifting
    

    View Slide

38. You can now shift
    incoming traffic between
    two AWS Lambda
    function versions based on
    pre-assigned weights.
    You can now also
    use AWS
    CodeDeploy to
    automatically manage
    the rollout of new
    function versions.
    This allows you to
    gradually shift traffic
    between two versions source: aws.amazon.com
    

    View Slide

39. Traffic Shifting configurations
    AllAtOnce
    Canary10Percent30Minutes
    Canary10Percent5Minutes
    Canary10Percent10Minutes
    Canary10Percent15Minutes
    

    View Slide

40. Linear10PercentEvery10Minutes
    Linear10PercentEvery1Minute
    Linear10PercentEvery2Minutes
    Linear10PercentEvery3Minutes
    Traffic Shifting configurations
    

    View Slide

41. all-at-once
    traffic shifting
    gradual
    traffic shifting
    

    View Slide

42. AWS
    command line tools
    

    View Slide

43. $ brew update
    $ brew tap aws/tap
    $ brew install aws/tap/aws-sam-cli
    $ brew install aws-cli
    

    View Slide

44. Live demo
    

    View Slide

45. aws cloudformation package \
    --template-file my-template.yaml \
    --s3-bucket mycorp-codedeploy \
    --s3-prefix my-lambda-func/jars \
    --output-template-file package/2325bd1e.yaml
    

    View Slide

46. aws cloudformation deploy \
    --template-file package/2325bd1e.yaml \
    --stack-name mylambda-func—xyz \
    --parameter-overrides Stage=xyz \
    --region us-east-1 \
    --capabilities CAPABILITY_IAM \
    --notification-arns arn:aws:sns:us-
    east-1:123456789:myteam-cf-notifications
    

    View Slide

47. aws cloudformation deploy \
    --template-file package/2325bd1e.yaml \
    --stack-name mylambda-func—xyz \
    --parameter-overrides Stage=xyz \
    --parameter-overrides
    DeploymentPref=Canary10Percent5Minutes \
    --region us-east-1 \
    --capabilities CAPABILITY_IAM \
    --notification-arns arn:aws:sns:us-
    east-1:123456789:myteam-cf-notifications
    

    View Slide

48. Traffic
    Hooks
    

    View Slide

49. AWS traffic hooks
    • PreTraffic hook
    • PostTraffic hook
    

    View Slide

50. AWS CodeDeploy UI
    https://console.aws.amazon.com/codesuite/codedeploy/deployments
    

    View Slide

51. https://github.com/awslabs/serverless-application-model/pull/605
    

    View Slide

52. AWS re:Invent 2018
    

    View Slide

53. Clare Liguori
    November 2018
    https://www.youtube.com/watch?v=01ewawuL-IY
    

    View Slide

54. Clare Liguori
    November 2018
    https://www.youtube.com/watch?v=01ewawuL-IY
    

    View Slide

55. additional validation logic in
    Pre Traffic hook
    implement Post Traffic hook
    utilize AWS Cloud9 IDE
    Future work at HBC
    

    View Slide

56. Final thoughts
    

    View Slide

57. traffic shifting
    

    View Slide

58. small incremental releases
    

    View Slide

59. engineering velocity is a
    competitive advantage
    

    View Slide

60. “I'd rather be hated for
    the code that I’ve deployed,
    than loved for
    the songs I've written”
    Kurt Cobain
    web developer
    1993
    * fake quote
    

    View Slide

61. Thank you!
    

    View Slide

62. View Slide