Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Best Practices for Safe Deployments on AWS Lambda and Amazon API Gateway

Best Practices for Safe Deployments on AWS Lambda and Amazon API Gateway

AWS Summit, Berlin, February 27th, 2019

Deploying frequently is fundamental to reducing the feedback loop and increasing developer productivity. There are multiple features available in AWS Lambda, Amazon API Gateway, and AWS Serverless Application Model (AWS SAM) that you can use to implement a continuous deployment pipeline with safe deployment strategies, such as canary releases. In this session, we review the possible options applied to different scenarios, such as microservices architectures, chaos engineering, and A/B testing to discover the best practices for your use cases.

Danilo Poccia

February 27, 2019
Tweet

More Decks by Danilo Poccia

Other Decks in Programming

Transcript

  1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Best Practices for Safe Deployments on
    AWS Lambda and Amazon API Gateway
    Danilo Poccia
    Principal Evangelist, Serverless
    AWS
    @danilop

    View Slide

  2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Listen
    Iterate
    Experiment
    Innovation
    Flywheel
    Experiments power the engine of rapid innovation

    View Slide

  3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Infrastructure as code

    Make infrastructure
    changes repeatable and
    predictable

    Release infrastructure
    changes using the same
    tools as code changes

    Replicate production
    environment in a
    staging environment to
    enable continuous
    testing

    View Slide

  4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Infrastructure as code
    Declarative
    I tell you
    what I need
    I tell you
    what to do
    Imperative

    View Slide

  5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Infrastructure as code best practices

    Infrastructure
    and application
    in the same
    source repository
    For example:
    AWS CloudFormation
    HashiCorp Terraform

    Deployments
    include
    infrastructure
    updates

    View Slide

  6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Infrastructure as code for serverless apps
    For example:
    AWS Serverless Application Model (SAM)
    Serverless Framework
    Lambda
    Functions
    DynamoDB
    Tables
    S3
    Buckets
    API
    Gateways

    View Slide

  7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    SAM template
    AWSTemplateFormatVersion: '2010-09-09’
    Transform: AWS::Serverless-2016-10-31
    Resources:
    GetFunction:
    Type: AWS::Serverless::Function
    Properties:
    Handler: index.get
    Runtime: nodejs8.10
    CodeUri: src/
    Policies:
    - DynamoDBReadPolicy:
    TableName: !Ref MyTable
    Events:
    GetResource:
    Type: Api
    Properties:
    Path: /resource/{resourceId}
    Method: get
    MyTable:
    Type: AWS::Serverless::SimpleTable
    Just 20 lines to create:
    • Lambda function
    • IAM role
    • API Gateway
    • DynamoDB table
    O
    pen
    Source

    View Slide

  8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    SAM CLI
    https://github.com/awslabs/aws-sam-cli

    View Slide

  9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    SAM CLI
    https://github.com/awslabs/aws-sam-cli

    View Slide

  10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    SAM CLI
    sam init --name my-function --runtime python
    cd my-function/
    sam build
    sam package --s3-bucket my-packages-bucket \
    --output-template-file packaged.yaml
    sam deploy --template-file packaged.yaml \
    --stack-name my-function-prod
    sam publish # To the AWS Serverless Application Repository

    View Slide

  11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    TweetSource:
    Type: AWS::Serverless::Application
    Properties:
    Location:
    ApplicationId: arn:aws:serverlessrepo:...
    SemanticVersion: 2.0.0
    Parameters:
    TweetProcessorFunctionName: !Ref MyFunction
    SearchText: '#serverless -filter:nativeretweets'
    Nested apps to simplify solving recurring problems
    Standard
    Component
    Custom
    Business
    Logic
    Polling schedule
    (CloudWatch
    Events rule)
    trigger
    TwitterProcessor
    SearchCheckpoint
    TwitterSearchPoller
    Twitter
    Search API

    View Slide

  12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Serverless deployments
    Code
    Stack
    Package Deploy
    Template

    View Slide

  13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Serverless deployments with a test environment
    Feedback
    Loop
    Production
    Stack
    Deploy
    Code
    Test
    Stack
    Package Deploy
    Template

    View Slide

  14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    CodeDeploy – Lambda deployments
    Enable in your serverless application template
    Resources:
    GetFunction:
    Type: AWS::Serverless::Function
    Properties:
    DeploymentPreference:
    Type: Canary10Percent10Minutes
    Alarms:
    - !Ref ErrorsAlarm
    Hooks:
    PreTraffic: !Ref PreTrafficHook
    Canary10Percent30Minutes
    Canary10Percent5Minutes
    Canary10Percent10Minutes
    Canary10Percent15Minutes
    Linear10PercentEvery10Minutes
    Linear10PercentEvery1Minute
    Linear10PercentEvery2Minutes
    Linear10PercentEvery3Minutes
    AllAtOnce

    View Slide

  15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    CodeDeploy – Lambda canary deployment
    API
    Gateway
    Lambda
    function
    alias “live”
    v1
    Lambda
    function
    code
    100%

    View Slide

  16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    CodeDeploy – Lambda canary deployment
    API
    Gateway
    Lambda
    function
    weighted
    alias “live”
    v1 code
    100%
    Run PreTraffic hook against v2 code before it receives traffic
    v2 code
    0%

    View Slide

  17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    CodeDeploy – Lambda canary deployment
    API
    Gateway
    Lambda
    function
    weighted
    alias “live”
    v1 code
    90%
    Wait for 10 minutes, roll back in case of alarm
    v2 code
    10%

    View Slide

  18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    CodeDeploy – Lambda canary deployment
    API
    Gateway
    Lambda
    function
    weighted
    alias “live”
    v1 code
    0%
    Run PostTraffic hook and complete deployment
    v2 code
    100%

    View Slide

  19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    API Gateway canary stage
    API
    Gateway
    Production
    stage
    v1 code
    v2 code
    99.5%
    0.5% Canary
    stage

    View Slide

  20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    BUSINESS
    LOGIC
    LIB
    B
    Before
    BUSINESS
    LOGIC
    LIB
    A
    LIB
    B
    BUSINESS
    LOGIC
    LIB
    A
    LIB
    B
    BUSINESS
    LOGIC
    LIB
    A
    LIB
    B
    LIB
    A
    Use
    Lambda Layers
    for shared code that
    doesn’t change
    frequently

    View Slide

  21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    BUSINESS
    LOGIC
    BUSINESS
    LOGIC
    BUSINESS
    LOGIC
    BUSINESS
    LOGIC
    LIB A LIB B
    Use
    Lambda Layers
    for shared code that
    doesn’t change
    frequently
    Focus on your
    business logic and
    speed up function
    deployments
    After

    View Slide

  22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Project
    Product
    v1 v2 v3
    Customer
    needs

    View Slide

  23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Project
    Product
    Reach
    milestone
    Customer
    value
    Lifecycle
    costs
    Cost to reach
    milestone
    Backward
    looking
    Forward
    looking

    View Slide

  24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    “The process becomes the
    proxy for the result you want.
    You stop looking at outcomes
    and just make sure you’re
    doing the process right.”
    Jeff Bezos
    2016 Letter to Shareholders
    “Jeff, what does Day 2 look like?”
    That’s a question I just got at our most recent all-hands meeting. I’ve been reminding people that it’s Day 1 for a
    couple of decades. I work in an Amazon building named Day 1, and when I moved buildings, I took the name
    with me. I spend time thinking about this topic.
    “Day 2 is stasis. Followed by irrelevance. Followed by excruciating, painful decline. Followed by death. And
    that is why it is always Day 1.”
    To be sure, this kind of decline would happen in extreme slow motion. An established company might harvest
    Day 2 for decades, but the final result would still come.
    I’m interested in the question, how do you fend off Day 2? What are the techniques and tactics? How do you
    keep the vitality of Day 1, even inside a large organization?
    Such a question can’t have a simple answer. There will be many elements, multiple paths, and many traps. I don’t
    know the whole answer, but I may know bits of it. Here’s a starter pack of essentials for Day 1 defense: customer
    obsession, a skeptical view of proxies, the eager adoption of external trends, and high-velocity decision making.
    True Customer Obsession
    There are many ways to center a business. You can be competitor focused, you can be product focused, you can
    be technology focused, you can be business model focused, and there are more. But in my view, obsessive
    customer focus is by far the most protective of Day 1 vitality.
    Why? There are many advantages to a customer-centric approach, but here’s the big one: customers are always
    beautifully, wonderfully dissatisfied, even when they report being happy and business is great. Even when they
    don’t yet know it, customers want something better, and your desire to delight customers will drive you to invent
    on their behalf. No customer ever asked Amazon to create the Prime membership program, but it sure turns out
    they wanted it, and I could give you many such examples.
    Staying in Day 1 requires you to experiment patiently, accept failures, plant seeds, protect saplings, and double
    down when you see customer delight. A customer-obsessed culture best creates the conditions where all of that
    can happen.
    Resist Proxies
    As companies get larger and more complex, there’s a tendency to manage to proxies. This comes in many shapes
    and sizes, and it’s dangerous, subtle, and very Day 2.
    A common example is process as proxy. Good process serves you so you can serve customers. But if you’re not
    watchful, the process can become the thing. This can happen very easily in large organizations. The process
    becomes the proxy for the result you want. You stop looking at outcomes and just make sure you’re doing the
    process right. Gulp. It’s not that rare to hear a junior leader defend a bad outcome with something like, “Well, we
    followed the process.” A more experienced leader will use it as an opportunity to investigate and improve the
    process. The process is not the thing. It’s always worth asking, do we own the process or does the process own
    us? In a Day 2 company, you might find it’s the second.
    Another example: market research and customer surveys can become proxies for customers – something that’s
    especially dangerous when you’re inventing and designing products. “Fifty-five percent of beta testers report
    being satisfied with this feature. That is up from 47% in the first survey.” That’s hard to interpret and could
    unintentionally mislead.
    Resist proxies

    View Slide

  25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Feedback to business
    Feedback
    Loop
    Production
    Stack
    Deploy
    Code
    Test
    Stack
    Package Deploy
    Template
    Business

    View Slide

  26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Product
    Features
    Defects
    Risks
    Debts
    Product development
    Business
    Customers
    Security &
    Compliance
    Developers &
    Architects
    Avoid
    Overutilization

    View Slide

  27. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    View Slide

  28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Capital One – Credit Offers API serverless architecture
    Affiliates
    www.capitalone.com/
    credit-cards/prequalify
    AWS Cloud
    Capital One
    API Gateway
    VPC
    Lambda
    Function
    Traces Logs
    Production Support
    Command Center
    COAT
    Credit Offers API Team
    Lambda
    Function
    S3 Bucket
    TTL
    Third-Party
    API

    View Slide

  29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Capital One – Credit Offers API CI/CD pipeline
    Continuous Improvement, Continuous Delivery!
    GitHub LGTM Bot Jenkins AWS SAM
    S3 Bucket
    (Versioning)
    Lambda
    Function
    DeploymentType:
    dev: AllAtOnce
    qa: AllAtOnce
    qaw: AllAtOnce
    prod: Canary10Percent10Minutes
    prodw: Canary10Percent10Minutes
    canary5xxGetProductsAlarm:
    Type: AWS::CloudFormation::Alarm
    Properties:
    AlarmActions:
    - !FindInMap:
    - params
    - AdminSNSTopic
    - !Ref Environment
    AlarmDescription: 500 error from product
    listing Lambda.
    ComparisonOperator:
    GreatherThanOrEqualTothreshold
    Period: 300
    Statistic: Sum
    Threshold: 1
    EvaluationPeriod: 1

    View Slide

  30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Capital One – Benefits from taking the API serverless
    Performance gains
    From the time the request
    is received by lambda to
    the time to send the
    response back
    70%
    Cost savings
    By removing EC2, ELB and
    RDS from our solution
    90%
    Increase in team velocity
    Reduce investment in team’s time
    on DevOps and dedicate back to
    feature development!
    30%

    View Slide

  31. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

    View Slide

  32. Thank you!
    S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Danilo Poccia
    @danilop

    View Slide