Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Serverless for Developers: Tips for Your Next App

Serverless for Developers: Tips for Your Next App

Serverless Meetup, Milan, November 3rd, 2016

Serverless architectures simplify operations and maintenance of production applications, abstracting from the underlining infrastructure and OS. But what about development? We are now two years in with tools such as AWS Lambda and what are the best practices to share from a development perspective? In this session you will see some of the key points that can empower a serverless developer: from event-driven design, to simplification (less is more, especially for a small team), extending the functionalities of the platform (per prefix stats for S3 buckets, geohash for DynamoDB tables, API authentication, and so on), integrating new products.

Danilo Poccia

November 03, 2016
Tweet

More Decks by Danilo Poccia

Other Decks in Programming

Transcript

  1. Serverless for Developers:
    Tips for Your Next App
    Danilo Poccia, Technical Evangelist
    @danilop
    danilop

    View Slide

  2. View Slide

  3. Credit: Gerry Cranham/Fox Photos/Getty Images
    http://www.telegraph.co.uk/travel/destinations/europe/united-kingdom/england/london/galleries/The-history-of-the-Tube-in-pictures-150-years-of-London-Underground/1939-ticket-examin/

    View Slide

  4. Credit: Gerry Cranham/Fox Photos/Getty Images
    http://www.telegraph.co.uk/travel/destinations/europe/united-kingdom/england/london/galleries/The-history-of-the-Tube-in-pictures-150-years-of-London-Underground/1939-ticket-examin/
    1939 London Underground

    View Slide

  5. AWS
    IoT
    Kinesis
    Stream
    AWS
    Lambda
    DynamoDB
    Table
    S3
    Bucket

    View Slide

  6. Virtual Machines Containers Functions

    View Slide

  7. Code Microseconds
    to Seconds
    Versioning Single-Threaded
    Single-Task
    Per Memory/Second
    Per Request
    Free Tier
    The World of Lambda Functions

    View Slide

  8. No server is easier to
    manage than no server
    Serverless Computing

    View Slide

  9. Low Cost
    Security
    Simplicity Reliability
    Scalability Low Latency
    S3: Simple By Name, Simple By Nature

    View Slide

  10. Mobile Apps
    Websites
    Services
    Internet
    API Gateway Cache
    API Gateway
    CloudWatch
    Lambda Functions
    EC2 Endpoints
    Any Other Endpoint
    Amazon API Gateway

    View Slide

  11. Building An Infinitely Scalable Online Recording Campaign
    For David Guetta & UEFA

    View Slide

  12. Dev + Ops

    View Slide

  13. Dev + Ops
    + Event Driven

    View Slide

  14. Name
    Email
    Create New User
    Submit
    Check the “Name” syntax is right
    (only letters and spaces)
    Check the “Email” syntax is right
    ([email protected])
    Create the new user using
    provided Name and Email
    User Interface Actions

    View Slide

  15. Name
    Email
    Create New User
    Submit
    Check the “Name” syntax is right
    (only letters and spaces)
    Check the “Email” syntax is right
    ([email protected])
    Create the new user using
    provided Name and Email
    User Interface Actions
    Observers
    Target Action

    View Slide

  16. Generate thumbnails
    Keep another database in sync
    Process streaming data
    Services Functions
    Platform
    Target Action
    Kinesis
    Stream
    DynamoDB
    Table
    S3
    Bucket

    View Slide

  17. area = length x width

    View Slide

  18. area = length x width
    Procedural Programming
    “This is a function!”

    View Slide

  19. area = length x width
    Procedural Programming
    “This is a function!”
    Reactive Programming
    “This is data binding!”

    View Slide

  20. A B
    Event-Driven Design
    For Services
    A triggers B
    or better
    B is caused by A

    View Slide

  21. A B
    C
    D
    New Services C and D
    are triggered caused by B

    View Slide

  22. A B
    C
    D
    E
    F
    ?
    ?
    What is causing E and F?

    View Slide

  23. A B
    C
    D
    E
    F
    New Services (E, F)
    are triggered by C

    View Slide

  24. A B
    C
    D
    E
    F
    It can be cyclic
    Think of
    acknowledgements

    View Slide

  25. A B
    C
    D
    E
    F
    Each service
    has local visibility

    View Slide

  26. A B
    C
    D
    E
    F
    What I need to know
    (input events)
    1

    View Slide

  27. A B
    C
    D
    E
    F
    What I need to do
    (internal logic)
    2

    View Slide

  28. A B
    C
    D
    E
    F
    Who I need to notify
    (output events)
    3

    View Slide

  29. A B
    C
    D
    E
    F
    Who I need to notify
    (output events)
    What I need to know
    (input events)
    What I need to do
    (internal logic)
    1
    2
    3

    View Slide

  30. Distributed Data Flow
    Vs
    Centralised Workflow

    View Slide

  31. Choreography
    Vs
    Orchestration

    View Slide

  32. Distributed Systems

    View Slide

  33. Monolith
    Service
    Service
    μ
    μ
    μ
    space (distribution, mobility)
    time (concurrency, latency)
    μ
    μ
    μ
    μ
    “Distributed
    in Time and Space”
    [Jonas Bonér]

    View Slide

  34. “…a diagram of
    two microservices
    and their shared database”
    Data should drive
    the boundaries

    View Slide

  35. A
    C
    I
    D
    tomic
    onsistent
    isolated
    urable

    View Slide

  36. A
    C
    I
    D
    tomic
    onsistent
    isolated
    urable
    A
    C
    I
    D
    ssociative
    ommutative
    dempotent
    istributed
    ACID 2.0

    View Slide

  37. A B
    C
    D
    E
    F
    Event-Driven Interactions replacing Distributed Transactions
    using Eventual Consistency

    View Slide

  38. A B
    C
    D
    E
    F
    Event-Driven Interactions replacing Distributed Transactions
    using Eventual Consistency
    Differentiate reads (queries)
    from writes (commands),
    as in the Command Query
    Responsibility Segregation
    (CQRS) pattern
    command query

    View Slide

  39. Customers Orders
    Warehouse Deliveries
    Payments
    Catalog
    Event-Driven Interactions replacing Distributed Transactions
    using Eventual Consistency

    View Slide

  40. Scale
    from Prototype to Production

    View Slide

  41. Responsive
    Resilient
    Elastic
    Message Driven
    The Reactive Manifesto

    View Slide

  42. The Reactive Manifesto
    Asynchronous
    communication
    is not enforced
    by serverless architectures
    Responsive
    Resilient
    Elastic
    Message Driven

    View Slide

  43. Serverless Architectural Patterns

    View Slide

  44. AWS
    Lambda
    User Client
    Application
    Back End for a Web or Mobile App
    Calling the Lambda API directly
    Web or Mobile Back End
    DynamoDB
    Table
    S3
    Bucket
    Amazon
    Cognito
    AWS API

    View Slide

  45. AWS
    Lambda
    Amazon
    Cognito
    User
    Client
    Application
    OAuth JWT
    Amazon
    API Gateway
    Back End for a Web or Mobile App
    Using an API Gateway for authentication and decoupling
    Web or Mobile Back End
    Via API Gateway
    DynamoDB
    Table
    S3
    Bucket
    HTTPS

    View Slide

  46. Using WebSockets instead of Web API to talk to the Back End
    WebSockets Back End
    AWS
    Lambda
    User
    Client
    Application
    AWS
    IoT
    DynamoDB
    Table
    S3
    Bucket
    Amazon
    Cognito
    MQTT or
    Device
    Shadow
    MQTT over

    WebSockets

    View Slide

  47. Server ! Client
    {
    "id": "...",
    "html": "..."
    }
    {
    "js": "..."
    }
    Client ! Server
    {
    "cmd": "...",
    "data": "..."
    }
    MQTT Topics
    myapp/client/in/{ID}
    myapp/client/out/{ID}
    Cognito Identity ID ! {ID}
    1
    2
    3
    4
    AWS
    Lambda
    User
    Client
    Application
    AWS
    IoT
    DynamoDB
    Table
    S3
    Bucket
    Amazon
    Cognito
    MQTT or
    Device
    Shadow
    MQTT over

    WebSockets

    View Slide

  48. Possible optimisations depending on your data flow
    and your acceptable latency
    Data
    Source
    Kinesis
    Stream
    Kinesis
    Firehose
    Kinesis
    Analytics
    (augment
    & aggregate)
    AWS
    Lambda
    (batch)
    S3
    Bucket
    AWS
    Lambda
    (file)
    Data Streams
    Processing
    Reference
    Data

    View Slide

  49. AWS
    Lambda
    (batch)
    Amazon
    Cognito
    User
    Client
    Application
    Buffering Requests to the Back End to reduce Lambda invocations
    Kinesis
    Stream
    Buffered
    Back End For Mobile
    Amazon
    SNS
    DynamoDB
    Table
    S3
    Bucket

    View Slide

  50. AWS
    Lambda
    (batch)
    Amazon
    Cognito
    User
    Client
    Application
    Buffering Requests to the Back End to reduce Lambda invocations
    Kinesis
    Stream
    Buffered & Aggregated

    Back End For Mobile
    Amazon
    SNS
    DynamoDB
    Table
    S3
    Bucket
    Kinesis
    Analytics
    (aggregate)
    Kinesis
    Stream

    View Slide

  51. AWS
    Lambda
    (batch)
    User
    Client
    Application
    Using WebSockets instead of Web API to talk to the Back End
    Buffering Requests to reduce Lambda invocations
    AWS
    IoT
    Kinesis
    Stream
    Buffered Back End
    For Everything
    MQTT or
    Device Shadow
    Amazon
    Cognito
    DynamoDB
    Table
    S3
    Bucket

    View Slide

  52. User
    Client
    Application
    Using WebSockets instead of Web API to talk to the Back End
    Buffering Requests to reduce Lambda invocations
    Storing all customer interactions for further analysis
    AWS
    IoT
    Kinesis
    Firehose
    Buffered Back End
    Saving Interactions
    MQTT or
    Device Shadow
    Amazon
    Cognito
    AWS
    Lambda
    (file)
    S3
    Bucket
    DynamoDB
    Table

    View Slide

  53. AWS
    Lambda
    DynamoDB
    Table
    User
    Client
    Application
    Amazon
    API Gateway
    (HTTPS)
    S3 Bucket
    (Website Hosting
    Routing Rules)
    For example, to generate image thumbnails on the first request,
    S3 LifeCycle Rules can delete content after N days
    Dynamic Cache
    Amazon
    Cognito
    HTTP
    Redirect
    Dynamically
    Build Content

    View Slide

  54. Content Delivery
    Amazon
    API Gateway
    CloudFront
    Distribution
    S3 Bucket
    AWS
    Lambda
    DynamoDB
    Table
    S3
    Bucket
    Single DNS domain for both static and dynamic content, protected by the WAF,

    DNS health checks can react automatically to service downtime
    User
    Client
    Application
    Amazon
    Route 53
    DNS
    Resolution
    Health
    Checks
    AWS
    WAF

    View Slide

  55. Data Analytics
    S3
    Bucket
    Redshift
    Database
    DynamoDB
    Table #N
    AWS
    Lambda
    Kinesis
    Firehose
    Synchronize operational NoSQL databases

    with a columnar store for business intelligence and analytics
    Business
    Intelligence
    DynamoDB
    Table #1
    AWS
    Lambda
    Kinesis
    Firehose
    .
    .
    .
    .
    .
    .
    .
    .
    .

    View Slide

  56. To receive events from everywhere on the Internet
    Webhooks are callbacks for the Web
    AWS
    Lambda
    Amazon
    API Gateway
    Internet
    Service
    HTTP(S) GET or POST to https://domain/SOURCE/RANDOM-HOOK
    Webhook
    Other AWS
    Services
    Other public
    APIs
    AWS
    KMS
    Secrets &
    Credentials

    View Slide

  57. To generate events
    for repositories
    that don’t support
    AWS Lambda natively
    Log
    Monitor
    AWS
    Lambda
    AWS
    Lambda
    AWS
    Lambda
    Amazon
    SNS
    Kinesis
    Stream
    Log
    Files
    External
    Application
    Log Monitor
    CloudWatch
    Logs or Metrics
    AWS
    Lambda

    View Slide

  58. S3 object auto compress
    S3 bucket per prefix statistics
    DynamoDB provisioned throughput dynamic scaling
    DynamoDB item support for geohash
    Relational / NoSQL data synchronisation
    MapReduce with Lambda functions on S3 buckets
    Service discovery
    . . .
    Extending Services with Your Own Functionalities

    View Slide

  59. Serverless === Architecture Simplification

    View Slide

  60. Serverless === Architecture Simplification
    For example, let’s build a Media Sharing App

    View Slide

  61. Client
    Application
    Get Content
    Index
    Get Content
    and Thumbnails
    Upload Content
    with Metadata
    Update Content
    Metadata
    Get Content
    Metadata
    Files
    (Multimedia)
    Database
    Build
    Thumbnails
    Update
    Content Index
    Extract
    and Update
    Metadata
    Functions
    User
    Resources
    Basic
    features
    M
    edia
    Sharing
    App

    View Slide

  62. Client
    Application
    Get Content
    Index
    Get Content
    and Thumbnails
    Upload Content
    with Metadata
    Update Content
    Metadata
    Get Content
    Metadata
    Files
    (Multimedia)
    Database
    Build
    Thumbnails
    Update
    Content Index
    Extract
    and Update
    Metadata
    Event:
    new or
    updated
    file
    Event:
    new or updated
    content metadata
    Functions
    User
    Resources
    Event-driven
    design
    M
    edia
    Sharing
    App

    View Slide

  63. Client
    Application
    Get Content
    Index
    Get Content
    and Thumbnails
    Upload Content
    with Metadata
    Update Content
    Metadata
    Get Content
    Metadata
    Files
    (Multimedia)
    Database
    Build
    Thumbnails
    Update
    Content Index
    Extract
    and Update
    Metadata
    Event:
    new or
    updated
    file
    Event:
    new or updated
    content metadata
    Functions
    User
    Resources
    Understanding
    the overall flow
    M
    edia
    Sharing
    App

    View Slide

  64. Client
    Application Get Content Index
    (S3 API)
    Get Content
    and Thumbnails
    (S3 API)
    Upload Content

    with Metadata
    (S3 API)
    Update Content
    Metadata
    (DynamoDB API)
    Get Content Metadata
    (DynamoDB API)
    S3
    Bucket
    DynamoDB
    Table
    Build Thumbnails
    (Lambda function)
    Update
    Content Index
    (Lambda function)
    Extract and Update
    Metadata
    (Lambda function)
    Event:
    new or
    updated
    file
    Event:
    new or updated
    content metadata
    User
    Choosing
    the technology
    M
    edia
    Sharing
    App

    View Slide

  65. Client
    Application Get Content Index
    (S3 API)
    Get Content
    and Thumbnails
    (S3 API)
    Upload Content

    with Metadata
    (S3 API)
    Update Content
    Metadata
    (DynamoDB API)
    Get Content Metadata

    (DynamoDB API)
    S3
    Bucket
    DynamoDB
    Table
    Build Thumbnails
    (Lambda function)
    Update
    Content Index
    (Lambda function)
    Extract and Update
    Metadata
    (Lambda function)
    Event:
    new or
    updated
    file
    Event:
    new or updated
    content metadata
    User
    Amazon
    Cognito
    Defining
    the security model
    M
    edia
    Sharing
    App

    View Slide

  66. Client
    Application Get Content Index
    (S3 API)
    Get Content
    and Thumbnails
    (S3 API)
    Upload Content

    with Metadata
    (S3 API)
    Update Content
    Metadata
    (DynamoDB API)
    Get Content Metadata

    (DynamoDB API)
    S3
    Bucket
    DynamoDB
    Table
    Build Thumbnails
    (Lambda function)
    Update
    Content Index
    (Lambda function)
    Extract and Update
    Metadata
    (Lambda function)
    Event:
    new or
    updated
    file
    Event:
    new or updated
    content metadata
    User
    Amazon
    Cognito
    From the features
    viewpoint…
    M
    edia
    Sharing
    App

    View Slide

  67. Client
    Application
    GET Object
    S3 API
    PUT Object
    S3 API
    UpdateItem
    DynamoDB API
    GetItem

    DynamoDB API
    S3
    Bucket
    DynamoDB
    Table
    buildThumbnails
    (Lambda function)
    updateContentIndex
    (Lambda function)
    extractAndUpdateMetadata
    (Lambda function)
    Event:
    new or
    updated
    file
    Event:
    new or updated
    content metadata
    User
    Amazon
    Cognito
    Amazon S3
    Amazon DynamoDB
    AWS Lambda
    …To the technology
    viewpoint
    M
    edia
    Sharing
    App

    View Slide

  68. Client
    Application
    GET Object
    S3 API
    PUT Object
    S3 API
    UpdateItem
    DynamoDB API
    GetItem

    DynamoDB API
    S3
    Bucket
    DynamoDB
    Table
    buildThumbnails
    (Lambda function)
    updateContentIndex
    (Lambda function)
    extractAndUpdateMetadata
    (Lambda function)
    Event:
    new or
    updated
    file
    Event:
    new or updated
    content metadata
    User
    Amazon
    Cognito
    Amazon S3
    Amazon DynamoDB
    AWS Lambda
    Some events
    are correlated
    M
    edia
    Sharing
    App

    View Slide

  69. Client
    Application
    GET Object
    S3 API
    PUT Object
    S3 API
    UpdateItem
    DynamoDB API
    GetItem

    DynamoDB API
    S3
    Bucket
    DynamoDB
    Table
    contentUpdated
    (Lambda function)
    buildThumbnails
    extractAndUpdateMetadata
    updateContentIndex
    (Lambda function)
    Event:
    new or
    updated
    file
    User
    Amazon
    Cognito
    Amazon S3 AWS Lambda
    Amazon DynamoDB
    Event:
    new or updated
    content metadata
    Some functions
    can be tied together
    M
    edia
    Sharing
    App

    View Slide

  70. Client
    Application
    GET Object
    S3 API
    PUT Object
    S3 API
    UpdateItem
    DynamoDB API
    GetItem

    DynamoDB API
    S3
    Bucket
    DynamoDB
    Table
    contentUpdated
    (Lambda function)
    buildThumbnails
    extractAndUpdateMetadata
    updateContentIndex
    (Lambda function)
    Event:
    new or
    updated
    file
    User
    Amazon
    Cognito
    Amazon S3 AWS Lambda
    Amazon DynamoDB
    Event:
    new or updated
    content metadata
    What if I want to
    delete content?
    M
    edia
    Sharing
    App

    View Slide

  71. Client
    Application
    GET Object
    S3 API
    PUT Object
    S3 API
    UpdateItem
    DynamoDB API
    GetItem

    DynamoDB API
    S3
    Bucket
    DynamoDB
    Table
    contentUpdated
    (Lambda function)
    buildThumbnails
    extractAndUpdateMetadata
    deleteMetadata
    updateContentIndex
    (Lambda function)
    Event:
    new, updated,
    or deleted
    file
    User
    Amazon
    Cognito
    Amazon S3 AWS Lambda
    Amazon DynamoDB
    DELETE Object
    S3 API
    Event:
    new or updated
    content metadata
    REST to the rescue,
    the events flow is
    almost unchanged
    M
    edia
    Sharing
    App

    View Slide

  72. Client
    Application
    GET Object
    S3 API
    PUT Object
    S3 API
    UpdateItem
    DynamoDB API
    GetItem

    DynamoDB API
    S3
    Bucket
    DynamoDB
    Table
    contentUpdated
    (Lambda function)
    buildThumbnails
    extractAndUpdateMetadata
    deleteMetadata
    updateContentIndex
    (Lambda function)
    Event:
    new, updated,
    or deleted
    file
    User
    Amazon
    Cognito
    Amazon S3 AWS Lambda
    Amazon DynamoDB
    DELETE Object
    S3 API
    Event:
    new or updated
    content metadata
    From 8 functions to 2,
    using 5 native
    API calls
    M
    edia
    Sharing
    App

    View Slide

  73. Serverless Architectures are not just simplifying Operations

    View Slide

  74. Developers can benefit from:
    event-driven design
    architectural patterns
    integrating new products
    architecture simplification
    customization of the platform
    Serverless Architectures are not just simplifying Operations

    View Slide

  75. Build Apps With Services,
    Not Servers

    View Slide

  76. Thank you
    @danilop
    danilop

    View Slide