Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AWS AppSync & Step Functions

Danilo Poccia
February 20, 2019

AWS AppSync & Step Functions

AWS User Group Netherlands + J-AWS, Haarlem, February 20th, 2019

The latest updates on AWS AppSync & Step Functions.
From frontend to backend, there and back again!

Danilo Poccia

February 20, 2019
Tweet

More Decks by Danilo Poccia

Other Decks in Programming

Transcript

  1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
    Danilo Poccia
    Principal Evangelist, Serverless
    @danilop
    AWS AppSync & Step Functions
    From frontend to backend, there and back again

    View Slide

  2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
    © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
    https://aws.amazon.com/events/summits/amsterdam/

    View Slide

  3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    What is GraphQL?
    GraphQL is a query language for APIs and a runtime for fulfilling
    those queries with your existing data
    Traditional data-fetching GraphQL
    /posts
    /postInfo
    /postJustTitle
    /postsByAuthor
    /postNameStartsWithX
    /commentsOnPost

    View Slide

  4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    How does GraphQL work?
    {
    "id": "1",
    "name": "Get Milk",
    "priority": "1"
    },
    {
    "id": "2",
    "name": "Go to gym",
    "priority": "5"
    },…
    type Query {
    getTodos: [Todo]
    }
    type Todo {
    id: ID!
    name: String
    description: String
    priority: Int
    duedate: String
    }
    query {
    getTodos {
    id
    name
    priority
    }
    }
    Model data with
    application schema
    Client requests what it
    needs
    Only that data is
    returned

    View Slide

  5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Is REST dead then?
    • When data drives UI
    • Structured data
    • Query-driven
    • Client-driven
    development
    Use GraphQL
    • When you leverage HTTP
    • Caching, Content types
    • Hypermedia (HATEOAS)
    • For resources
    (e.g., Amazon S3)
    Use REST
    Pick the appropriate protocol for your use case

    View Slide

  6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    GraphQL
    Schema
    type Event {
    id: ID!
    name: String
    where: String
    when: String
    description: String
    comments: [Comment]
    }
    type Comment {
    commentId: String!
    eventId: ID!
    content: String!
    createdAt: String!
    }

    View Slide

  7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    GraphQL
    Schema
    Mutation
    type Mutation {
    createEvent(
    name: String!,
    when: String!,
    where: String!,
    description: String!
    ): Event
    deleteEvent(id: ID!): Event
    commentOnEvent(
    eventId: ID!,
    content: String!,
    createdAt: String!
    ): Comment
    }

    View Slide

  8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    GraphQL
    Schema
    Mutation
    Query
    type Query {
    getEvent(id: ID!): Event
    listEvents(
    filter: TableEventFilterInput,
    limit: Int = 10,
    nextToken: String
    ): EventConnection
    }

    View Slide

  9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    GraphQL
    Schema
    Mutation
    Query
    Subscription
    type Subscription {
    subscribeToEventComments(eventId: String!): Comment
    @aws_subscribe(mutations: ["commentOnEvent"])
    }

    View Slide

  10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    GraphQL
    Schema
    Mutation
    Query
    Subscription
    Realtime? YES
    Batching? YES
    Pagination? YES
    Relations? YES
    Aggregations? YES
    Search? YES
    Offline? YES

    View Slide

  11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Introducing AWS AppSync
    AWS AppSync is a managed service for application data
    using GraphQL with real-time capabilities and an offline
    programming model
    Real-time
    collaboration
    Offline programming
    model with sync
    Flexible database
    options
    Fine-grained
    access control

    View Slide

  12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    AWS AppSync
    DynamoDB
    Table
    Lambda
    Function
    GraphQL
    Schema
    Upload
    Schema
    GraphQL
    Query
    Mutation
    Subscription
    Real-time
    Offline
    Users
    AppSync
    API
    Data
    Sources
    Resolvers
    Pipelines / Functions
    Velocity
    Templates (VTL) Auth
    Cognito
    User Pool
    AWS
    IAM
    API
    Key
    OpenID
    Connect
    Elasticsearch
    Service
    HTTP
    Endpoint
    Relational
    Database

    View Slide

  13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    AWS AppSync
    DynamoDB
    Table
    Lambda
    Function
    GraphQL
    Schema
    Upload
    Schema
    GraphQL
    Query
    Mutation
    Subscription
    Real-time
    Offline
    Users
    AppSync
    API
    Data
    Sources
    Auth
    Cognito
    User Pool
    AWS
    IAM
    API
    Key
    OpenID
    Connect
    Create
    DynamoDB
    Table
    Create
    Resource
    Elasticsearch
    Service
    Resolvers
    Pipelines / Functions
    Velocity
    Templates (VTL)
    HTTP
    Endpoint
    Relational
    Database

    View Slide

  14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    AWS AppSync
    DynamoDB
    Table
    Lambda
    Function
    GraphQL
    Schema
    Upload
    Schema
    GraphQL
    Query
    Mutation
    Subscription
    Real-time
    Offline
    Users
    AppSync
    API
    Data
    Sources
    Auth
    Cognito
    User Pool
    AWS
    IAM
    API
    Key
    OpenID
    Connect
    Autogenerate
    Schema &
    Resolvers
    Im
    port a
    D
    ynam
    oD
    B
    Table
    Elasticsearch
    Service
    Resolvers
    Pipelines / Functions
    Velocity
    Templates (VTL)
    HTTP
    Endpoint
    Relational
    Database

    View Slide

  15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    AWS AppSync
    DynamoDB
    Table
    Lambda
    Function
    GraphQL
    Schema
    Upload
    Schema
    GraphQL
    Query
    Mutation
    Subscription
    Real-time
    Offline
    Users
    AppSync
    API
    Data
    Sources
    Auth
    Cognito
    User Pool
    AWS
    IAM
    API
    Key
    OpenID
    Connect
    Other
    Databases
    O
    ther
    D
    atabases
    Elasticsearch
    Service
    Resolvers
    Pipelines / Functions
    Velocity
    Templates (VTL)
    HTTP
    Endpoint
    Relational
    Database

    View Slide

  16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    AWS AppSync
    DynamoDB
    Table
    Lambda
    Function
    GraphQL
    Schema
    Upload
    Schema
    GraphQL
    Query
    Mutation
    Subscription
    Real-time
    Offline
    Users
    AppSync
    API
    Data
    Sources
    Auth
    Cognito
    User Pool
    AWS
    IAM
    API
    Key
    OpenID
    Connect
    Legacy
    Applications
    Legacy
    Application
    Elasticsearch
    Service
    Resolvers
    Pipelines / Functions
    Velocity
    Templates (VTL)
    HTTP
    Endpoint
    Relational
    Database

    View Slide

  17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    AWS AppSync
    DynamoDB
    Table
    Lambda
    Function
    Elasticsearch
    Service
    GraphQL
    Schema
    Upload
    Schema
    GraphQL
    Query
    Mutation
    Subscription
    Real-time
    Offline
    Users
    AppSync
    API
    Data
    Sources
    Auth
    Cognito
    User Pool
    AWS
    IAM
    API
    Key
    OpenID
    Connect
    Com
    plex
    Q
    ueries
    DynamoDB to Elasticsearch
    Sync Function
    Resolvers
    Pipelines / Functions
    Velocity
    Templates (VTL)
    HTTP
    Endpoint
    Relational
    Database

    View Slide

  18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    M
    usixm
    atch
    AW
    S
    Sum
    m
    it
    M
    ilan
    2018

    View Slide

  19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    AWS Amplify
    Amplify your apps.
    Build on a flexible, scalable, and reliable serverless backend.
    Easy-to-Use Library
    Choose your cloud services
    and easily connect them to
    your app with just a few
    lines of code.
    Powerful Toolchain
    Effortlessly create and
    maintain sophisticated
    serverless backends for
    your apps.
    Beautiful UI Components
    Accelerate app development
    by leveraging our beautiful
    out-of-the box UI
    components.
    O
    pen
    Source

    View Slide

  20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    AW
    S
    Am
    plify
    CLI

    View Slide

  21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    AW
    S
    Am
    plify
    CLI
    Effortlessly configure
    backends
    Hosting for static assets
    and websites
    Easily create APIs and
    generate client code
    $ cd
    $ amplify init
    $ amplify add
    $ amplify push
    $ amplify add hosting
    $ amplify publish
    $ amplify codegen add
    $ amplify codegen generate

    View Slide

  22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    AW
    S
    Am
    plify
    Console

    View Slide

  23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    AW
    S
    Am
    plify
    Console

    View Slide

  24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    GraphQL Sample Schema
    type Note {
    id: ID!
    title: String!
    content: String!
    }
    type Query {
    getNote(id: ID!): Note
    listNotes: [Note]
    }
    type Mutation {
    createNote(title: String!, content: String!): Note
    updateNote(id: ID!, title: String!, content: String!): Note
    deleteNote(id: ID!): Note
    }
    schema {
    query: Query
    mutation: Mutation
    }

    View Slide

  25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    GraphQL Transformers
    type Note @model {
    id: ID!
    title: String!
    content: String!
    }
    AW
    S
    Am
    plify

    View Slide

  26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    GraphQL Transformers
    type Note
    @model
    @auth(rules: [ { allow: owner } ])
    @searchable
    @versioned
    {
    id: ID!
    title: String!
    content: String!
    }
    AW
    S
    Am
    plify

    View Slide

  27. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
    AWS AppSync & Amplify Demo

    View Slide

  28. λ
    λ
    λ
    DBMS
    λ
    λ
    λ
    λ
    λ
    λ λ
    λ
    λ
    Queue
    Modern
    serverless
    app

    View Slide

  29. Modern
    serverless
    app

    View Slide

  30. "I want to sequence functions"
    "I want to select functions based on data"
    "I want to retry functions"
    "I want try/catch/finally"
    Functions into apps
    "I have code that runs for hours"
    "I want to run functions in parallel"

    View Slide

  31. Coordination must-haves
    • Scales out
    • Doesn’t lose state
    • Deals with errors/timeouts
    • Easy to build & operate
    • Auditable

    View Slide

  32. Application Lifecycle in AWS Step Functions
    Visualize in the
    Console
    Define in JSON Monitor
    Executions

    View Slide

  33. Define in JSON and Then Visualize in the Console
    {
    "Comment": "Hello World Example",
    "StartAt" : "HelloWorld",
    "States" : {
    "HelloWorld" : {
    "Type" : "Task",
    "Resource" : "${lambdaArn}",
    "End" : true
    }
    }
    }

    View Slide

  34. Execute One or One Million
    Start
    End
    HelloWorld
    Start
    End
    HelloWorld
    Start
    End
    HelloWorld
    Start
    End
    HelloWorld
    Start
    End
    HelloWorld
    Start
    End
    HelloWorld
    Start
    End
    HelloWorld
    Start
    End
    HelloWorld

    View Slide

  35. Maximum Execution Time
    AWS Lambda
    Functions
    15 minutes
    AWS Step Functions
    State Machines
    1 year

    View Slide

  36. Monitor Executions from the Console

    View Slide

  37. Seven State Types
    Task A single unit of work
    Choice Adds branching logic
    Parallel Fork and join the data across tasks
    Wait Delay for a specified time
    Fail Stops an execution and marks it as a failure
    Succeed Stops an execution successfully
    Pass Passes its input to its output

    View Slide

  38. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    ExtractImageMetadata
    ImageTypeCheck
    TransformMetadata
    Rekognition Thumbnail
    StoreMetadata
    Start
    End
    Synchronous task
    Task: a single unit of work
    Synchronous: Lambda
    {
    "ExtractImageMetadata": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:mars-ter …",

    }
    }

    View Slide

  39. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Step Functions Tasks – Service Integrations
    AWS Lambda invoke a Lambda function
    AWS Batch submit a Batch job and wait for it to complete
    Amazon DynamoDB get, put, update or delete an item
    Amazon ECS/Fargate run an ECS task and waits for it to complete
    Amazon SNS publish a message to a SNS topic
    Amazon SQS send a SQS message
    AWS Glue start a Glue job
    Amazon SageMaker create a training or transform job

    View Slide

  40. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    ExtractImageMetadata
    TransformMetadata
    Rekognition Thumbnail
    StoreMetadata
    Start
    End
    Task: a single unit of work
    Asynchronous: "Activity Task", Polled by workers
    {
    "ExtractImageMetadata": {
    "Type": "Task",
    "Resource": "arn:aws:states:mars-ter …",

    }
    }
    Asynchronous task

    View Slide

  41. API actions on activity task
    Register Activity Task - Returns ARN
    Poll For task (by ARN)
    Report Success
    Report Failure
    Report Heartbeat

    View Slide

  42. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    O
    pen
    Source
    A
    pache
    License
    https://states-language.net/spec.html

    View Slide

  43. Build Visual Workflows from State Types
    Task
    Choice
    Fail
    Parallel

    View Slide

  44. Image Recognition and Processing Backend
    Task
    Choice
    Fail
    Parallel
    https://github.com/awslabs/lambda-refarch-imagerecognition

    View Slide

  45. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
    AWS Step Functions Demo

    View Slide

  46. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

    View Slide

  47. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
    AWS Step Functions Case Studies

    View Slide

  48. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Vending Pass problem
    1. Thirsty consumer presents
    card, can buy a drink with
    Vending Pass or debit
    2. But mobile wallet display can
    get out of sync under certain
    conditions
    3.

    View Slide

  49. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Vending Pass problem
    • Just wait for backend
    to catch up
    • Create a two-step state
    machine:
    • Wait (90 seconds)
    • Update mobile wallet
    • Cheap and simple!

    View Slide

  50. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    • Transcode 350 clips/day into
    14 formats, fast
    • It’s all done with FFmpeg. The
    processing time is just about
    100% of the video length
    • Aargh!
    Video processing problem

    View Slide

  51. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    • Derive keyframe locations
    within the source
    • Split the source at the
    keyframes
    • Process segments (typically
    0.5 sec per) in parallel
    • Concatenate segments
    • Elapsed time: ~20 min down
    to ~2 minutes
    Video processing solution

    View Slide

  52. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Details
    How to get from Lambda to Amazon S3:
    ffmpeg -f concat –safe 0 -I filelist.ffcat –c copy pipe:0 |
    aws s3 cp – s3://output-bucket/output-file.mp4

    View Slide

  53. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Details

    View Slide

  54. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Let’s peek at the code!
    "Init": {
    "Type": "Pass",
    "ResultPath": "$.operation",
    "Result": "prepare-split",
    "Next": "PrepareSplit"
    },
    "PrepareSplit": {
    "Type": "Task",
    "Resource": "arn:…:ffmpeg-18HFP9FXFL6P",
    "Next": "SplitPrepared"
    },
    "SplitPrepared": {
    "Type": "Pass",
    "ResultPath": "$.operation",
    "Result": "perform-split",
    "Next": "PerformSplit"
    },
    "PerformSplit": {
    "Type": "Task",
    "Resource": "arn:…:ffmpeg-18HFP9FXFL6P",
    "Next": "SplitPerformed"
    },
    "SplitPerformed": {
    "Type": "Pass",
    "ResultPath": "$.operation",
    "Result": "poll-results",
    "Next": "PreparePoll"
    },
    "PreparePoll": {
    "Type": "Pass",
    "ResultPath": "$.pollstatus",
    "Result": "Submitted",
    "Next": "WaitForResults"
    }
    Hmm…the same function,
    throughout

    View Slide

  55. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Let’s peek at the code!
    "WaitForResults": {
    "Type": "Choice",
    "Choices": [
    {
    "Variable": "$.pollstatus",
    "StringEquals": "Complete",
    "Next": "SegmentsCompleteState"
    },
    {
    "Variable": "$.pollstatus",
    "StringEquals": "Progressing",
    "Next": "loop_wait_using_seconds"
    },
    {
    "Variable": "$.pollstatus",
    "StringEquals": "Submitted",
    "Next": "loop_wait_using_seconds"
    }
    ]
    }
    "loop_wait_using_seconds": {
    "Type": "Wait",
    "Seconds": 3,
    "Next": "loop"
    },
    "loop": {
    "Type": "Task",
    "Resource": "arn:…:ffmpeg-18HFP9FXFL6P",
    "Next": "WaitForResults"
    },
    "SegmentsCompleteState": {
    "Type": "Pass",
    "ResultPath": "$.operation",
    "Result": "concat",
    "Next": "CompleteState"
    },
    "CompleteState": {
    "Type": "Task",
    "Resource": " arn:…:ffmpeg-18HFP9FXFL6P",
    "End": true
    }
    Counts the segments,
    sets $.pollstatus
    Stitches segments together

    View Slide

  56. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Takeaways
    Use Amazon S3
    data events to
    trigger parallel
    Lambda
    processing: win
    Use Amazon S3
    URLs to stream
    video to
    Lambda: win
    Scaling to 1,000
    Lambdas, rather
    than 1,000 EC2
    instances: win
    Process video
    segments in
    parallel: win

    View Slide

  57. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    More state machines
    Image Recognition and Processing Backend
    EBS Snapshot Management
    https://aws.amazon.com/step-functions/getting-started

    View Slide

  58. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    What are you building? #stepfunctions
    https://console.aws.amazon.com/states/

    View Slide

  59. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
    Thank you!
    @danilop

    View Slide