AWS AppSync & Step Functions

AWS User Group Netherlands + J-AWS, Haarlem, February 20th, 2019

The latest updates on AWS AppSync & Step Functions.
From frontend to backend, there and back again!

Transcript

1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
   Danilo Poccia
   Principal Evangelist, Serverless
   @danilop
   AWS AppSync & Step Functions
   From frontend to backend, there and back again
   

   View Slide

2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
   © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
   https://aws.amazon.com/events/summits/amsterdam/
   

   View Slide

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
   What is GraphQL?
   GraphQL is a query language for APIs and a runtime for fulfilling
   those queries with your existing data
   Traditional data-fetching GraphQL
   /posts
   /postInfo
   /postJustTitle
   /postsByAuthor
   /postNameStartsWithX
   /commentsOnPost
   

   View Slide

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
   How does GraphQL work?
   {
   "id": "1",
   "name": "Get Milk",
   "priority": "1"
   },
   {
   "id": "2",
   "name": "Go to gym",
   "priority": "5"
   },…
   type Query {
   getTodos: [Todo]
   }
   type Todo {
   id: ID!
   name: String
   description: String
   priority: Int
   duedate: String
   }
   query {
   getTodos {
   id
   name
   priority
   }
   }
   Model data with
   application schema
   Client requests what it
   needs
   Only that data is
   returned
   

   View Slide

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
   Is REST dead then?
   • When data drives UI
   • Structured data
   • Query-driven
   • Client-driven
   development
   Use GraphQL
   • When you leverage HTTP
   • Caching, Content types
   • Hypermedia (HATEOAS)
   • For resources
   (e.g., Amazon S3)
   Use REST
   Pick the appropriate protocol for your use case
   

   View Slide

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
   GraphQL
   Schema
   type Event {
   id: ID!
   name: String
   where: String
   when: String
   description: String
   comments: [Comment]
   }
   type Comment {
   commentId: String!
   eventId: ID!
   content: String!
   createdAt: String!
   }
   

   View Slide

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
   GraphQL
   Schema
   Mutation
   type Mutation {
   createEvent(
   name: String!,
   when: String!,
   where: String!,
   description: String!
   ): Event
   deleteEvent(id: ID!): Event
   commentOnEvent(
   eventId: ID!,
   content: String!,
   createdAt: String!
   ): Comment
   }
   

   View Slide

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
   GraphQL
   Schema
   Mutation
   Query
   type Query {
   getEvent(id: ID!): Event
   listEvents(
   filter: TableEventFilterInput,
   limit: Int = 10,
   nextToken: String
   ): EventConnection
   }
   

   View Slide

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
   GraphQL
   Schema
   Mutation
   Query
   Subscription
   type Subscription {
   subscribeToEventComments(eventId: String!): Comment
   @aws_subscribe(mutations: ["commentOnEvent"])
   }
   

   View Slide

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    GraphQL
    Schema
    Mutation
    Query
    Subscription
    Realtime? YES
    Batching? YES
    Pagination? YES
    Relations? YES
    Aggregations? YES
    Search? YES
    Offline? YES
    

    View Slide

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Introducing AWS AppSync
    AWS AppSync is a managed service for application data
    using GraphQL with real-time capabilities and an offline
    programming model
    Real-time
    collaboration
    Offline programming
    model with sync
    Flexible database
    options
    Fine-grained
    access control
    

    View Slide

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    AWS AppSync
    DynamoDB
    Table
    Lambda
    Function
    GraphQL
    Schema
    Upload
    Schema
    GraphQL
    Query
    Mutation
    Subscription
    Real-time
    Offline
    Users
    AppSync
    API
    Data
    Sources
    Resolvers
    Pipelines / Functions
    Velocity
    Templates (VTL) Auth
    Cognito
    User Pool
    AWS
    IAM
    API
    Key
    OpenID
    Connect
    Elasticsearch
    Service
    HTTP
    Endpoint
    Relational
    Database
    

    View Slide

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    AWS AppSync
    DynamoDB
    Table
    Lambda
    Function
    GraphQL
    Schema
    Upload
    Schema
    GraphQL
    Query
    Mutation
    Subscription
    Real-time
    Offline
    Users
    AppSync
    API
    Data
    Sources
    Auth
    Cognito
    User Pool
    AWS
    IAM
    API
    Key
    OpenID
    Connect
    Create
    DynamoDB
    Table
    Create
    Resource
    Elasticsearch
    Service
    Resolvers
    Pipelines / Functions
    Velocity
    Templates (VTL)
    HTTP
    Endpoint
    Relational
    Database
    

    View Slide

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    AWS AppSync
    DynamoDB
    Table
    Lambda
    Function
    GraphQL
    Schema
    Upload
    Schema
    GraphQL
    Query
    Mutation
    Subscription
    Real-time
    Offline
    Users
    AppSync
    API
    Data
    Sources
    Auth
    Cognito
    User Pool
    AWS
    IAM
    API
    Key
    OpenID
    Connect
    Autogenerate
    Schema &
    Resolvers
    Im
    port a
    D
    ynam
    oD
    B
    Table
    Elasticsearch
    Service
    Resolvers
    Pipelines / Functions
    Velocity
    Templates (VTL)
    HTTP
    Endpoint
    Relational
    Database
    

    View Slide

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    AWS AppSync
    DynamoDB
    Table
    Lambda
    Function
    GraphQL
    Schema
    Upload
    Schema
    GraphQL
    Query
    Mutation
    Subscription
    Real-time
    Offline
    Users
    AppSync
    API
    Data
    Sources
    Auth
    Cognito
    User Pool
    AWS
    IAM
    API
    Key
    OpenID
    Connect
    Other
    Databases
    O
    ther
    D
    atabases
    Elasticsearch
    Service
    Resolvers
    Pipelines / Functions
    Velocity
    Templates (VTL)
    HTTP
    Endpoint
    Relational
    Database
    

    View Slide

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    AWS AppSync
    DynamoDB
    Table
    Lambda
    Function
    GraphQL
    Schema
    Upload
    Schema
    GraphQL
    Query
    Mutation
    Subscription
    Real-time
    Offline
    Users
    AppSync
    API
    Data
    Sources
    Auth
    Cognito
    User Pool
    AWS
    IAM
    API
    Key
    OpenID
    Connect
    Legacy
    Applications
    Legacy
    Application
    Elasticsearch
    Service
    Resolvers
    Pipelines / Functions
    Velocity
    Templates (VTL)
    HTTP
    Endpoint
    Relational
    Database
    

    View Slide

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    AWS AppSync
    DynamoDB
    Table
    Lambda
    Function
    Elasticsearch
    Service
    GraphQL
    Schema
    Upload
    Schema
    GraphQL
    Query
    Mutation
    Subscription
    Real-time
    Offline
    Users
    AppSync
    API
    Data
    Sources
    Auth
    Cognito
    User Pool
    AWS
    IAM
    API
    Key
    OpenID
    Connect
    Com
    plex
    Q
    ueries
    DynamoDB to Elasticsearch
    Sync Function
    Resolvers
    Pipelines / Functions
    Velocity
    Templates (VTL)
    HTTP
    Endpoint
    Relational
    Database
    

    View Slide

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    M
    usixm
    atch
    AW
    S
    Sum
    m
    it
    M
    ilan
    2018
    

    View Slide

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    AWS Amplify
    Amplify your apps.
    Build on a flexible, scalable, and reliable serverless backend.
    Easy-to-Use Library
    Choose your cloud services
    and easily connect them to
    your app with just a few
    lines of code.
    Powerful Toolchain
    Effortlessly create and
    maintain sophisticated
    serverless backends for
    your apps.
    Beautiful UI Components
    Accelerate app development
    by leveraging our beautiful
    out-of-the box UI
    components.
    O
    pen
    Source
    

    View Slide

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    AW
    S
    Am
    plify
    CLI
    

    View Slide

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    AW
    S
    Am
    plify
    CLI
    Effortlessly configure
    backends
    Hosting for static assets
    and websites
    Easily create APIs and
    generate client code
    $ cd
    $ amplify init
    $ amplify add
    $ amplify push
    $ amplify add hosting
    $ amplify publish
    $ amplify codegen add
    $ amplify codegen generate
    

    View Slide

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    AW
    S
    Am
    plify
    Console
    

    View Slide

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    AW
    S
    Am
    plify
    Console
    

    View Slide

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    GraphQL Sample Schema
    type Note {
    id: ID!
    title: String!
    content: String!
    }
    type Query {
    getNote(id: ID!): Note
    listNotes: [Note]
    }
    type Mutation {
    createNote(title: String!, content: String!): Note
    updateNote(id: ID!, title: String!, content: String!): Note
    deleteNote(id: ID!): Note
    }
    schema {
    query: Query
    mutation: Mutation
    }
    

    View Slide

25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    GraphQL Transformers
    type Note @model {
    id: ID!
    title: String!
    content: String!
    }
    AW
    S
    Am
    plify
    

    View Slide

26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    GraphQL Transformers
    type Note
    @model
    @auth(rules: [ { allow: owner } ])
    @searchable
    @versioned
    {
    id: ID!
    title: String!
    content: String!
    }
    AW
    S
    Am
    plify
    

    View Slide

27. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
    AWS AppSync & Amplify Demo
    

    View Slide

28. λ
    λ
    λ
    DBMS
    λ
    λ
    λ
    λ
    λ
    λ λ
    λ
    λ
    Queue
    Modern
    serverless
    app
    

    View Slide

29. Modern
    serverless
    app
    

    View Slide

30. "I want to sequence functions"
    "I want to select functions based on data"
    "I want to retry functions"
    "I want try/catch/finally"
    Functions into apps
    "I have code that runs for hours"
    "I want to run functions in parallel"
    

    View Slide

31. Coordination must-haves
    • Scales out
    • Doesn’t lose state
    • Deals with errors/timeouts
    • Easy to build & operate
    • Auditable
    

    View Slide

32. Application Lifecycle in AWS Step Functions
    Visualize in the
    Console
    Define in JSON Monitor
    Executions
    

    View Slide

33. Define in JSON and Then Visualize in the Console
    {
    "Comment": "Hello World Example",
    "StartAt" : "HelloWorld",
    "States" : {
    "HelloWorld" : {
    "Type" : "Task",
    "Resource" : "${lambdaArn}",
    "End" : true
    }
    }
    }
    

    View Slide

34. Execute One or One Million
    Start
    End
    HelloWorld
    Start
    End
    HelloWorld
    Start
    End
    HelloWorld
    Start
    End
    HelloWorld
    Start
    End
    HelloWorld
    Start
    End
    HelloWorld
    Start
    End
    HelloWorld
    Start
    End
    HelloWorld
    

    View Slide

35. Maximum Execution Time
    AWS Lambda
    Functions
    15 minutes
    AWS Step Functions
    State Machines
    1 year
    

    View Slide

36. Monitor Executions from the Console
    

    View Slide

37. Seven State Types
    Task A single unit of work
    Choice Adds branching logic
    Parallel Fork and join the data across tasks
    Wait Delay for a specified time
    Fail Stops an execution and marks it as a failure
    Succeed Stops an execution successfully
    Pass Passes its input to its output
    

    View Slide

38. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    ExtractImageMetadata
    ImageTypeCheck
    TransformMetadata
    Rekognition Thumbnail
    StoreMetadata
    Start
    End
    Synchronous task
    Task: a single unit of work
    Synchronous: Lambda
    {
    "ExtractImageMetadata": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:mars-ter …",
    …
    }
    }
    

    View Slide

39. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Step Functions Tasks – Service Integrations
    AWS Lambda invoke a Lambda function
    AWS Batch submit a Batch job and wait for it to complete
    Amazon DynamoDB get, put, update or delete an item
    Amazon ECS/Fargate run an ECS task and waits for it to complete
    Amazon SNS publish a message to a SNS topic
    Amazon SQS send a SQS message
    AWS Glue start a Glue job
    Amazon SageMaker create a training or transform job
    

    View Slide

40. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    ExtractImageMetadata
    TransformMetadata
    Rekognition Thumbnail
    StoreMetadata
    Start
    End
    Task: a single unit of work
    Asynchronous: "Activity Task", Polled by workers
    {
    "ExtractImageMetadata": {
    "Type": "Task",
    "Resource": "arn:aws:states:mars-ter …",
    …
    }
    }
    Asynchronous task
    

    View Slide

41. API actions on activity task
    Register Activity Task - Returns ARN
    Poll For task (by ARN)
    Report Success
    Report Failure
    Report Heartbeat
    

    View Slide

42. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    O
    pen
    Source
    A
    pache
    License
    https://states-language.net/spec.html
    

    View Slide

43. Build Visual Workflows from State Types
    Task
    Choice
    Fail
    Parallel
    

    View Slide

44. Image Recognition and Processing Backend
    Task
    Choice
    Fail
    Parallel
    https://github.com/awslabs/lambda-refarch-imagerecognition
    

    View Slide

45. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
    AWS Step Functions Demo
    

    View Slide

46. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    

    View Slide

47. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
    AWS Step Functions Case Studies
    

    View Slide

48. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Vending Pass problem
    1. Thirsty consumer presents
    card, can buy a drink with
    Vending Pass or debit
    2. But mobile wallet display can
    get out of sync under certain
    conditions
    3.
    

    View Slide

49. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Vending Pass problem
    • Just wait for backend
    to catch up
    • Create a two-step state
    machine:
    • Wait (90 seconds)
    • Update mobile wallet
    • Cheap and simple!
    

    View Slide

50. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    • Transcode 350 clips/day into
    14 formats, fast
    • It’s all done with FFmpeg. The
    processing time is just about
    100% of the video length
    • Aargh!
    Video processing problem
    

    View Slide

51. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    • Derive keyframe locations
    within the source
    • Split the source at the
    keyframes
    • Process segments (typically
    0.5 sec per) in parallel
    • Concatenate segments
    • Elapsed time: ~20 min down
    to ~2 minutes
    Video processing solution
    

    View Slide

52. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Details
    How to get from Lambda to Amazon S3:
    ffmpeg -f concat –safe 0 -I filelist.ffcat –c copy pipe:0 |
    aws s3 cp – s3://output-bucket/output-file.mp4
    

    View Slide

53. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Details
    

    View Slide

54. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Let’s peek at the code!
    "Init": {
    "Type": "Pass",
    "ResultPath": "$.operation",
    "Result": "prepare-split",
    "Next": "PrepareSplit"
    },
    "PrepareSplit": {
    "Type": "Task",
    "Resource": "arn:…:ffmpeg-18HFP9FXFL6P",
    "Next": "SplitPrepared"
    },
    "SplitPrepared": {
    "Type": "Pass",
    "ResultPath": "$.operation",
    "Result": "perform-split",
    "Next": "PerformSplit"
    },
    "PerformSplit": {
    "Type": "Task",
    "Resource": "arn:…:ffmpeg-18HFP9FXFL6P",
    "Next": "SplitPerformed"
    },
    "SplitPerformed": {
    "Type": "Pass",
    "ResultPath": "$.operation",
    "Result": "poll-results",
    "Next": "PreparePoll"
    },
    "PreparePoll": {
    "Type": "Pass",
    "ResultPath": "$.pollstatus",
    "Result": "Submitted",
    "Next": "WaitForResults"
    }
    Hmm…the same function,
    throughout
    

    View Slide

55. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Let’s peek at the code!
    "WaitForResults": {
    "Type": "Choice",
    "Choices": [
    {
    "Variable": "$.pollstatus",
    "StringEquals": "Complete",
    "Next": "SegmentsCompleteState"
    },
    {
    "Variable": "$.pollstatus",
    "StringEquals": "Progressing",
    "Next": "loop_wait_using_seconds"
    },
    {
    "Variable": "$.pollstatus",
    "StringEquals": "Submitted",
    "Next": "loop_wait_using_seconds"
    }
    ]
    }
    "loop_wait_using_seconds": {
    "Type": "Wait",
    "Seconds": 3,
    "Next": "loop"
    },
    "loop": {
    "Type": "Task",
    "Resource": "arn:…:ffmpeg-18HFP9FXFL6P",
    "Next": "WaitForResults"
    },
    "SegmentsCompleteState": {
    "Type": "Pass",
    "ResultPath": "$.operation",
    "Result": "concat",
    "Next": "CompleteState"
    },
    "CompleteState": {
    "Type": "Task",
    "Resource": " arn:…:ffmpeg-18HFP9FXFL6P",
    "End": true
    }
    Counts the segments,
    sets $.pollstatus
    Stitches segments together
    

    View Slide

56. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Takeaways
    Use Amazon S3
    data events to
    trigger parallel
    Lambda
    processing: win
    Use Amazon S3
    URLs to stream
    video to
    Lambda: win
    Scaling to 1,000
    Lambdas, rather
    than 1,000 EC2
    instances: win
    Process video
    segments in
    parallel: win
    

    View Slide

57. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    More state machines
    Image Recognition and Processing Backend
    EBS Snapshot Management
    https://aws.amazon.com/step-functions/getting-started
    

    View Slide

58. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    What are you building? #stepfunctions
    https://console.aws.amazon.com/states/
    

    View Slide

59. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
    Thank you!
    @danilop
    

    View Slide