AWS AppSync & Step Functions

AWS AppSync & Step Functions

AWS User Group Netherlands + J-AWS, Haarlem, February 20th, 2019

The latest updates on AWS AppSync & Step Functions.
From frontend to backend, there and back again!

7c9b8b368924556d8642bdaed3ded1f5?s=128

Danilo Poccia

February 20, 2019
Tweet

Transcript

  1. © 2019, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Amazon Confidential and Trademark Danilo Poccia Principal Evangelist, Serverless @danilop AWS AppSync & Step Functions From frontend to backend, there and back again
  2. © 2019, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Amazon Confidential and Trademark © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark https://aws.amazon.com/events/summits/amsterdam/
  3. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. What is GraphQL? GraphQL is a query language for APIs and a runtime for fulfilling those queries with your existing data Traditional data-fetching GraphQL /posts /postInfo /postJustTitle /postsByAuthor /postNameStartsWithX /commentsOnPost
  4. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. How does GraphQL work? { "id": "1", "name": "Get Milk", "priority": "1" }, { "id": "2", "name": "Go to gym", "priority": "5" },… type Query { getTodos: [Todo] } type Todo { id: ID! name: String description: String priority: Int duedate: String } query { getTodos { id name priority } } Model data with application schema Client requests what it needs Only that data is returned
  5. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Is REST dead then? • When data drives UI • Structured data • Query-driven • Client-driven development Use GraphQL • When you leverage HTTP • Caching, Content types • Hypermedia (HATEOAS) • For resources (e.g., Amazon S3) Use REST Pick the appropriate protocol for your use case
  6. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. GraphQL Schema type Event { id: ID! name: String where: String when: String description: String comments: [Comment] } type Comment { commentId: String! eventId: ID! content: String! createdAt: String! }
  7. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. GraphQL Schema Mutation type Mutation { createEvent( name: String!, when: String!, where: String!, description: String! ): Event deleteEvent(id: ID!): Event commentOnEvent( eventId: ID!, content: String!, createdAt: String! ): Comment }
  8. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. GraphQL Schema Mutation Query type Query { getEvent(id: ID!): Event listEvents( filter: TableEventFilterInput, limit: Int = 10, nextToken: String ): EventConnection }
  9. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. GraphQL Schema Mutation Query Subscription type Subscription { subscribeToEventComments(eventId: String!): Comment @aws_subscribe(mutations: ["commentOnEvent"]) }
  10. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. GraphQL Schema Mutation Query Subscription Realtime? YES Batching? YES Pagination? YES Relations? YES Aggregations? YES Search? YES Offline? YES
  11. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Introducing AWS AppSync AWS AppSync is a managed service for application data using GraphQL with real-time capabilities and an offline programming model Real-time collaboration Offline programming model with sync Flexible database options Fine-grained access control
  12. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. AWS AppSync DynamoDB Table Lambda Function GraphQL Schema Upload Schema GraphQL Query Mutation Subscription Real-time Offline Users AppSync API Data Sources Resolvers Pipelines / Functions Velocity Templates (VTL) Auth Cognito User Pool AWS IAM API Key OpenID Connect Elasticsearch Service HTTP Endpoint Relational Database
  13. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. AWS AppSync DynamoDB Table Lambda Function GraphQL Schema Upload Schema GraphQL Query Mutation Subscription Real-time Offline Users AppSync API Data Sources Auth Cognito User Pool AWS IAM API Key OpenID Connect Create DynamoDB Table Create Resource Elasticsearch Service Resolvers Pipelines / Functions Velocity Templates (VTL) HTTP Endpoint Relational Database
  14. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. AWS AppSync DynamoDB Table Lambda Function GraphQL Schema Upload Schema GraphQL Query Mutation Subscription Real-time Offline Users AppSync API Data Sources Auth Cognito User Pool AWS IAM API Key OpenID Connect Autogenerate Schema & Resolvers Im port a D ynam oD B Table Elasticsearch Service Resolvers Pipelines / Functions Velocity Templates (VTL) HTTP Endpoint Relational Database
  15. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. AWS AppSync DynamoDB Table Lambda Function GraphQL Schema Upload Schema GraphQL Query Mutation Subscription Real-time Offline Users AppSync API Data Sources Auth Cognito User Pool AWS IAM API Key OpenID Connect Other Databases O ther D atabases Elasticsearch Service Resolvers Pipelines / Functions Velocity Templates (VTL) HTTP Endpoint Relational Database
  16. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. AWS AppSync DynamoDB Table Lambda Function GraphQL Schema Upload Schema GraphQL Query Mutation Subscription Real-time Offline Users AppSync API Data Sources Auth Cognito User Pool AWS IAM API Key OpenID Connect Legacy Applications Legacy Application Elasticsearch Service Resolvers Pipelines / Functions Velocity Templates (VTL) HTTP Endpoint Relational Database
  17. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. AWS AppSync DynamoDB Table Lambda Function Elasticsearch Service GraphQL Schema Upload Schema GraphQL Query Mutation Subscription Real-time Offline Users AppSync API Data Sources Auth Cognito User Pool AWS IAM API Key OpenID Connect Com plex Q ueries DynamoDB to Elasticsearch Sync Function Resolvers Pipelines / Functions Velocity Templates (VTL) HTTP Endpoint Relational Database
  18. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. M usixm atch AW S Sum m it M ilan 2018
  19. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. AWS Amplify Amplify your apps. Build on a flexible, scalable, and reliable serverless backend. Easy-to-Use Library Choose your cloud services and easily connect them to your app with just a few lines of code. Powerful Toolchain Effortlessly create and maintain sophisticated serverless backends for your apps. Beautiful UI Components Accelerate app development by leveraging our beautiful out-of-the box UI components. O pen Source
  20. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. AW S Am plify CLI
  21. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. AW S Am plify CLI Effortlessly configure backends Hosting for static assets and websites Easily create APIs and generate client code $ cd <your-app> $ amplify init $ amplify add <category> $ amplify push $ amplify add hosting $ amplify publish $ amplify codegen add $ amplify codegen generate
  22. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. AW S Am plify Console
  23. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. AW S Am plify Console
  24. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. GraphQL Sample Schema type Note { id: ID! title: String! content: String! } type Query { getNote(id: ID!): Note listNotes: [Note] } type Mutation { createNote(title: String!, content: String!): Note updateNote(id: ID!, title: String!, content: String!): Note deleteNote(id: ID!): Note } schema { query: Query mutation: Mutation }
  25. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. GraphQL Transformers type Note @model { id: ID! title: String! content: String! } AW S Am plify
  26. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. GraphQL Transformers type Note @model @auth(rules: [ { allow: owner } ]) @searchable @versioned { id: ID! title: String! content: String! } AW S Am plify
  27. © 2019, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Amazon Confidential and Trademark AWS AppSync & Amplify Demo
  28. λ λ λ DBMS λ λ λ λ λ λ

    λ λ λ Queue Modern serverless app
  29. Modern serverless app

  30. "I want to sequence functions" "I want to select functions

    based on data" "I want to retry functions" "I want try/catch/finally" Functions into apps "I have code that runs for hours" "I want to run functions in parallel"
  31. Coordination must-haves • Scales out • Doesn’t lose state •

    Deals with errors/timeouts • Easy to build & operate • Auditable
  32. Application Lifecycle in AWS Step Functions Visualize in the Console

    Define in JSON Monitor Executions
  33. Define in JSON and Then Visualize in the Console {

    "Comment": "Hello World Example", "StartAt" : "HelloWorld", "States" : { "HelloWorld" : { "Type" : "Task", "Resource" : "${lambdaArn}", "End" : true } } }
  34. Execute One or One Million Start End HelloWorld Start End

    HelloWorld Start End HelloWorld Start End HelloWorld Start End HelloWorld Start End HelloWorld Start End HelloWorld Start End HelloWorld
  35. Maximum Execution Time AWS Lambda Functions 15 minutes AWS Step

    Functions State Machines 1 year
  36. Monitor Executions from the Console

  37. Seven State Types Task A single unit of work Choice

    Adds branching logic Parallel Fork and join the data across tasks Wait Delay for a specified time Fail Stops an execution and marks it as a failure Succeed Stops an execution successfully Pass Passes its input to its output
  38. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. ExtractImageMetadata ImageTypeCheck TransformMetadata Rekognition Thumbnail StoreMetadata Start End Synchronous task Task: a single unit of work Synchronous: Lambda { "ExtractImageMetadata": { "Type": "Task", "Resource": "arn:aws:lambda:mars-ter …", … } }
  39. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Step Functions Tasks – Service Integrations AWS Lambda invoke a Lambda function AWS Batch submit a Batch job and wait for it to complete Amazon DynamoDB get, put, update or delete an item Amazon ECS/Fargate run an ECS task and waits for it to complete Amazon SNS publish a message to a SNS topic Amazon SQS send a SQS message AWS Glue start a Glue job Amazon SageMaker create a training or transform job
  40. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. ExtractImageMetadata TransformMetadata Rekognition Thumbnail StoreMetadata Start End Task: a single unit of work Asynchronous: "Activity Task", Polled by workers { "ExtractImageMetadata": { "Type": "Task", "Resource": "arn:aws:states:mars-ter …", … } } Asynchronous task
  41. API actions on activity task Register Activity Task - Returns

    ARN Poll For task (by ARN) Report Success Report Failure Report Heartbeat
  42. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. O pen Source A pache License https://states-language.net/spec.html
  43. Build Visual Workflows from State Types Task Choice Fail Parallel

  44. Image Recognition and Processing Backend Task Choice Fail Parallel https://github.com/awslabs/lambda-refarch-imagerecognition

  45. © 2019, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Amazon Confidential and Trademark AWS Step Functions Demo
  46. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved.
  47. © 2019, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Amazon Confidential and Trademark AWS Step Functions Case Studies
  48. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Vending Pass problem 1. Thirsty consumer presents card, can buy a drink with Vending Pass or debit 2. But mobile wallet display can get out of sync under certain conditions 3.
  49. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Vending Pass problem • Just wait for backend to catch up • Create a two-step state machine: • Wait (90 seconds) • Update mobile wallet • Cheap and simple!
  50. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. • Transcode 350 clips/day into 14 formats, fast • It’s all done with FFmpeg. The processing time is just about 100% of the video length • Aargh! Video processing problem
  51. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. • Derive keyframe locations within the source • Split the source at the keyframes • Process segments (typically 0.5 sec per) in parallel • Concatenate segments • Elapsed time: ~20 min down to ~2 minutes Video processing solution
  52. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Details How to get from Lambda to Amazon S3: ffmpeg -f concat –safe 0 -I filelist.ffcat –c copy pipe:0 | aws s3 cp – s3://output-bucket/output-file.mp4
  53. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Details
  54. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Let’s peek at the code! "Init": { "Type": "Pass", "ResultPath": "$.operation", "Result": "prepare-split", "Next": "PrepareSplit" }, "PrepareSplit": { "Type": "Task", "Resource": "arn:…:ffmpeg-18HFP9FXFL6P", "Next": "SplitPrepared" }, "SplitPrepared": { "Type": "Pass", "ResultPath": "$.operation", "Result": "perform-split", "Next": "PerformSplit" }, "PerformSplit": { "Type": "Task", "Resource": "arn:…:ffmpeg-18HFP9FXFL6P", "Next": "SplitPerformed" }, "SplitPerformed": { "Type": "Pass", "ResultPath": "$.operation", "Result": "poll-results", "Next": "PreparePoll" }, "PreparePoll": { "Type": "Pass", "ResultPath": "$.pollstatus", "Result": "Submitted", "Next": "WaitForResults" } Hmm…the same function, throughout
  55. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Let’s peek at the code! "WaitForResults": { "Type": "Choice", "Choices": [ { "Variable": "$.pollstatus", "StringEquals": "Complete", "Next": "SegmentsCompleteState" }, { "Variable": "$.pollstatus", "StringEquals": "Progressing", "Next": "loop_wait_using_seconds" }, { "Variable": "$.pollstatus", "StringEquals": "Submitted", "Next": "loop_wait_using_seconds" } ] } "loop_wait_using_seconds": { "Type": "Wait", "Seconds": 3, "Next": "loop" }, "loop": { "Type": "Task", "Resource": "arn:…:ffmpeg-18HFP9FXFL6P", "Next": "WaitForResults" }, "SegmentsCompleteState": { "Type": "Pass", "ResultPath": "$.operation", "Result": "concat", "Next": "CompleteState" }, "CompleteState": { "Type": "Task", "Resource": " arn:…:ffmpeg-18HFP9FXFL6P", "End": true } Counts the segments, sets $.pollstatus Stitches segments together
  56. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Takeaways Use Amazon S3 data events to trigger parallel Lambda processing: win Use Amazon S3 URLs to stream video to Lambda: win Scaling to 1,000 Lambdas, rather than 1,000 EC2 instances: win Process video segments in parallel: win
  57. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. More state machines Image Recognition and Processing Backend EBS Snapshot Management https://aws.amazon.com/step-functions/getting-started
  58. © 2018, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. What are you building? #stepfunctions https://console.aws.amazon.com/states/
  59. © 2019, Amazon Web Services, Inc. or its Affiliates. All

    rights reserved. Amazon Confidential and Trademark Thank you! @danilop