AWS AppSync & Step Functions

AWS User Group Netherlands + J-AWS, Haarlem, February 20th, 2019

The latest updates on AWS AppSync & Step Functions.
From frontend to backend, there and back again!

Transcript

1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Danilo Poccia Principal Evangelist, Serverless @danilop AWS AppSync & Step Functions From frontend to backend, there and back again
2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark https://aws.amazon.com/events/summits/amsterdam/
3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What is GraphQL? GraphQL is a query language for APIs and a runtime for fulfilling those queries with your existing data Traditional data-fetching GraphQL /posts /postInfo /postJustTitle /postsByAuthor /postNameStartsWithX /commentsOnPost
4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. How does GraphQL work? { "id": "1", "name": "Get Milk", "priority": "1" }, { "id": "2", "name": "Go to gym", "priority": "5" },… type Query { getTodos: [Todo] } type Todo { id: ID! name: String description: String priority: Int duedate: String } query { getTodos { id name priority } } Model data with application schema Client requests what it needs Only that data is returned
5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Is REST dead then? • When data drives UI • Structured data • Query-driven • Client-driven development Use GraphQL • When you leverage HTTP • Caching, Content types • Hypermedia (HATEOAS) • For resources (e.g., Amazon S3) Use REST Pick the appropriate protocol for your use case
6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. GraphQL Schema type Event { id: ID! name: String where: String when: String description: String comments: [Comment] } type Comment { commentId: String! eventId: ID! content: String! createdAt: String! }
7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. GraphQL Schema Mutation type Mutation { createEvent( name: String!, when: String!, where: String!, description: String! ): Event deleteEvent(id: ID!): Event commentOnEvent( eventId: ID!, content: String!, createdAt: String! ): Comment }
8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. GraphQL Schema Mutation Query type Query { getEvent(id: ID!): Event listEvents( filter: TableEventFilterInput, limit: Int = 10, nextToken: String ): EventConnection }
9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. GraphQL Schema Mutation Query Subscription type Subscription { subscribeToEventComments(eventId: String!): Comment @aws_subscribe(mutations: ["commentOnEvent"]) }
10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. GraphQL Schema Mutation Query Subscription Realtime? YES Batching? YES Pagination? YES Relations? YES Aggregations? YES Search? YES Offline? YES
11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Introducing AWS AppSync AWS AppSync is a managed service for application data using GraphQL with real-time capabilities and an offline programming model Real-time collaboration Offline programming model with sync Flexible database options Fine-grained access control
12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS AppSync DynamoDB Table Lambda Function GraphQL Schema Upload Schema GraphQL Query Mutation Subscription Real-time Offline Users AppSync API Data Sources Resolvers Pipelines / Functions Velocity Templates (VTL) Auth Cognito User Pool AWS IAM API Key OpenID Connect Elasticsearch Service HTTP Endpoint Relational Database
13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS AppSync DynamoDB Table Lambda Function GraphQL Schema Upload Schema GraphQL Query Mutation Subscription Real-time Offline Users AppSync API Data Sources Auth Cognito User Pool AWS IAM API Key OpenID Connect Create DynamoDB Table Create Resource Elasticsearch Service Resolvers Pipelines / Functions Velocity Templates (VTL) HTTP Endpoint Relational Database
14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS AppSync DynamoDB Table Lambda Function GraphQL Schema Upload Schema GraphQL Query Mutation Subscription Real-time Offline Users AppSync API Data Sources Auth Cognito User Pool AWS IAM API Key OpenID Connect Autogenerate Schema & Resolvers Im port a D ynam oD B Table Elasticsearch Service Resolvers Pipelines / Functions Velocity Templates (VTL) HTTP Endpoint Relational Database
15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS AppSync DynamoDB Table Lambda Function GraphQL Schema Upload Schema GraphQL Query Mutation Subscription Real-time Offline Users AppSync API Data Sources Auth Cognito User Pool AWS IAM API Key OpenID Connect Other Databases O ther D atabases Elasticsearch Service Resolvers Pipelines / Functions Velocity Templates (VTL) HTTP Endpoint Relational Database
16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS AppSync DynamoDB Table Lambda Function GraphQL Schema Upload Schema GraphQL Query Mutation Subscription Real-time Offline Users AppSync API Data Sources Auth Cognito User Pool AWS IAM API Key OpenID Connect Legacy Applications Legacy Application Elasticsearch Service Resolvers Pipelines / Functions Velocity Templates (VTL) HTTP Endpoint Relational Database
17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS AppSync DynamoDB Table Lambda Function Elasticsearch Service GraphQL Schema Upload Schema GraphQL Query Mutation Subscription Real-time Offline Users AppSync API Data Sources Auth Cognito User Pool AWS IAM API Key OpenID Connect Com plex Q ueries DynamoDB to Elasticsearch Sync Function Resolvers Pipelines / Functions Velocity Templates (VTL) HTTP Endpoint Relational Database
18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. M usixm atch AW S Sum m it M ilan 2018
19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Amplify Amplify your apps. Build on a flexible, scalable, and reliable serverless backend. Easy-to-Use Library Choose your cloud services and easily connect them to your app with just a few lines of code. Powerful Toolchain Effortlessly create and maintain sophisticated serverless backends for your apps. Beautiful UI Components Accelerate app development by leveraging our beautiful out-of-the box UI components. O pen Source
20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AW S Am plify CLI
21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AW S Am plify CLI Effortlessly configure backends Hosting for static assets and websites Easily create APIs and generate client code $ cd <your-app> $ amplify init $ amplify add <category> $ amplify push $ amplify add hosting $ amplify publish $ amplify codegen add $ amplify codegen generate
22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AW S Am plify Console
23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AW S Am plify Console
24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. GraphQL Sample Schema type Note { id: ID! title: String! content: String! } type Query { getNote(id: ID!): Note listNotes: [Note] } type Mutation { createNote(title: String!, content: String!): Note updateNote(id: ID!, title: String!, content: String!): Note deleteNote(id: ID!): Note } schema { query: Query mutation: Mutation }
25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. GraphQL Transformers type Note @model { id: ID! title: String! content: String! } AW S Am plify
26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. GraphQL Transformers type Note @model @auth(rules: [ { allow: owner } ]) @searchable @versioned { id: ID! title: String! content: String! } AW S Am plify
27. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark AWS AppSync & Amplify Demo
28. λ λ λ DBMS λ λ λ λ λ λ λ λ λ Queue Modern serverless app
29. Modern serverless app
30. "I want to sequence functions" "I want to select functions based on data" "I want to retry functions" "I want try/catch/finally" Functions into apps "I have code that runs for hours" "I want to run functions in parallel"
31. Coordination must-haves • Scales out • Doesn’t lose state • Deals with errors/timeouts • Easy to build & operate • Auditable
32. Application Lifecycle in AWS Step Functions Visualize in the Console Define in JSON Monitor Executions
33. Define in JSON and Then Visualize in the Console { "Comment": "Hello World Example", "StartAt" : "HelloWorld", "States" : { "HelloWorld" : { "Type" : "Task", "Resource" : "${lambdaArn}", "End" : true } } }
34. Execute One or One Million Start End HelloWorld Start End HelloWorld Start End HelloWorld Start End HelloWorld Start End HelloWorld Start End HelloWorld Start End HelloWorld Start End HelloWorld
35. Maximum Execution Time AWS Lambda Functions 15 minutes AWS Step Functions State Machines 1 year
36. Monitor Executions from the Console
37. Seven State Types Task A single unit of work Choice Adds branching logic Parallel Fork and join the data across tasks Wait Delay for a specified time Fail Stops an execution and marks it as a failure Succeed Stops an execution successfully Pass Passes its input to its output
38. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ExtractImageMetadata ImageTypeCheck TransformMetadata Rekognition Thumbnail StoreMetadata Start End Synchronous task Task: a single unit of work Synchronous: Lambda { "ExtractImageMetadata": { "Type": "Task", "Resource": "arn:aws:lambda:mars-ter …", … } }
39. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Step Functions Tasks – Service Integrations AWS Lambda invoke a Lambda function AWS Batch submit a Batch job and wait for it to complete Amazon DynamoDB get, put, update or delete an item Amazon ECS/Fargate run an ECS task and waits for it to complete Amazon SNS publish a message to a SNS topic Amazon SQS send a SQS message AWS Glue start a Glue job Amazon SageMaker create a training or transform job
40. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. ExtractImageMetadata TransformMetadata Rekognition Thumbnail StoreMetadata Start End Task: a single unit of work Asynchronous: "Activity Task", Polled by workers { "ExtractImageMetadata": { "Type": "Task", "Resource": "arn:aws:states:mars-ter …", … } } Asynchronous task
41. API actions on activity task Register Activity Task - Returns ARN Poll For task (by ARN) Report Success Report Failure Report Heartbeat
42. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. O pen Source A pache License https://states-language.net/spec.html
43. Build Visual Workflows from State Types Task Choice Fail Parallel
44. Image Recognition and Processing Backend Task Choice Fail Parallel https://github.com/awslabs/lambda-refarch-imagerecognition
45. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark AWS Step Functions Demo
46. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
47. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark AWS Step Functions Case Studies
48. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Vending Pass problem 1. Thirsty consumer presents card, can buy a drink with Vending Pass or debit 2. But mobile wallet display can get out of sync under certain conditions 3.
49. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Vending Pass problem • Just wait for backend to catch up • Create a two-step state machine: • Wait (90 seconds) • Update mobile wallet • Cheap and simple!
50. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • Transcode 350 clips/day into 14 formats, fast • It’s all done with FFmpeg. The processing time is just about 100% of the video length • Aargh! Video processing problem
51. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • Derive keyframe locations within the source • Split the source at the keyframes • Process segments (typically 0.5 sec per) in parallel • Concatenate segments • Elapsed time: ~20 min down to ~2 minutes Video processing solution
52. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Details How to get from Lambda to Amazon S3: ffmpeg -f concat –safe 0 -I filelist.ffcat –c copy pipe:0 | aws s3 cp – s3://output-bucket/output-file.mp4
53. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Details
54. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Let’s peek at the code! "Init": { "Type": "Pass", "ResultPath": "$.operation", "Result": "prepare-split", "Next": "PrepareSplit" }, "PrepareSplit": { "Type": "Task", "Resource": "arn:…:ffmpeg-18HFP9FXFL6P", "Next": "SplitPrepared" }, "SplitPrepared": { "Type": "Pass", "ResultPath": "$.operation", "Result": "perform-split", "Next": "PerformSplit" }, "PerformSplit": { "Type": "Task", "Resource": "arn:…:ffmpeg-18HFP9FXFL6P", "Next": "SplitPerformed" }, "SplitPerformed": { "Type": "Pass", "ResultPath": "$.operation", "Result": "poll-results", "Next": "PreparePoll" }, "PreparePoll": { "Type": "Pass", "ResultPath": "$.pollstatus", "Result": "Submitted", "Next": "WaitForResults" } Hmm…the same function, throughout
55. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Let’s peek at the code! "WaitForResults": { "Type": "Choice", "Choices": [ { "Variable": "$.pollstatus", "StringEquals": "Complete", "Next": "SegmentsCompleteState" }, { "Variable": "$.pollstatus", "StringEquals": "Progressing", "Next": "loop_wait_using_seconds" }, { "Variable": "$.pollstatus", "StringEquals": "Submitted", "Next": "loop_wait_using_seconds" } ] } "loop_wait_using_seconds": { "Type": "Wait", "Seconds": 3, "Next": "loop" }, "loop": { "Type": "Task", "Resource": "arn:…:ffmpeg-18HFP9FXFL6P", "Next": "WaitForResults" }, "SegmentsCompleteState": { "Type": "Pass", "ResultPath": "$.operation", "Result": "concat", "Next": "CompleteState" }, "CompleteState": { "Type": "Task", "Resource": " arn:…:ffmpeg-18HFP9FXFL6P", "End": true } Counts the segments, sets $.pollstatus Stitches segments together
56. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Takeaways Use Amazon S3 data events to trigger parallel Lambda processing: win Use Amazon S3 URLs to stream video to Lambda: win Scaling to 1,000 Lambdas, rather than 1,000 EC2 instances: win Process video segments in parallel: win
57. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. More state machines Image Recognition and Processing Backend EBS Snapshot Management https://aws.amazon.com/step-functions/getting-started
58. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What are you building? #stepfunctions https://console.aws.amazon.com/states/
59. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Thank you! @danilop