Share
Embed
AWS AppSync & Step Functions
AWS User Group Netherlands + J-AWS, Haarlem, February 20th, 2019
The latest updates on AWS AppSync & Step Functions.
From frontend to backend, there and back again!
Danilo Poccia
February 20, 2019
Transcript
59 slides
-
© 2019, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. Amazon Confidential and Trademark Danilo Poccia Principal Evangelist, Serverless @danilop AWS AppSync & Step Functions From frontend to backend, there and back again -
© 2019, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. Amazon Confidential and Trademark © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark https://aws.amazon.com/events/summits/amsterdam/ -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. What is GraphQL? GraphQL is a query language for APIs and a runtime for fulfilling those queries with your existing data Traditional data-fetching GraphQL /posts /postInfo /postJustTitle /postsByAuthor /postNameStartsWithX /commentsOnPost -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. How does GraphQL work? { "id": "1", "name": "Get Milk", "priority": "1" }, { "id": "2", "name": "Go to gym", "priority": "5" },… type Query { getTodos: [Todo] } type Todo { id: ID! name: String description: String priority: Int duedate: String } query { getTodos { id name priority } } Model data with application schema Client requests what it needs Only that data is returned -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. Is REST dead then? • When data drives UI • Structured data • Query-driven • Client-driven development Use GraphQL • When you leverage HTTP • Caching, Content types • Hypermedia (HATEOAS) • For resources (e.g., Amazon S3) Use REST Pick the appropriate protocol for your use case -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. GraphQL Schema type Event { id: ID! name: String where: String when: String description: String comments: [Comment] } type Comment { commentId: String! eventId: ID! content: String! createdAt: String! } -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. GraphQL Schema Mutation type Mutation { createEvent( name: String!, when: String!, where: String!, description: String! ): Event deleteEvent(id: ID!): Event commentOnEvent( eventId: ID!, content: String!, createdAt: String! ): Comment } -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. GraphQL Schema Mutation Query type Query { getEvent(id: ID!): Event listEvents( filter: TableEventFilterInput, limit: Int = 10, nextToken: String ): EventConnection } -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. GraphQL Schema Mutation Query Subscription type Subscription { subscribeToEventComments(eventId: String!): Comment @aws_subscribe(mutations: ["commentOnEvent"]) } -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. GraphQL Schema Mutation Query Subscription Realtime? YES Batching? YES Pagination? YES Relations? YES Aggregations? YES Search? YES Offline? YES -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. Introducing AWS AppSync AWS AppSync is a managed service for application data using GraphQL with real-time capabilities and an offline programming model Real-time collaboration Offline programming model with sync Flexible database options Fine-grained access control -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. AWS AppSync DynamoDB Table Lambda Function GraphQL Schema Upload Schema GraphQL Query Mutation Subscription Real-time Offline Users AppSync API Data Sources Resolvers Pipelines / Functions Velocity Templates (VTL) Auth Cognito User Pool AWS IAM API Key OpenID Connect Elasticsearch Service HTTP Endpoint Relational Database -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. AWS AppSync DynamoDB Table Lambda Function GraphQL Schema Upload Schema GraphQL Query Mutation Subscription Real-time Offline Users AppSync API Data Sources Auth Cognito User Pool AWS IAM API Key OpenID Connect Create DynamoDB Table Create Resource Elasticsearch Service Resolvers Pipelines / Functions Velocity Templates (VTL) HTTP Endpoint Relational Database -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. AWS AppSync DynamoDB Table Lambda Function GraphQL Schema Upload Schema GraphQL Query Mutation Subscription Real-time Offline Users AppSync API Data Sources Auth Cognito User Pool AWS IAM API Key OpenID Connect Autogenerate Schema & Resolvers Im port a D ynam oD B Table Elasticsearch Service Resolvers Pipelines / Functions Velocity Templates (VTL) HTTP Endpoint Relational Database -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. AWS AppSync DynamoDB Table Lambda Function GraphQL Schema Upload Schema GraphQL Query Mutation Subscription Real-time Offline Users AppSync API Data Sources Auth Cognito User Pool AWS IAM API Key OpenID Connect Other Databases O ther D atabases Elasticsearch Service Resolvers Pipelines / Functions Velocity Templates (VTL) HTTP Endpoint Relational Database -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. AWS AppSync DynamoDB Table Lambda Function GraphQL Schema Upload Schema GraphQL Query Mutation Subscription Real-time Offline Users AppSync API Data Sources Auth Cognito User Pool AWS IAM API Key OpenID Connect Legacy Applications Legacy Application Elasticsearch Service Resolvers Pipelines / Functions Velocity Templates (VTL) HTTP Endpoint Relational Database -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. AWS AppSync DynamoDB Table Lambda Function Elasticsearch Service GraphQL Schema Upload Schema GraphQL Query Mutation Subscription Real-time Offline Users AppSync API Data Sources Auth Cognito User Pool AWS IAM API Key OpenID Connect Com plex Q ueries DynamoDB to Elasticsearch Sync Function Resolvers Pipelines / Functions Velocity Templates (VTL) HTTP Endpoint Relational Database -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. M usixm atch AW S Sum m it M ilan 2018 -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. AWS Amplify Amplify your apps. Build on a flexible, scalable, and reliable serverless backend. Easy-to-Use Library Choose your cloud services and easily connect them to your app with just a few lines of code. Powerful Toolchain Effortlessly create and maintain sophisticated serverless backends for your apps. Beautiful UI Components Accelerate app development by leveraging our beautiful out-of-the box UI components. O pen Source -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. AW S Am plify CLI -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. AW S Am plify CLI Effortlessly configure backends Hosting for static assets and websites Easily create APIs and generate client code $ cd <your-app> $ amplify init $ amplify add <category> $ amplify push $ amplify add hosting $ amplify publish $ amplify codegen add $ amplify codegen generate -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. AW S Am plify Console -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. AW S Am plify Console -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. GraphQL Sample Schema type Note { id: ID! title: String! content: String! } type Query { getNote(id: ID!): Note listNotes: [Note] } type Mutation { createNote(title: String!, content: String!): Note updateNote(id: ID!, title: String!, content: String!): Note deleteNote(id: ID!): Note } schema { query: Query mutation: Mutation } -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. GraphQL Transformers type Note @model { id: ID! title: String! content: String! } AW S Am plify -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. GraphQL Transformers type Note @model @auth(rules: [ { allow: owner } ]) @searchable @versioned { id: ID! title: String! content: String! } AW S Am plify -
© 2019, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. Amazon Confidential and Trademark AWS AppSync & Amplify Demo -
λ λ λ DBMS λ λ λ λ λ λ
λ λ λ Queue Modern serverless app -
"I want to sequence functions" "I want to select functions
based on data" "I want to retry functions" "I want try/catch/finally" Functions into apps "I have code that runs for hours" "I want to run functions in parallel" -
Coordination must-haves • Scales out • Doesn’t lose state •
Deals with errors/timeouts • Easy to build & operate • Auditable -
Application Lifecycle in AWS Step Functions Visualize in the Console
Define in JSON Monitor Executions -
Define in JSON and Then Visualize in the Console {
"Comment": "Hello World Example", "StartAt" : "HelloWorld", "States" : { "HelloWorld" : { "Type" : "Task", "Resource" : "${lambdaArn}", "End" : true } } } -
Execute One or One Million Start End HelloWorld Start End
HelloWorld Start End HelloWorld Start End HelloWorld Start End HelloWorld Start End HelloWorld Start End HelloWorld Start End HelloWorld -
Maximum Execution Time AWS Lambda Functions 15 minutes AWS Step
Functions State Machines 1 year -
Seven State Types Task A single unit of work Choice
Adds branching logic Parallel Fork and join the data across tasks Wait Delay for a specified time Fail Stops an execution and marks it as a failure Succeed Stops an execution successfully Pass Passes its input to its output -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. ExtractImageMetadata ImageTypeCheck TransformMetadata Rekognition Thumbnail StoreMetadata Start End Synchronous task Task: a single unit of work Synchronous: Lambda { "ExtractImageMetadata": { "Type": "Task", "Resource": "arn:aws:lambda:mars-ter …", … } } -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. Step Functions Tasks – Service Integrations AWS Lambda invoke a Lambda function AWS Batch submit a Batch job and wait for it to complete Amazon DynamoDB get, put, update or delete an item Amazon ECS/Fargate run an ECS task and waits for it to complete Amazon SNS publish a message to a SNS topic Amazon SQS send a SQS message AWS Glue start a Glue job Amazon SageMaker create a training or transform job -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. ExtractImageMetadata TransformMetadata Rekognition Thumbnail StoreMetadata Start End Task: a single unit of work Asynchronous: "Activity Task", Polled by workers { "ExtractImageMetadata": { "Type": "Task", "Resource": "arn:aws:states:mars-ter …", … } } Asynchronous task -
API actions on activity task Register Activity Task - Returns
ARN Poll For task (by ARN) Report Success Report Failure Report Heartbeat -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. O pen Source A pache License https://states-language.net/spec.html -
© 2019, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. Amazon Confidential and Trademark AWS Step Functions Demo -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. -
© 2019, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. Amazon Confidential and Trademark AWS Step Functions Case Studies -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. Vending Pass problem 1. Thirsty consumer presents card, can buy a drink with Vending Pass or debit 2. But mobile wallet display can get out of sync under certain conditions 3. -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. Vending Pass problem • Just wait for backend to catch up • Create a two-step state machine: • Wait (90 seconds) • Update mobile wallet • Cheap and simple! -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. • Transcode 350 clips/day into 14 formats, fast • It’s all done with FFmpeg. The processing time is just about 100% of the video length • Aargh! Video processing problem -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. • Derive keyframe locations within the source • Split the source at the keyframes • Process segments (typically 0.5 sec per) in parallel • Concatenate segments • Elapsed time: ~20 min down to ~2 minutes Video processing solution -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. Details How to get from Lambda to Amazon S3: ffmpeg -f concat –safe 0 -I filelist.ffcat –c copy pipe:0 | aws s3 cp – s3://output-bucket/output-file.mp4 -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. Details -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. Let’s peek at the code! "Init": { "Type": "Pass", "ResultPath": "$.operation", "Result": "prepare-split", "Next": "PrepareSplit" }, "PrepareSplit": { "Type": "Task", "Resource": "arn:…:ffmpeg-18HFP9FXFL6P", "Next": "SplitPrepared" }, "SplitPrepared": { "Type": "Pass", "ResultPath": "$.operation", "Result": "perform-split", "Next": "PerformSplit" }, "PerformSplit": { "Type": "Task", "Resource": "arn:…:ffmpeg-18HFP9FXFL6P", "Next": "SplitPerformed" }, "SplitPerformed": { "Type": "Pass", "ResultPath": "$.operation", "Result": "poll-results", "Next": "PreparePoll" }, "PreparePoll": { "Type": "Pass", "ResultPath": "$.pollstatus", "Result": "Submitted", "Next": "WaitForResults" } Hmm…the same function, throughout -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. Let’s peek at the code! "WaitForResults": { "Type": "Choice", "Choices": [ { "Variable": "$.pollstatus", "StringEquals": "Complete", "Next": "SegmentsCompleteState" }, { "Variable": "$.pollstatus", "StringEquals": "Progressing", "Next": "loop_wait_using_seconds" }, { "Variable": "$.pollstatus", "StringEquals": "Submitted", "Next": "loop_wait_using_seconds" } ] } "loop_wait_using_seconds": { "Type": "Wait", "Seconds": 3, "Next": "loop" }, "loop": { "Type": "Task", "Resource": "arn:…:ffmpeg-18HFP9FXFL6P", "Next": "WaitForResults" }, "SegmentsCompleteState": { "Type": "Pass", "ResultPath": "$.operation", "Result": "concat", "Next": "CompleteState" }, "CompleteState": { "Type": "Task", "Resource": " arn:…:ffmpeg-18HFP9FXFL6P", "End": true } Counts the segments, sets $.pollstatus Stitches segments together -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. Takeaways Use Amazon S3 data events to trigger parallel Lambda processing: win Use Amazon S3 URLs to stream video to Lambda: win Scaling to 1,000 Lambdas, rather than 1,000 EC2 instances: win Process video segments in parallel: win -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. More state machines Image Recognition and Processing Backend EBS Snapshot Management https://aws.amazon.com/step-functions/getting-started -
© 2018, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. What are you building? #stepfunctions https://console.aws.amazon.com/states/ -
© 2019, Amazon Web Services, Inc. or its Affiliates. All
rights reserved. Amazon Confidential and Trademark Thank you! @danilop