Building Advanced Serverless Applications with AWS Step Functions

Builders' Day, Edinburgh, February 21st, 2018

Transcript

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
   Building Advanced Serverless
   Applications with AWS Step Functions
   Danilo Poccia
   Technical Evangelist
   danilop@amazon.com
   @danilop
   danilop
   

   View Slide

2. λ
   λ
   λ
   DBMS
   λ
   λ
   λ
   λ
   λ
   λ λ
   λ
   λ
   Queue
   Modern
   serverless
   app
   

   View Slide

3. Modern
   serverless
   app
   

   View Slide

4. "I want to sequence functions"
   "I want to select functions based on data"
   "I want to retry functions"
   "I want try/catch/finally"
   Functions into apps
   "I have code that runs for hours"
   "I want to run functions in parallel"
   

   View Slide

5. Coordination must-haves
   • Scales out
   • Doesn’t lose state
   • Deals with errors/timeouts
   • Easy to build & operate
   • Auditable
   

   View Slide

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
   

   View Slide

7. Application Lifecycle in AWS Step Functions
   Visualize in the
   Console
   Define in JSON Monitor
   Executions
   

   View Slide

8. Define in JSON and Then Visualize in the Console
   {
   "Comment": "Hello World Example",
   "StartAt" : "HelloWorld",
   "States" : {
   "HelloWorld" : {
   "Type" : "Task",
   "Resource" : "${lambdaArn}",
   "End" : true
   }
   }
   }
   

   View Slide

9. Execute One or One Million
   Start
   End
   HelloWorld
   Start
   End
   HelloWorld
   Start
   End
   HelloWorld
   Start
   End
   HelloWorld
   Start
   End
   HelloWorld
   Start
   End
   HelloWorld
   Start
   End
   HelloWorld
   Start
   End
   HelloWorld
   

   View Slide

10. Maximum Execution Time
    AWS Lambda
    Functions
    5 minutes
    AWS Step Functions
    State Machines
    1 year
    

    View Slide

11. Monitor Executions from the Console
    

    View Slide

12. Seven State Types
    Task A single unit of work
    Choice Adds branching logic
    Parallel Fork and join the data across tasks
    Wait Delay for a specified time
    Fail Stops an execution and marks it as a failure
    Succeed Stops an execution successfully
    Pass Passes its input to its output
    

    View Slide

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    ExtractImageMetadata
    ImageTypeCheck
    TransformMetadata
    Rekognition Thumbnail
    StoreMetadata
    Start
    End
    Synchronous task
    Task: a single unit of work
    Synchronous: Lambda
    {
    "ExtractImageMetadata": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:mars-ter …",
    …
    }
    }
    

    View Slide

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    ExtractImageMetadata
    TransformMetadata
    Rekognition Thumbnail
    StoreMetadata
    Start
    End
    Task: a single unit of work
    Asynchronous: "Activity Task", Polled by workers
    {
    "ExtractImageMetadata": {
    "Type": "Task",
    "Resource": "arn:aws:states:mars-ter …",
    …
    }
    }
    Asynchronous task
    

    View Slide

15. API actions on activity task
    Register Activity Task - Returns ARN
    Poll For task (by ARN)
    Report Success
    Report Failure
    Report Heartbeat
    

    View Slide

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    O
    pen
    Source
    A
    pache
    License
    https://states-language.net/spec.html
    

    View Slide

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Integrate with other AWS services
    • Create state machines and activities with AWS CloudFormation
    • Call AWS Step Functions with Amazon API Gateway
    • Start state machines in response to events, or on a schedule,
    with Amazon CloudWatch Events
    • Monitor state machine executions with Amazon CloudWatch
    • Log API calls with AWS CloudTrail
    • Start state machine using AWS Lambda
    

    View Slide

18. Build Visual Workflows from State Types
    Task
    Choice
    Fail
    Parallel
    

    View Slide

19. Serverless Human Approval Tasks
    https://aws.amazon.com/blogs/compute/
    implementing-serverless-manual-approval-steps-in-aws-step-functions-and-amazon-api-gateway/
    

    View Slide

20. Image Recognition and Processing Backend
    Task
    Choice
    Fail
    Parallel
    https://github.com/awslabs/lambda-refarch-imagerecognition
    

    View Slide

21. Amazon EBS Snapshot Management
    State Machine State Machine
    create-snapshot
    complete event
    copy-snapshot
    complete event
    Primary Region DR Region
    Parallel State
    Choice
    Tag Snapshot
    Count Snapshots
    CopyToDRRegion
    Success
    State
    Delete
    Snapshots
    Choice
    Tag SnapshotCopy
    Count Snapshots
    Pass State
    Delete
    Snapshots
    Notification
    Topic
    Errors Sent
    to SNS
    Notification
    Topic
    Errors Sent
    to SNS
    https://github.com/awslabs/aws-step-functions-ebs-snapshot-mgmt
    1. Tag
    2. Count
    3. Copy to DR region
    4. Delete Expired
    Whenever a new EBS
    snapshot completes:
    State Machines
    invoked by Amazon
    CloudWatch Events
    

    View Slide

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Demo:
    AWS Step Functions
    

    View Slide

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    

    View Slide

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Case Studies
    

    View Slide

25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Vending Pass problem
    1. Thirsty consumer presents
    card, can buy a drink with
    Vending Pass or debit
    2. But mobile wallet display can
    get out of sync under certain
    conditions
    3.
    

    View Slide

26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Vending Pass problem
    • Just wait for backend
    to catch up
    • Create a two-step state
    machine:
    • Wait (90 seconds)
    • Update mobile wallet
    • Cheap and simple!
    

    View Slide

27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    • Transcode 350 clips/day into
    14 formats, fast
    • It’s all done with FFmpeg. The
    processing time is just about
    100% of the video length
    • Aargh!
    Video processing problem
    

    View Slide

28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    • Derive keyframe locations
    within the source
    • Split the source at the
    keyframes
    • Process segments (typically
    0.5 sec per) in parallel
    • Concatenate segments
    • Elapsed time: ~20 min down
    to ~2 minutes
    Video processing solution
    

    View Slide

29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Details
    How to get from Lambda to Amazon S3:
    ffmpeg -f concat –safe 0 -I filelist.ffcat –c copy pipe:0 |
    aws s3 cp – s3://output-bucket/output-file.mp4
    

    View Slide

30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Details
    

    View Slide

31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Let’s peek at the code!
    "Init": {
    "Type": "Pass",
    "ResultPath": "$.operation",
    "Result": "prepare-split",
    "Next": "PrepareSplit"
    },
    "PrepareSplit": {
    "Type": "Task",
    "Resource": "arn:…:ffmpeg-18HFP9FXFL6P",
    "Next": "SplitPrepared"
    },
    "SplitPrepared": {
    "Type": "Pass",
    "ResultPath": "$.operation",
    "Result": "perform-split",
    "Next": "PerformSplit"
    },
    "PerformSplit": {
    "Type": "Task",
    "Resource": "arn:…:ffmpeg-18HFP9FXFL6P",
    "Next": "SplitPerformed"
    },
    "SplitPerformed": {
    "Type": "Pass",
    "ResultPath": "$.operation",
    "Result": "poll-results",
    "Next": "PreparePoll"
    },
    "PreparePoll": {
    "Type": "Pass",
    "ResultPath": "$.pollstatus",
    "Result": "Submitted",
    "Next": "WaitForResults"
    }
    Hmm…the same function,
    throughout
    

    View Slide

32. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Let’s peek at the code!
    "WaitForResults": {
    "Type": "Choice",
    "Choices": [
    {
    "Variable": "$.pollstatus",
    "StringEquals": "Complete",
    "Next": "SegmentsCompleteState"
    },
    {
    "Variable": "$.pollstatus",
    "StringEquals": "Progressing",
    "Next": "loop_wait_using_seconds"
    },
    {
    "Variable": "$.pollstatus",
    "StringEquals": "Submitted",
    "Next": "loop_wait_using_seconds"
    }
    ]
    }
    "loop_wait_using_seconds": {
    "Type": "Wait",
    "Seconds": 3,
    "Next": "loop"
    },
    "loop": {
    "Type": "Task",
    "Resource": "arn:…:ffmpeg-18HFP9FXFL6P",
    "Next": "WaitForResults"
    },
    "SegmentsCompleteState": {
    "Type": "Pass",
    "ResultPath": "$.operation",
    "Result": "concat",
    "Next": "CompleteState"
    },
    "CompleteState": {
    "Type": "Task",
    "Resource": " arn:…:ffmpeg-18HFP9FXFL6P",
    "End": true
    }
    Counts the segments,
    sets $.pollstatus
    Stitches segments together
    

    View Slide

33. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Takeaways
    Use Amazon S3
    data events to
    trigger parallel
    Lambda
    processing: win
    Use Amazon S3
    URLs to stream
    video to
    Lambda: win
    Scaling to 1,000
    Lambdas, rather
    than 1,000 EC2
    instances: win
    Process video
    segments in
    parallel: win
    

    View Slide

34. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    1. Sometimes Lambda is best, sometimes ECS
    2. Previously, all tied together with
    pub/sub and
    procedural code
    3. Aargh!
    Media transcoding problem
    

    View Slide

35. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    1. Use state machines to pick
    execution engine
    2. Use CloudWatch Events for
    messaging and triggering
    Step Functions
    Media transcoding solution
    

    View Slide

36. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Let’s peek at the code!
    "Encoder Decider": {
    "Type": "Choice",
    "Default": "Run ECS Encoder",
    "Choices": [
    {
    "Next": "Run Lambda Encoder",
    "And": [
    {
    "Variable": "$.asset.size",
    "NumericLessThanEquals": 2000000000
    },
    {
    "Variable": "$.asset.duration_ms",
    "NumericLessThanEquals": 10000
    }
    ]
    }
    ]
    }
    Hmm…maybe "Default"
    should have been called "Else"
    

    View Slide

37. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Let’s peek at the code!
    "Run ECS Encoder": {
    "Type": "Task",
    "Resource": "arn:…SubmitECSTask",
    "Retry": [
    {
    "ErrorEquals": [
    "NoResourceInCluster"
    ],
    "IntervalSeconds": 5,
    "MaxAttempts": 720,
    "BackoffRate": 1.0
    }
    ],
    "ResultPath": "$.task",
    "Next": "Wait X Seconds"
    }
    Hmm…every 5 seconds,
    for an hour?!
    "Wait X Seconds": {
    "Type": "Wait",
    "SecondsPath": "$.task.wait_time",
    "Next": "Get ECS Task Status"
    },
    "Get ECS Task Status": {
    "Type": "Task",
    "Resource": "arn:…ECSTaskStatus",
    "Next": "ECS Task Complete?",
    "ResultPath": "$.task.status"
    }
    

    View Slide

38. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Let’s peek at the code!
    "ECS Task Complete?": {
    "Type": "Choice",
    "Choices": [
    {
    "And": [
    {
    "Variable": "$.task.status",
    "StringEquals": "FAILED"
    },
    {
    "Variable": "$.task.attempt",
    "NumericGreaterThanEquals": 3
    }
    ],
    "Next": "Fire Failed Event"
    },
    {
    "Variable": "$.task.status",
    "StringEquals": "FAILED",
    "Next": "Run ECS Encoder"
    },
    {
    "Variable": "$.task.status",
    "StringEquals": "SUCCEEDED",
    "Next": "Fire Successful Event"
    }
    ],
    "Default": "Wait X Seconds"
    },
    "Fire Successful Event": {
    "Type": "Task",
    "Resource": "aws:…SendSuccessfulEvent",
    "End": true
    },
    "Fire Failed Event": {
    "Type": "Task",
    "Resource": "aws:…SendFailedEvent",
    "End": true
    }
    These fire
    CloudWatch
    Events
    

    View Slide

39. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Takeaways
    Passing a
    correlation ID
    through a complex
    serverless app: win
    SAM to make
    code changes
    auditable: win
    Code Pipeline
    for CI/CD: win
    CloudWatch
    Events pattern
    matching: win
    

    View Slide

40. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    More state machines
    Image Recognition and Processing Backend
    EBS Snapshot Management
    https://aws.amazon.com/step-functions/getting-started
    

    View Slide

41. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    What are you building? #stepfunctions
    https://console.aws.amazon.com/states/
    

    View Slide

42. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Building Advanced Serverless
    Applications with AWS Step Functions
    Danilo Poccia
    Technical Evangelist
    danilop@amazon.com
    @danilop
    danilop
    

    View Slide