Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building Advanced Serverless Applications with AWS Step Functions

Building Advanced Serverless Applications with AWS Step Functions

Builders' Day, Edinburgh, February 21st, 2018

Danilo Poccia

February 21, 2018
Tweet

More Decks by Danilo Poccia

Other Decks in Programming

Transcript

  1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Building Advanced Serverless
    Applications with AWS Step Functions
    Danilo Poccia
    Technical Evangelist
    [email protected]
    @danilop
    danilop

    View Slide

  2. λ
    λ
    λ
    DBMS
    λ
    λ
    λ
    λ
    λ
    λ λ
    λ
    λ
    Queue
    Modern
    serverless
    app

    View Slide

  3. Modern
    serverless
    app

    View Slide

  4. "I want to sequence functions"
    "I want to select functions based on data"
    "I want to retry functions"
    "I want try/catch/finally"
    Functions into apps
    "I have code that runs for hours"
    "I want to run functions in parallel"

    View Slide

  5. Coordination must-haves
    • Scales out
    • Doesn’t lose state
    • Deals with errors/timeouts
    • Easy to build & operate
    • Auditable

    View Slide

  6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

    View Slide

  7. Application Lifecycle in AWS Step Functions
    Visualize in the
    Console
    Define in JSON Monitor
    Executions

    View Slide

  8. Define in JSON and Then Visualize in the Console
    {
    "Comment": "Hello World Example",
    "StartAt" : "HelloWorld",
    "States" : {
    "HelloWorld" : {
    "Type" : "Task",
    "Resource" : "${lambdaArn}",
    "End" : true
    }
    }
    }

    View Slide

  9. Execute One or One Million
    Start
    End
    HelloWorld
    Start
    End
    HelloWorld
    Start
    End
    HelloWorld
    Start
    End
    HelloWorld
    Start
    End
    HelloWorld
    Start
    End
    HelloWorld
    Start
    End
    HelloWorld
    Start
    End
    HelloWorld

    View Slide

  10. Maximum Execution Time
    AWS Lambda
    Functions
    5 minutes
    AWS Step Functions
    State Machines
    1 year

    View Slide

  11. Monitor Executions from the Console

    View Slide

  12. Seven State Types
    Task A single unit of work
    Choice Adds branching logic
    Parallel Fork and join the data across tasks
    Wait Delay for a specified time
    Fail Stops an execution and marks it as a failure
    Succeed Stops an execution successfully
    Pass Passes its input to its output

    View Slide

  13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    ExtractImageMetadata
    ImageTypeCheck
    TransformMetadata
    Rekognition Thumbnail
    StoreMetadata
    Start
    End
    Synchronous task
    Task: a single unit of work
    Synchronous: Lambda
    {
    "ExtractImageMetadata": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:mars-ter …",

    }
    }

    View Slide

  14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    ExtractImageMetadata
    TransformMetadata
    Rekognition Thumbnail
    StoreMetadata
    Start
    End
    Task: a single unit of work
    Asynchronous: "Activity Task", Polled by workers
    {
    "ExtractImageMetadata": {
    "Type": "Task",
    "Resource": "arn:aws:states:mars-ter …",

    }
    }
    Asynchronous task

    View Slide

  15. API actions on activity task
    Register Activity Task - Returns ARN
    Poll For task (by ARN)
    Report Success
    Report Failure
    Report Heartbeat

    View Slide

  16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    O
    pen
    Source
    A
    pache
    License
    https://states-language.net/spec.html

    View Slide

  17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Integrate with other AWS services
    • Create state machines and activities with AWS CloudFormation
    • Call AWS Step Functions with Amazon API Gateway
    • Start state machines in response to events, or on a schedule,
    with Amazon CloudWatch Events
    • Monitor state machine executions with Amazon CloudWatch
    • Log API calls with AWS CloudTrail
    • Start state machine using AWS Lambda

    View Slide

  18. Build Visual Workflows from State Types
    Task
    Choice
    Fail
    Parallel

    View Slide

  19. Serverless Human Approval Tasks
    https://aws.amazon.com/blogs/compute/
    implementing-serverless-manual-approval-steps-in-aws-step-functions-and-amazon-api-gateway/

    View Slide

  20. Image Recognition and Processing Backend
    Task
    Choice
    Fail
    Parallel
    https://github.com/awslabs/lambda-refarch-imagerecognition

    View Slide

  21. Amazon EBS Snapshot Management
    State Machine State Machine
    create-snapshot
    complete event
    copy-snapshot
    complete event
    Primary Region DR Region
    Parallel State
    Choice
    Tag Snapshot
    Count Snapshots
    CopyToDRRegion
    Success
    State
    Delete
    Snapshots
    Choice
    Tag SnapshotCopy
    Count Snapshots
    Pass State
    Delete
    Snapshots
    Notification
    Topic
    Errors Sent
    to SNS
    Notification
    Topic
    Errors Sent
    to SNS
    https://github.com/awslabs/aws-step-functions-ebs-snapshot-mgmt
    1. Tag
    2. Count
    3. Copy to DR region
    4. Delete Expired
    Whenever a new EBS
    snapshot completes:
    State Machines
    invoked by Amazon
    CloudWatch Events

    View Slide

  22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Demo:
    AWS Step Functions

    View Slide

  23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

    View Slide

  24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Case Studies

    View Slide

  25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Vending Pass problem
    1. Thirsty consumer presents
    card, can buy a drink with
    Vending Pass or debit
    2. But mobile wallet display can
    get out of sync under certain
    conditions
    3.

    View Slide

  26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Vending Pass problem
    • Just wait for backend
    to catch up
    • Create a two-step state
    machine:
    • Wait (90 seconds)
    • Update mobile wallet
    • Cheap and simple!

    View Slide

  27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    • Transcode 350 clips/day into
    14 formats, fast
    • It’s all done with FFmpeg. The
    processing time is just about
    100% of the video length
    • Aargh!
    Video processing problem

    View Slide

  28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    • Derive keyframe locations
    within the source
    • Split the source at the
    keyframes
    • Process segments (typically
    0.5 sec per) in parallel
    • Concatenate segments
    • Elapsed time: ~20 min down
    to ~2 minutes
    Video processing solution

    View Slide

  29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Details
    How to get from Lambda to Amazon S3:
    ffmpeg -f concat –safe 0 -I filelist.ffcat –c copy pipe:0 |
    aws s3 cp – s3://output-bucket/output-file.mp4

    View Slide

  30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Details

    View Slide

  31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Let’s peek at the code!
    "Init": {
    "Type": "Pass",
    "ResultPath": "$.operation",
    "Result": "prepare-split",
    "Next": "PrepareSplit"
    },
    "PrepareSplit": {
    "Type": "Task",
    "Resource": "arn:…:ffmpeg-18HFP9FXFL6P",
    "Next": "SplitPrepared"
    },
    "SplitPrepared": {
    "Type": "Pass",
    "ResultPath": "$.operation",
    "Result": "perform-split",
    "Next": "PerformSplit"
    },
    "PerformSplit": {
    "Type": "Task",
    "Resource": "arn:…:ffmpeg-18HFP9FXFL6P",
    "Next": "SplitPerformed"
    },
    "SplitPerformed": {
    "Type": "Pass",
    "ResultPath": "$.operation",
    "Result": "poll-results",
    "Next": "PreparePoll"
    },
    "PreparePoll": {
    "Type": "Pass",
    "ResultPath": "$.pollstatus",
    "Result": "Submitted",
    "Next": "WaitForResults"
    }
    Hmm…the same function,
    throughout

    View Slide

  32. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Let’s peek at the code!
    "WaitForResults": {
    "Type": "Choice",
    "Choices": [
    {
    "Variable": "$.pollstatus",
    "StringEquals": "Complete",
    "Next": "SegmentsCompleteState"
    },
    {
    "Variable": "$.pollstatus",
    "StringEquals": "Progressing",
    "Next": "loop_wait_using_seconds"
    },
    {
    "Variable": "$.pollstatus",
    "StringEquals": "Submitted",
    "Next": "loop_wait_using_seconds"
    }
    ]
    }
    "loop_wait_using_seconds": {
    "Type": "Wait",
    "Seconds": 3,
    "Next": "loop"
    },
    "loop": {
    "Type": "Task",
    "Resource": "arn:…:ffmpeg-18HFP9FXFL6P",
    "Next": "WaitForResults"
    },
    "SegmentsCompleteState": {
    "Type": "Pass",
    "ResultPath": "$.operation",
    "Result": "concat",
    "Next": "CompleteState"
    },
    "CompleteState": {
    "Type": "Task",
    "Resource": " arn:…:ffmpeg-18HFP9FXFL6P",
    "End": true
    }
    Counts the segments,
    sets $.pollstatus
    Stitches segments together

    View Slide

  33. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Takeaways
    Use Amazon S3
    data events to
    trigger parallel
    Lambda
    processing: win
    Use Amazon S3
    URLs to stream
    video to
    Lambda: win
    Scaling to 1,000
    Lambdas, rather
    than 1,000 EC2
    instances: win
    Process video
    segments in
    parallel: win

    View Slide

  34. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    1. Sometimes Lambda is best, sometimes ECS
    2. Previously, all tied together with
    pub/sub and
    procedural code
    3. Aargh!
    Media transcoding problem

    View Slide

  35. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    1. Use state machines to pick
    execution engine
    2. Use CloudWatch Events for
    messaging and triggering
    Step Functions
    Media transcoding solution

    View Slide

  36. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Let’s peek at the code!
    "Encoder Decider": {
    "Type": "Choice",
    "Default": "Run ECS Encoder",
    "Choices": [
    {
    "Next": "Run Lambda Encoder",
    "And": [
    {
    "Variable": "$.asset.size",
    "NumericLessThanEquals": 2000000000
    },
    {
    "Variable": "$.asset.duration_ms",
    "NumericLessThanEquals": 10000
    }
    ]
    }
    ]
    }
    Hmm…maybe "Default"
    should have been called "Else"

    View Slide

  37. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Let’s peek at the code!
    "Run ECS Encoder": {
    "Type": "Task",
    "Resource": "arn:…SubmitECSTask",
    "Retry": [
    {
    "ErrorEquals": [
    "NoResourceInCluster"
    ],
    "IntervalSeconds": 5,
    "MaxAttempts": 720,
    "BackoffRate": 1.0
    }
    ],
    "ResultPath": "$.task",
    "Next": "Wait X Seconds"
    }
    Hmm…every 5 seconds,
    for an hour?!
    "Wait X Seconds": {
    "Type": "Wait",
    "SecondsPath": "$.task.wait_time",
    "Next": "Get ECS Task Status"
    },
    "Get ECS Task Status": {
    "Type": "Task",
    "Resource": "arn:…ECSTaskStatus",
    "Next": "ECS Task Complete?",
    "ResultPath": "$.task.status"
    }

    View Slide

  38. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Let’s peek at the code!
    "ECS Task Complete?": {
    "Type": "Choice",
    "Choices": [
    {
    "And": [
    {
    "Variable": "$.task.status",
    "StringEquals": "FAILED"
    },
    {
    "Variable": "$.task.attempt",
    "NumericGreaterThanEquals": 3
    }
    ],
    "Next": "Fire Failed Event"
    },
    {
    "Variable": "$.task.status",
    "StringEquals": "FAILED",
    "Next": "Run ECS Encoder"
    },
    {
    "Variable": "$.task.status",
    "StringEquals": "SUCCEEDED",
    "Next": "Fire Successful Event"
    }
    ],
    "Default": "Wait X Seconds"
    },
    "Fire Successful Event": {
    "Type": "Task",
    "Resource": "aws:…SendSuccessfulEvent",
    "End": true
    },
    "Fire Failed Event": {
    "Type": "Task",
    "Resource": "aws:…SendFailedEvent",
    "End": true
    }
    These fire
    CloudWatch
    Events

    View Slide

  39. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Takeaways
    Passing a
    correlation ID
    through a complex
    serverless app: win
    SAM to make
    code changes
    auditable: win
    Code Pipeline
    for CI/CD: win
    CloudWatch
    Events pattern
    matching: win

    View Slide

  40. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    More state machines
    Image Recognition and Processing Backend
    EBS Snapshot Management
    https://aws.amazon.com/step-functions/getting-started

    View Slide

  41. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    What are you building? #stepfunctions
    https://console.aws.amazon.com/states/

    View Slide

  42. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Building Advanced Serverless
    Applications with AWS Step Functions
    Danilo Poccia
    Technical Evangelist
    [email protected]
    @danilop
    danilop

    View Slide