Slide 1
Slide 1 text
Orchestrating Serverless
with AWS Step Functions
Slide 2
Slide 2 text
No content
Slide 3
Slide 3 text
matt williams - @technovangelist
Baltimore DevOps
Slide 4
Slide 4 text
No content
Slide 5
Slide 5 text
No content
Slide 6
Slide 6 text
No content
Slide 7
Slide 7 text
matt williams - @technovangelist
Baltimore DevOps
Slide 8
Slide 8 text
matt williams - @technovangelist
Baltimore DevOps
…but I didn’t bring a laptop
Slide 9
Slide 9 text
matt williams - @technovangelist
Baltimore DevOps
Who am I
Matt Williams
Evangelist at Datadog
[email protected]
TW: technovangelist
YT: technovangelist
IG/GH/XYZ: technovangelist
Organizer of:
DevOps Days Boston 2017 & 2018
Serverless Days Boston 2018
Slide 10
Slide 10 text
matt williams - @technovangelist
Baltimore DevOps
Who is Datadog
SaaS-based Monitoring & Analytics
Infrastructure, APM, Logs
Open Source Agent
Trillions of data
points per day
We are hiring!!
Slide 11
Slide 11 text
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
I like to talk about Serverless…
Slide 12
Slide 12 text
matt williams - @technovangelist
Baltimore DevOps
What is AWS Lambda
Its all about the function
Serverless
As far as you are concerned
Don’t pay for idle
Scale with ease
FaaS
Node 8.10/6.10, Golang, C# 1/2/2.1, Python 2.7/3.6, Java8
Trigger-based
Slide 13
Slide 13 text
matt williams - @technovangelist
Baltimore DevOps
Trigger Based?
S3
DynamoDB
Kinesis
SNS
SES
Cognito
CloudFormation
Cloudwatch Logs & Events
CodeCommit
Config
Alexa
Lex
API Gateway
IoT
CloudFront
Lambda
http://tvl.st/khvid
Slide 14
Slide 14 text
matt williams - @technovangelist
Baltimore DevOps
AWS
Lambda
Slide 15
Slide 15 text
matt williams - @technovangelist
Baltimore DevOps
Slide 16
Slide 16 text
matt williams - @technovangelist
Baltimore DevOps
That’s pretty cool right? But not perfect
You still have to configure everything else
S3
DynamoDB
API Gateway
etc
SAM
Cloudformation
Serverless
Slide 17
Slide 17 text
matt williams - @technovangelist
Baltimore DevOps
AWS
Lambda
Slide 18
Slide 18 text
matt williams - @technovangelist
Baltimore DevOps
…but serverless !== ec2
Slide 19
Slide 19 text
matt williams - @technovangelist
Baltimore DevOps
…but serverless !== applications
Slide 20
Slide 20 text
matt williams - @technovangelist
Baltimore DevOps
…applications are complicated
Slide 21
Slide 21 text
matt williams - @technovangelist
Baltimore DevOps
…serverless functions should
respect the single responsibility
principle…
Slide 22
Slide 22 text
matt williams - @technovangelist
Baltimore DevOps
Slide 23
Slide 23 text
matt williams - @technovangelist
Baltimore DevOps
Slide 24
Slide 24 text
matt williams - @technovangelist
Baltimore DevOps
we can create many serverless functions
Slide 25
Slide 25 text
matt williams - @technovangelist
Baltimore DevOps
we can create many serverless functions
but how do we orchestrate them?
Slide 26
Slide 26 text
matt williams - @technovangelist
Baltimore DevOps
Event/Message/Database as Traffic Cop
Amazon Kinesis Amazon SQS Amazon DynamoDB
Slide 27
Slide 27 text
matt williams - @technovangelist
Baltimore DevOps
https://github.com/Nordstrom/hello-retail
Slide 28
Slide 28 text
matt williams - @technovangelist
Baltimore DevOps
Enter Step Functions
Slide 29
Slide 29 text
matt williams - @technovangelist
Baltimore DevOps
Step Function Concepts
State Machine
States
Task
Choice
Succeed/Fail
Pass
Wait
Parallel
Transitions
Executions
You define the State Machine with the
Amazon States Language
A diagram will be generated based on that
structure
Slide 30
Slide 30 text
matt williams - @technovangelist
Baltimore DevOps
State Example: Task
Slide 31
Slide 31 text
matt williams - @technovangelist
Baltimore DevOps
State Example: Choice
Slide 32
Slide 32 text
matt williams - @technovangelist
Baltimore DevOps
State Example: Succeed / Fail
Slide 33
Slide 33 text
matt williams - @technovangelist
Baltimore DevOps
State Example: Pass
Slide 34
Slide 34 text
matt williams - @technovangelist
Baltimore DevOps
State Example: Wait
Slide 35
Slide 35 text
matt williams - @technovangelist
Baltimore DevOps
State Example: Parallel
Slide 36
Slide 36 text
matt williams - @technovangelist
Baltimore DevOps
Many ways to build AWS Step Functions
I used Serverless Framework (http://tvl.st/sls)
Slide 37
Slide 37 text
matt williams - @technovangelist
Baltimore DevOps
Benefits of Serverless Framework
Define everything in a single file:
IAM Role Statements
Function Handlers
Function Triggers
State Machine States
AWS CloudFormation for other things
Slide 38
Slide 38 text
matt williams - @technovangelist
Baltimore DevOps
Amazon ECR
AWS
Lambda
Amazon
S3
Amazon
DynamoDB
Amazon
CloudFront
AWS
CodeBuild
AWS
CodeCommit
AWS
CloudFormation
IAM
Amazon API
Gateway*
AWS Step
Functions
Amazon
SNS
Slide 39
Slide 39 text
matt williams - @technovangelist
Baltimore DevOps
Slide 40
Slide 40 text
No content
Slide 41
Slide 41 text
matt williams - @technovangelist
Baltimore DevOps
Slide 42
Slide 42 text
matt williams - @technovangelist
Baltimore DevOps
Slide 43
Slide 43 text
matt williams - @technovangelist
Baltimore DevOps
Slide 44
Slide 44 text
matt williams - @technovangelist
Baltimore DevOps
Slide 45
Slide 45 text
matt williams - @technovangelist
Baltimore DevOps
1.Use ML to figure out which slide
2.Is this slide different than the last time?
3.If so, pull slide data from Dynamo
4.If slide different then create document
in s3
5.Build the site
6.Wait for it
1.Is it built
7.Done
Slide 46
Slide 46 text
matt williams - @technovangelist
Baltimore DevOps
Type: Task
Resource: Function
Slide 47
Slide 47 text
matt williams - @technovangelist
Baltimore DevOps
Slide 48
Slide 48 text
No content
Slide 49
Slide 49 text
matt williams - @technovangelist
Baltimore DevOps
Type: Task
Resource: Function
Slide 50
Slide 50 text
matt williams - @technovangelist
Baltimore DevOps
Slide 51
Slide 51 text
matt williams - @technovangelist
Baltimore DevOps
Slide 52
Slide 52 text
matt williams - @technovangelist
Baltimore DevOps
Type: Task
Resource: Function
Slide 53
Slide 53 text
matt williams - @technovangelist
Baltimore DevOps
Slide 54
Slide 54 text
No content
Slide 55
Slide 55 text
matt williams - @technovangelist
Baltimore DevOps
How about monitoring?
We monitor serverless via Logs
MONITORING|unix_epoch_timestamp|value|count|
my.metric.name|#tag1:value,tag2
console.log(monitoringstring);
Slide 56
Slide 56 text
matt williams - @technovangelist
Baltimore DevOps
What are the AWS Lambda Metrics
aws.lambda.duration, min, max, sum
aws.lambda.errors
aws.lambda.invocations
aws.lambda.throttles
aws.lambda.iterator_age
Slide 57
Slide 57 text
matt williams - @technovangelist
Baltimore DevOps
What are the AWS Step Function Metrics
aws.states.execution_time (.maximum, .minimum)
aws.states.executions_aborted, failed, started, succeeded, timed_out
aws.states.lambda_functions_scheduled, started, succeeded, timed_out
aws.states.execution_throttled
aws.states.lambda_function_time (.maximum, .minimum)
aws.states.lambda_function_run_time (.maximum, .minimum)
aws.states.lambda_function_schedule_time (.maximum, .minimum)
aws.states.lambda_functions_started
Slide 58
Slide 58 text
matt williams - @technovangelist
Baltimore DevOps
Slide 59
Slide 59 text
matt williams - @technovangelist
Baltimore DevOps
Slide 60
Slide 60 text
matt williams - @technovangelist
Baltimore DevOps
Slide 61
Slide 61 text
matt williams - @technovangelist
Baltimore DevOps
Slide 62
Slide 62 text
matt williams - @technovangelist
Baltimore DevOps
Slide 63
Slide 63 text
matt williams - @technovangelist
Baltimore DevOps
Slide 64
Slide 64 text
matt williams - @technovangelist
Baltimore DevOps
AWS Lambda and AWS Step Function Challenges
• Logging and debugging
• Don’t trust any function when it claims a step is complete
• Cold start depends on your use case
• Every function should do one thing…but granularity
depends
Slide 65
Slide 65 text
matt williams - @technovangelist
Baltimore DevOps
Matt Williams
Evangelist at Datadog
[email protected]
all the socials: technovangelist
Organizer of DevOps Days Boston
2017 & 2018