Building a Processing Pipeline with Serverless and CloudFormation 

Serverless Focus on application logic, not server provisioning or maintenance. 

CloudFormation • Infrastructure as code • Fine-grained security policies 

SAM Superset of CloudFormation to define serverless applications, including some convenient resource types: • Function: simpler Lambda definition • API: simpler API Gateway definition • SimpleTable: DynamoDB 

Day-to-day Serverless 

Run Invoke Lambda functions and start API locally using AWS SAM CLI. sam local invoke ProcessMoveToEnd --debug \ -t ../../template.yaml \ -e testdata/sample-event.json 

Test and Debug • Unit tests • Dependency injection • SAM CLI debugging support 

Deploy Use SAM CLI to package and deploy the app. sam package \ --template-file template.yaml \ --s3-bucket my-bucket \ --output-template-file template.packaged.yaml aws cloudformation deploy \ --template-file template.packaged.yaml \ --stack-name MyApp 

Nested stacks • Separate concerns, e.g. vpc, bastion, database, application1 • Prevent accidental deletion Vpc: Type: AWS::CloudFormation::Stack Properties: TemplateURL: !Sub https://s3.amazonaws.com/${TemplateBucket}/templates/vpc.cfn.yaml Parameters: AvailabilityZone1: !Ref AvailabilityZone1 AvailabilityZone2: !Ref AvailabilityZone2 SSHFrom: !Ref SSHFrom ELBIngressPort: !Ref ELBIngressPort AppIngressPort: !Ref AppIngressPort 1 AWS startup kit templates 

SAM in nested stacks SAM templates cannot be nested directly – expand them first2. sam-translate.py --input-file sam.yaml --output-file sam.cfn.json Include expanded template in root stack. MyApp: Type: AWS::CloudFormation::Stack Properties: TemplateURL: !Sub https://s3.amazonaws.com/${TemplateBucket}/sam.cfn.json 2 sam-translate.py available here 

Develop, staging and production Use separate stacks for each environment. Managing multi-environment serverless architecture using AWS —  an investigation — ! 

Demo time 

Demo time First compile the Lambda functions since we are using Go. GOOS=linux go build -o ./cmd/move-to-end/move-to-end ./cmd/move-to-end GOOS=linux go build -o ./cmd/process-execute/process-execute ./cmd/process-execute 

Demo time Invoke a Lambda function locally. sam local invoke ProcessMoveToEnd --debug \ -t ../../template.yaml \ -e testdata/sample-event.json 

Demo time Package the app to S3. sam package --template-file ./template.yaml \ --s3-bucket playground-artifacts \ --output-template-file template.packaged.yaml 

Demo time Deploy the app aws cloudformation deploy --template-file "$(PWD)/template.packaged.yaml" \ --stack-name Playground \ --capabilities CAPABILITY_NAMED_IAM 

Demo time Invoke ProcessExectute Lambda function with test event. { "input": "hello,world" } 

Demo time Invoke the state machine directly. aws stepfunctions start-execution --state-machine-arn \ --input "{\"jobs\": [{\"input\": \"First job\", \"done\": false}, {\"input\": \"Second job\", \"done\": false}]}" 

Trigger it from S3 Set up S3 notifications in SAM template. Assets: Type: AWS::S3::Bucket DeletionPolicy: Retain Properties: BucketName: "playground-assets" NotificationConfiguration: LambdaConfigurations: - Function: !GetAtt ProcessExecute.Arn Event: "s3:ObjectCreated:*" Filter: S3Key: Rules: - Name: suffix Value: completed 

Trigger it from S3 Update Lambda handler to accept S3 event. func handle(e events.S3Event) error { ... } 

Thank you ! https://medium.com/@christianklotz/c185c2d2608 https://medium.com/@sgarcez/6cd6501d261e https://github.com/christianklotz/aws-step-functions-iterate- sample https://github.com/awslabs/video-on-demand-on-aws https://github.com/aws-samples/startup-kit-templates ! @christianklotz