Building a Processing Pipeline with Serverless and CloudFormation

An introduction to serverless application model including typical day-to-day issues and a demo project using AWS Step Functions.

CloudFormation is the AWS version of infrastructure as code and SAM (Serverless Application Model) is a superset of CloudFormation aimed at developers who want to build "serverless" applications, i.e. focus on the business logic without worrying too much about provisioning and maintaining servers.

This kind of application development does come with its own set of challenges, from testing to deployment to the integration into other parts of the infrastructure.

The demo project using AWS Step Functions shown in this presentation is available on GitHub at https://github.com/christianklotz/aws-step-functions-iterate-sample.

Transcript

1. Building a Processing Pipeline with Serverless and CloudFormation
2. Serverless Focus on application logic, not server provisioning or maintenance.
3. CloudFormation • Infrastructure as code • Fine-grained security policies
4. SAM Superset of CloudFormation to define serverless applications, including some convenient resource types: • Function: simpler Lambda definition • API: simpler API Gateway definition • SimpleTable: DynamoDB
5. Day-to-day Serverless
6. Run Invoke Lambda functions and start API locally using AWS SAM CLI. sam local invoke ProcessMoveToEnd --debug \ -t ../../template.yaml \ -e testdata/sample-event.json
7. Test and Debug • Unit tests • Dependency injection • SAM CLI debugging support
8. Deploy Use SAM CLI to package and deploy the app. sam package \ --template-file template.yaml \ --s3-bucket my-bucket \ --output-template-file template.packaged.yaml aws cloudformation deploy \ --template-file template.packaged.yaml \ --stack-name MyApp
9. Nested stacks • Separate concerns, e.g. vpc, bastion, database, application1 • Prevent accidental deletion Vpc: Type: AWS::CloudFormation::Stack Properties: TemplateURL: !Sub https://s3.amazonaws.com/${TemplateBucket}/templates/vpc.cfn.yaml Parameters: AvailabilityZone1: !Ref AvailabilityZone1 AvailabilityZone2: !Ref AvailabilityZone2 SSHFrom: !Ref SSHFrom ELBIngressPort: !Ref ELBIngressPort AppIngressPort: !Ref AppIngressPort 1 AWS startup kit templates
10. SAM in nested stacks SAM templates cannot be nested directly – expand them first2. sam-translate.py --input-file sam.yaml --output-file sam.cfn.json Include expanded template in root stack. MyApp: Type: AWS::CloudFormation::Stack Properties: TemplateURL: !Sub https://s3.amazonaws.com/${TemplateBucket}/sam.cfn.json 2 sam-translate.py available here
11. Develop, staging and production Use separate stacks for each environment. Managing multi-environment serverless architecture using AWS —  an investigation — !
12. Demo time
13. Demo time First compile the Lambda functions since we are using Go. GOOS=linux go build -o ./cmd/move-to-end/move-to-end ./cmd/move-to-end GOOS=linux go build -o ./cmd/process-execute/process-execute ./cmd/process-execute
14. Demo time Invoke a Lambda function locally. sam local invoke ProcessMoveToEnd --debug \ -t ../../template.yaml \ -e testdata/sample-event.json
15. Demo time Package the app to S3. sam package --template-file ./template.yaml \ --s3-bucket playground-artifacts \ --output-template-file template.packaged.yaml
16. Demo time Deploy the app aws cloudformation deploy --template-file "$(PWD)/template.packaged.yaml" \ --stack-name Playground \ --capabilities CAPABILITY_NAMED_IAM
17. Demo time Invoke ProcessExectute Lambda function with test event. { "input": "hello,world" }
18. Demo time Invoke the state machine directly. aws stepfunctions start-execution --state-machine-arn <STATE_MACHINE_ARN> \ --input "{\"jobs\": [{\"input\": \"First job\", \"done\": false}, {\"input\": \"Second job\", \"done\": false}]}"
19. Trigger it from S3 Set up S3 notifications in SAM template. Assets: Type: AWS::S3::Bucket DeletionPolicy: Retain Properties: BucketName: "playground-assets" NotificationConfiguration: LambdaConfigurations: - Function: !GetAtt ProcessExecute.Arn Event: "s3:ObjectCreated:*" Filter: S3Key: Rules: - Name: suffix Value: completed
20. Trigger it from S3 Update Lambda handler to accept S3 event. func handle(e events.S3Event) error { ... }
21. Thank you ! https://medium.com/@christianklotz/c185c2d2608 https://medium.com/@sgarcez/6cd6501d261e https://github.com/christianklotz/aws-step-functions-iterate- sample https://github.com/awslabs/video-on-demand-on-aws https://github.com/aws-samples/startup-kit-templates ! @christianklotz