Pipeline-driven CloudFormation

Use tools like cfn-lint and taskcat to create a pipeline to deploy tested CloudFormation templates.

Transcript

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Pipeline-driven CloudFormation: Authoring and Testing DevOps Meetup Chicago June 18, 2019 Dan Blanco (@thedanblanco) Dev Advocate, AWS CloudFormation dblanc@amazon.com Luis Colon (@luiscolon1) Sr Dev Advocate, AWS CloudFormation licolon@amazon.com
2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Agenda • Writing • Versioning • Editor • Configuring for CloudFormation • Testing • Linting • Taskcat • Pipeline • Maintaining • Reusability • Smaller Files • Bringing it all together
3. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
4. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Code in YAML/JSON directly, use SAM or macros, translate higher level languages (CDK), or use sample templates Upload local files via the browser console, from an S3 bucket, or via pipelines Create stacks or using console, CLI, or AWS SDK, or stack set instances across accounts and regions Stacks, stack sets, and resources are created and managed
5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Source Version control all templates and configuration Build Static analysis and tests Test Clean environment for integration testing Promote Deployment to live environments Infrastructure… is code!
6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Setting up a Pipeline Region Cloud Developers Git push Templates AWS CodeCommit AWS CodePipeline AWS CodeBuild taskcat AWS CloudFormation cfn-lint cfn-nag test stack change set staging production
7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Make the most out of your editor https://hodgkins.io/up-your-cloudformation-game-with-vscode
8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Configuring your editor https://github.com/aws-cloudformation/aws-cloudformation-template-schema
9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Linting https://github.com/aws-cloudformation/cfn-python-lint • Plugins for Atom, VisualStudio Code, Sublime, VIM • Run headless in pipelines • Process multiple files • Handles Conditions/Fn::If • SAM Local integration • Available now on GitHub, over 100,000 downloads
10. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Customizing your linter • Require specific tags • Black list of resource types • ie. Can’t create X resource type • Enforce/require a property • Forbid a property value • ie. Don’t let people create public buckets • More! https://binx.io/blog/2018/07/07/aws-cloudformation-validation-in-cicd-pipelines/
11. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Taskcat https://github.com/aws-quickstart/taskcat • From the AWS QuickStart team • Open Source • Catches problems that aren’t obvious in a single template or stack • Tests templates by creating stacks in multiple AWS regions simultaneously • Generates a report with a pass/fail grade for each region
12. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Proritize end-to-end tests As a general rule, we want to be writing less code in the serverless world. It's quite possible to write a useful service in AWS using mostly configuration. For example, you can build a CRUD API using API Gateway and DynamoDB alone - no Lambda functions in the middle required. The more "serverless" you get, the less code you can usefully unit test, and the more you have to rely on tests of your deployed infrastructure. https://dev.to/trek10inc/ci-cd-aws-and-serverless-5-tips-i-learned-the-hard-way-223p
13. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Scenario • Inheriting a template • Non-working code! • Set up pipeline and authoring environment • Validating pipeline • Resolving issues
14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Region Cloud Developers Git push Templates AWS CodeCommit AWS CodePipeline AWS CodeBuild taskcat AWS CloudFormation cfn-lint cfn-nag test stack change set staging production
15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Refactoring for easier maintenance Parameters Mappings Conditions Outputs Resources YAML/JSON Template • Parameters: avoids hardcoding of values, can add validation to users and improve UX with console grouping, labels and descriptions. Keep secrets in Parameter Store and Secrets Manager • Mappings: as a case statement, helps maintain a set of information for different environments • Conditions: simple if/then statements - “if dev do this, if prod do that” • Imports and Exports
16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Smaller templates, modular stacks Instances, AutoScaling groups API endpoints, functions Alarms, dashboards VPCs, NAT gateways, VPNs, subnets IAM users, groups, roles, policies Frontend Resources Backend Services Stateful Resources Base Network Identity & Security Dev Test Stage Prod 1 2 Break stacks by layers and lifecycle Reuse stacks by environments Monitoring Resources Databases and clusters, queues
17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Summary • Put all your assets in version control from the start • Leverage modern editors and plugins • Optimize your editor for CloudFormation • Use a pipeline • Use cfn-lint, both for editing and in the pipeline • Test on different environments with Taskcat • Refactor your files for reuse - parameters, mappings, etc • Smaller templates are easier to test and maintain, quicker to deploy
18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank You :) Get involved in slack - DM us on Twitter to join the #cloudformation channel Dan Blanco (@thedanblanco) Dev Advocate, AWS CloudFormation dblanc@amazon.com Luis Colon (@luiscolon1) Sr Dev Advocate, AWS CloudFormation licolon@amazon.com