Programming Dynamic CloudFormation Templates

From a workshop given at re:Invent 2018.

Transcript

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Programming Dynamic AWS CloudFormation Templates Luis Colon Senior Developer Advocate AWS CloudFormation D E V 4 1 8 Anuradha Garg Sw Development Engineer AWS CloudFormation Sayali Deshpande Sw Development Manager AWS CloudFormation Chelsey Salberg Web Development Engineer AWS CloudFormation
3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Welcome! Objective: Code transformation options to make dynamic CloudFormation templates Setting up an ideal template development environment Injecting code snippets and adding utility functions Create a validation rule for your custom code
4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Before We Start… A laptop, internet, and AWS account are required Check connectivity Pair up with someone if necessary Focus on lab activities Plan to use VS Code as a local editor Retrieve all lab instructions here: https://bit.ly/2DNayik
5. AWS CloudFormation Enables provisioning and management of your infrastructure as code (IaC) Supports YAML & JSON formats Understands dependencies Supports rollbacks Supports cross-account, cross-region deployments
6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. CloudFormation 101 Code in YAML or JSON directly or use sample templates Upload local files or from an S3 bucket Create stack using console, API or CLI Stacks and resources are provisioned
7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Building CloudFormation Stacks A CFN template is a file that lists the resources to be created in a stack in a declarative way: tell CFN what you need created, rather than how to create it When you change your template, you are declaring the need to update your stack’s resources Virtually every kind of resource can be managed by stacks You can execute an update directly, or via an update plan called a changeset
8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Authoring Options You can use traditional IDEs or lightweight editors Some have CFN plugins: VS Code, PyCharm, others The best plugins leverage our resource specification Use utilities, like cfn-flip (converts JSON to YAML), cfn- nag, many others available on GitHub Code in higher level languages, like CDK (TypeScript), Troposphere (Python), SparkleFormation (Ruby), other options
9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. CFN Linter (cfn-python-lint) Integrated with IDE’s via plugins to provide the quickest feedback on code errors, warnings Robust validation powered by our resource specification 100% Open Source - extend by building your own validation rules Use in headless mode for automated testing in pipelines
10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Our Workshop Plan LAB 2: Build and deploy a complex macro LAB 3: Create a specification to validate your macro LAB 1: Set up editor, linter and deploy a simple macro
11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lab 1 Allotted time: around 30 minutes Objectives: Set up editor and plugins Set up CFN linter Deploy a macro Deploy a stack using the macro
12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Post Lab 1 What did we learn? Where to next: Make a new, more complex macro Deploy the macro Use the macro in a different template Deploy the stack
13. Part 2 CloudFormation Macros
14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. CloudFormation Macros Enables template coders to write short-hand, abbreviated instructions that expand automatically when deployed Add utility functions, for example, iteration loops, strings, etc. Ensure resources are defined to comply to your standards Easy to share and reuse across stacks 3 step process
15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Step 1: Create your function import copy def process_template(template): new_template = copy.deepcopy(template) status = 'success' for name, resource in template['Resources'].items(): if 'Count' in resource: count = new_template['Resources'][name].pop('Count') multiplied = multiply(name, new_template['Resources'][name], count) if not set(multiplied.keys()) & set(new_template['Resources'].keys()): new_template['Resources'].update(multiplied) else: status = 'failed' return status, template return status, new_template def multiply(resource_name, resource_structure, count): resources = {} for iteration in range(1, count): resources[resource_name+str(iteration)] = resource_structure return resources def handler(event, context): result = process_template(event['fragment']) return { 'requestId': event['requestId'], 'status': result[0], 'fragment': result[1], }
16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Step 2: Register (and deploy) your Macro AWSTemplateFormatVersion: '2010-09-09' Transform: AWS::Serverless-2016-10-31 Resources: Macro: Type: AWS::CloudFormation::Macro Properties: Name: Count FunctionName: !GetAtt CountMacroFunction.Arn CountMacroFunction: Type: AWS::Serverless::Function Properties: CodeUri: src Handler: index.handler Runtime: python3.6 Timeout: 5
17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Step 3: Using the Macro Transform: - Count Resources: Bucket: Type: AWS::S3::Bucket Count: 3 Transform: - Count Sqs: Type: AWS:::SQS::Queue Count: 2 Resources: Bucket1: Type: AWS::S3::Bucket Bucket2: Type: AWS::S3::Bucket Bucket3: Type: AWS::S3::Bucket Resources: Sqs1: Type: AWS:::SQS::Queue Sqs2: Type: AWS:::SQS::Queue
18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Macro Example 2: Add String Functions Parameters: InputString: Default: "This is a test input string" Type: String Resources: S3Bucket: Type: "AWS::S3::Bucket" Properties: Tags: - Key: Upper Value: 'Fn::Transform': - Name: 'StringMacro' Parameters: InputString: !Ref InputString Operation: Upper Parameters: InputString: Default: "This is a test input string" Type: String Resources: S3Bucket: Type: "AWS::S3::Bucket" Properties: Tags: - Key: Upper Value: “THIS IS A TEST INPUT STRING”
19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Code: Add String Functions def handler(event, context): response = { "requestId": event["requestId"], "status": "success" } try: operation = event["params"]["Operation"] input = event["params"]["InputString"] no_param_string_funcs = ["Upper", "Lower", "Capitalize", "Title", "SwapCase"] if operation in no_param_string_funcs: response["fragment"] = getattr(input, operation.lower())() elif operation == "Strip": chars = None if "Chars" in event["params"]: chars = event["params"]["Chars"] response["fragment"] = input.strip(chars) elif operation == "Replace": old = event["params"]["Old"] new = event["params"]["New"] response["fragment"] = input.replace(old, new) elif operation == "MaxLength": length = int(event["params"]["Length"]) if len(input) <= length: response["fragment"] = input elif "StripFrom" in event["params"]: if event["params"]["StripFrom"] == "Left": response["fragment"] = input[len(input)-length:] elif event["params"]["StripFrom"] != "Right": response["status"] = "failure" else: response["fragment"] = input[:length] else: response["status"] = "failure" except Exception: traceback.print_exc() response["status"] = "failure" macro_response["errorMessage"] = str(e) return response Multiple Functions in a single macro: Upper Lower Capitalize Title SwapCase Strip Replace MaxLength
20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Macro Example 3: Generate Additional Resources Transform: Defaults Resources: Bucket1: Type: AWS::S3::Bucket Whenever a bucket is defined… Add access control property Add bucket policy Generate additional resources, intrinsic function calls, conditions, more Macro can allow user to override defaults Resources: Bucket1: Type: AWS::S3::Bucket Properties: AccessControl: Private Bucket1Policy: Type: AWS::S3::BucketPolicy Properties: Bucket: Ref: Bucket1 PolicyDocument: Version: "2012-10-17" Statement: - Effect: Deny Principal: "*" Action: "s3:Delete*" Resource: Fn::Sub: "arn:aws:s3:::${Bucket1}/*" Condition: Bool: aws:MultiFactorAuthPresent: "false"
21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Macro Example 4: Add Global Variables Transform: Globals Globals: SomeText: some-text ThingTag: Key: Thing Value: This is a thing Resources: Bucket: Type: AWS::S3::Bucket Properties: BucketName: "@SomeText" Tags: - "@ThingTag" - Key: OtherThing Value: Other thing value Resources: Bucket: Type: AWS::S3::Bucket Properties: BucketName: “some-text" Tags: - Key: Thing Value: This is a thing - Key: OtherThing Value: Other thing value
22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Globals Macro: Codeclass Repeater(): def __init__(self, template): self.repeaters = template["Globals"] del template["Globals"] self.template = template def process(self): return self.__walk(self.template) def __walk(self, fragment): if isinstance(fragment, str) and any(fragment == "@{}".format(key) for key in self.repeaters): return self.repeaters[fragment[1:]] elif isinstance(fragment, dict): return { key: self.__walk(value) for key, value in fragment.items() } elif isinstance(fragment, list): return [ self.__walk(value) for value in fragment ] return fragment def handler(event, context): return { "requestId": event["requestId"], "status": "success", "fragment": Repeater(event["fragment"]).process(), } Transform: Globals Globals: SomeText: some-text ThingTag: Key: Thing Value: This is a thing Resources: Bucket: Type: AWS::S3::Bucket Properties: BucketName: "@SomeText" Tags: - "@ThingTag" - Key: OtherThing Value: Other thing value
23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lab 2 Allotted time: around 30 minutes Objectives: Make a new macro Deploy the macro Use the macro in a different template Deploy the stack that uses the macro
24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Post Lab 2 What did we learn? Where to next: Create a specification to validate your macro
25. Part 3 Using the Linter to Validate Macros
26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. CFN Linter To install: (requires Python) Download: https://github.com/awslabs/cfn-python- lint Run: pip install cfn-lint To try: cfn-lint –t simple-vpc.yaml • Process multiple files • Better handling of Conditions/Fn::If • SAM Local integration for SAM templates • CloudFormation limit checks • Service rules for Route53 and CodePipeline • Used in other tools, like TaskCat GitHub 306 Stars 290 PRs merged 153 Issues closed 61 Forks Current release: v.0.9.1 Pypi 6,905 installs this week 23,232 installs this month 55,871 installs since release
27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Linting your own macro The linter knows about all resources known to CloudFormation, via the resource specification Strategy: to lint your own macros, you override the CloudFormation resource specification
28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What we’re doing Consider the following template: The Type AWS::S3::Object is not in the CloudFormation resource spec! --- AWSTemplateFormatVersion: '2010-09-09' Description: "Create an S3 Object" Parameters: TargetBucket: Type: "String" Default: "foo" Resources: NewObject: Type: AWS::S3::Object Properties: Target: Bucket: !Ref TargetBucket Key: README.md Body: | # My text file This is my text file; there are many like it, but this one is mine.
29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What we’re doing When we try to lint it, we get the following error:
30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Solution! (1 of 2) We override the default specification by creating a new one Part 1 - Property Types { "PropertyTypes": { "AWS::S3::Object.Target": { "Documentation": "https://github.com/awslabs/aws-cloudformation- templates/tree/master/aws/services/CloudFormation/MacrosExamples/S3Objects", "Properties": { "Bucket": { "Documentation": "https://github.com/awslabs/aws-cloudformation- templates/tree/master/aws/services/CloudFormation/MacrosExamples/S3Objects", "PrimitiveType": "String", "Required": true, "UpdateType": "Immutable" }, "Key": { "Documentation": "https://github.com/awslabs/aws-cloudformation- templates/tree/master/aws/services/CloudFormation/MacrosExamples/S3Objects", "PrimitiveType": "String", "Required": true, "UpdateType": "Immutable" } } } },
31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Solution! (2 of 2) We override the default specification by creating a new one Part 2 - Resource Types "ResourceTypes": { "AWS::S3::Object": { "Documentation": "https://github.com/awslabs/aws-cloudformation- templates/tree/master/aws/services/CloudFormation/MacrosExamples/S3Objects", "Properties": { "Target": { "Documentation": "https://github.com/awslabs/aws-cloudformation- templates/tree/master/aws/services/CloudFormation/MacrosExamples/S3Objects", "Type": "Target", "Required": true, "UpdateType": "Immutable" }, "Body": { "Documentation": "https://github.com/awslabs/aws-cloudformation- templates/tree/master/aws/services/CloudFormation/MacrosExamples/S3Object", "Required": true, "PrimitiveType": "String", "UpdateType": "Immutable" } } } } }
32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lab 3 Allotted time: around 30 minutes Objectives: Running a simple validation via command line Validating against a macro Overriding CloudFormation specifications Re-validating the macro with the new specifications Bonus! Validate the macro you worked with on Lab 2
33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Post Lab 3 What did we learn? Use linter to validate a specific template Creating custom specifications to validate your own macros
34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Wrap Up Leverage macros for many use cases Create a specification to validate your macro Set up optimal tools to improve your productivity
35. Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Luis Colon Senior Developer Advocate AWS CloudFormation Anuradha Garg Sw Development Engineer AWS CloudFormation Sayali Deshpande Sw Development Manager AWS CloudFormation Chelsey Salberg Web Development Engineer AWS CloudFormation
36. Please complete the session survey in the mobile app. ! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.