Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Programming Dynamic AWS CloudFormation Templates Luis Colon Senior Developer Advocate AWS CloudFormation D E V 4 1 8 Anuradha Garg Sw Development Engineer AWS CloudFormation Sayali Deshpande Sw Development Manager AWS CloudFormation Chelsey Salberg Web Development Engineer AWS CloudFormation

Slide 3

Slide 3 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Welcome! Objective: Code transformation options to make dynamic CloudFormation templates Setting up an ideal template development environment Injecting code snippets and adding utility functions Create a validation rule for your custom code

Slide 4

Slide 4 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Before We Start… A laptop, internet, and AWS account are required Check connectivity Pair up with someone if necessary Focus on lab activities Plan to use VS Code as a local editor Retrieve all lab instructions here: https://bit.ly/2DNayik

Slide 5

Slide 5 text

AWS CloudFormation Enables provisioning and management of your infrastructure as code (IaC) Supports YAML & JSON formats Understands dependencies Supports rollbacks Supports cross-account, cross-region deployments

Slide 6

Slide 6 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. CloudFormation 101 Code in YAML or JSON directly or use sample templates Upload local files or from an S3 bucket Create stack using console, API or CLI Stacks and resources are provisioned

Slide 7

Slide 7 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Building CloudFormation Stacks A CFN template is a file that lists the resources to be created in a stack in a declarative way: tell CFN what you need created, rather than how to create it When you change your template, you are declaring the need to update your stack’s resources Virtually every kind of resource can be managed by stacks You can execute an update directly, or via an update plan called a changeset

Slide 8

Slide 8 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Authoring Options You can use traditional IDEs or lightweight editors Some have CFN plugins: VS Code, PyCharm, others The best plugins leverage our resource specification Use utilities, like cfn-flip (converts JSON to YAML), cfn- nag, many others available on GitHub Code in higher level languages, like CDK (TypeScript), Troposphere (Python), SparkleFormation (Ruby), other options

Slide 9

Slide 9 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. CFN Linter (cfn-python-lint) Integrated with IDE’s via plugins to provide the quickest feedback on code errors, warnings Robust validation powered by our resource specification 100% Open Source - extend by building your own validation rules Use in headless mode for automated testing in pipelines

Slide 10

Slide 10 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Our Workshop Plan LAB 2: Build and deploy a complex macro LAB 3: Create a specification to validate your macro LAB 1: Set up editor, linter and deploy a simple macro

Slide 11

Slide 11 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lab 1 Allotted time: around 30 minutes Objectives: Set up editor and plugins Set up CFN linter Deploy a macro Deploy a stack using the macro

Slide 12

Slide 12 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Post Lab 1 What did we learn? Where to next: Make a new, more complex macro Deploy the macro Use the macro in a different template Deploy the stack

Slide 13

Slide 13 text

Part 2 CloudFormation Macros

Slide 14

Slide 14 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. CloudFormation Macros Enables template coders to write short-hand, abbreviated instructions that expand automatically when deployed Add utility functions, for example, iteration loops, strings, etc. Ensure resources are defined to comply to your standards Easy to share and reuse across stacks 3 step process

Slide 15

Slide 15 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Step 1: Create your function import copy def process_template(template): new_template = copy.deepcopy(template) status = 'success' for name, resource in template['Resources'].items(): if 'Count' in resource: count = new_template['Resources'][name].pop('Count') multiplied = multiply(name, new_template['Resources'][name], count) if not set(multiplied.keys()) & set(new_template['Resources'].keys()): new_template['Resources'].update(multiplied) else: status = 'failed' return status, template return status, new_template def multiply(resource_name, resource_structure, count): resources = {} for iteration in range(1, count): resources[resource_name+str(iteration)] = resource_structure return resources def handler(event, context): result = process_template(event['fragment']) return { 'requestId': event['requestId'], 'status': result[0], 'fragment': result[1], }

Slide 16

Slide 16 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Step 2: Register (and deploy) your Macro AWSTemplateFormatVersion: '2010-09-09' Transform: AWS::Serverless-2016-10-31 Resources: Macro: Type: AWS::CloudFormation::Macro Properties: Name: Count FunctionName: !GetAtt CountMacroFunction.Arn CountMacroFunction: Type: AWS::Serverless::Function Properties: CodeUri: src Handler: index.handler Runtime: python3.6 Timeout: 5

Slide 17

Slide 17 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Step 3: Using the Macro Transform: - Count Resources: Bucket: Type: AWS::S3::Bucket Count: 3 Transform: - Count Sqs: Type: AWS:::SQS::Queue Count: 2 Resources: Bucket1: Type: AWS::S3::Bucket Bucket2: Type: AWS::S3::Bucket Bucket3: Type: AWS::S3::Bucket Resources: Sqs1: Type: AWS:::SQS::Queue Sqs2: Type: AWS:::SQS::Queue

Slide 18

Slide 18 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Macro Example 2: Add String Functions Parameters: InputString: Default: "This is a test input string" Type: String Resources: S3Bucket: Type: "AWS::S3::Bucket" Properties: Tags: - Key: Upper Value: 'Fn::Transform': - Name: 'StringMacro' Parameters: InputString: !Ref InputString Operation: Upper Parameters: InputString: Default: "This is a test input string" Type: String Resources: S3Bucket: Type: "AWS::S3::Bucket" Properties: Tags: - Key: Upper Value: “THIS IS A TEST INPUT STRING”

Slide 19

Slide 19 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Code: Add String Functions def handler(event, context): response = { "requestId": event["requestId"], "status": "success" } try: operation = event["params"]["Operation"] input = event["params"]["InputString"] no_param_string_funcs = ["Upper", "Lower", "Capitalize", "Title", "SwapCase"] if operation in no_param_string_funcs: response["fragment"] = getattr(input, operation.lower())() elif operation == "Strip": chars = None if "Chars" in event["params"]: chars = event["params"]["Chars"] response["fragment"] = input.strip(chars) elif operation == "Replace": old = event["params"]["Old"] new = event["params"]["New"] response["fragment"] = input.replace(old, new) elif operation == "MaxLength": length = int(event["params"]["Length"]) if len(input) <= length: response["fragment"] = input elif "StripFrom" in event["params"]: if event["params"]["StripFrom"] == "Left": response["fragment"] = input[len(input)-length:] elif event["params"]["StripFrom"] != "Right": response["status"] = "failure" else: response["fragment"] = input[:length] else: response["status"] = "failure" except Exception: traceback.print_exc() response["status"] = "failure" macro_response["errorMessage"] = str(e) return response Multiple Functions in a single macro: Upper Lower Capitalize Title SwapCase Strip Replace MaxLength

Slide 20

Slide 20 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Macro Example 3: Generate Additional Resources Transform: Defaults Resources: Bucket1: Type: AWS::S3::Bucket Whenever a bucket is defined… Add access control property Add bucket policy Generate additional resources, intrinsic function calls, conditions, more Macro can allow user to override defaults Resources: Bucket1: Type: AWS::S3::Bucket Properties: AccessControl: Private Bucket1Policy: Type: AWS::S3::BucketPolicy Properties: Bucket: Ref: Bucket1 PolicyDocument: Version: "2012-10-17" Statement: - Effect: Deny Principal: "*" Action: "s3:Delete*" Resource: Fn::Sub: "arn:aws:s3:::${Bucket1}/*" Condition: Bool: aws:MultiFactorAuthPresent: "false"

Slide 21

Slide 21 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Macro Example 4: Add Global Variables Transform: Globals Globals: SomeText: some-text ThingTag: Key: Thing Value: This is a thing Resources: Bucket: Type: AWS::S3::Bucket Properties: BucketName: "@SomeText" Tags: - "@ThingTag" - Key: OtherThing Value: Other thing value Resources: Bucket: Type: AWS::S3::Bucket Properties: BucketName: “some-text" Tags: - Key: Thing Value: This is a thing - Key: OtherThing Value: Other thing value

Slide 22

Slide 22 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Globals Macro: Codeclass Repeater(): def __init__(self, template): self.repeaters = template["Globals"] del template["Globals"] self.template = template def process(self): return self.__walk(self.template) def __walk(self, fragment): if isinstance(fragment, str) and any(fragment == "@{}".format(key) for key in self.repeaters): return self.repeaters[fragment[1:]] elif isinstance(fragment, dict): return { key: self.__walk(value) for key, value in fragment.items() } elif isinstance(fragment, list): return [ self.__walk(value) for value in fragment ] return fragment def handler(event, context): return { "requestId": event["requestId"], "status": "success", "fragment": Repeater(event["fragment"]).process(), } Transform: Globals Globals: SomeText: some-text ThingTag: Key: Thing Value: This is a thing Resources: Bucket: Type: AWS::S3::Bucket Properties: BucketName: "@SomeText" Tags: - "@ThingTag" - Key: OtherThing Value: Other thing value

Slide 23

Slide 23 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lab 2 Allotted time: around 30 minutes Objectives: Make a new macro Deploy the macro Use the macro in a different template Deploy the stack that uses the macro

Slide 24

Slide 24 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Post Lab 2 What did we learn? Where to next: Create a specification to validate your macro

Slide 25

Slide 25 text

Part 3 Using the Linter to Validate Macros

Slide 26

Slide 26 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. CFN Linter To install: (requires Python) Download: https://github.com/awslabs/cfn-python- lint Run: pip install cfn-lint To try: cfn-lint –t simple-vpc.yaml • Process multiple files • Better handling of Conditions/Fn::If • SAM Local integration for SAM templates • CloudFormation limit checks • Service rules for Route53 and CodePipeline • Used in other tools, like TaskCat GitHub 306 Stars 290 PRs merged 153 Issues closed 61 Forks Current release: v.0.9.1 Pypi 6,905 installs this week 23,232 installs this month 55,871 installs since release

Slide 27

Slide 27 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Linting your own macro The linter knows about all resources known to CloudFormation, via the resource specification Strategy: to lint your own macros, you override the CloudFormation resource specification

Slide 28

Slide 28 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What we’re doing Consider the following template: The Type AWS::S3::Object is not in the CloudFormation resource spec! --- AWSTemplateFormatVersion: '2010-09-09' Description: "Create an S3 Object" Parameters: TargetBucket: Type: "String" Default: "foo" Resources: NewObject: Type: AWS::S3::Object Properties: Target: Bucket: !Ref TargetBucket Key: README.md Body: | # My text file This is my text file; there are many like it, but this one is mine.

Slide 29

Slide 29 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What we’re doing When we try to lint it, we get the following error:

Slide 30

Slide 30 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Solution! (1 of 2) We override the default specification by creating a new one Part 1 - Property Types { "PropertyTypes": { "AWS::S3::Object.Target": { "Documentation": "https://github.com/awslabs/aws-cloudformation- templates/tree/master/aws/services/CloudFormation/MacrosExamples/S3Objects", "Properties": { "Bucket": { "Documentation": "https://github.com/awslabs/aws-cloudformation- templates/tree/master/aws/services/CloudFormation/MacrosExamples/S3Objects", "PrimitiveType": "String", "Required": true, "UpdateType": "Immutable" }, "Key": { "Documentation": "https://github.com/awslabs/aws-cloudformation- templates/tree/master/aws/services/CloudFormation/MacrosExamples/S3Objects", "PrimitiveType": "String", "Required": true, "UpdateType": "Immutable" } } } },

Slide 31

Slide 31 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Solution! (2 of 2) We override the default specification by creating a new one Part 2 - Resource Types "ResourceTypes": { "AWS::S3::Object": { "Documentation": "https://github.com/awslabs/aws-cloudformation- templates/tree/master/aws/services/CloudFormation/MacrosExamples/S3Objects", "Properties": { "Target": { "Documentation": "https://github.com/awslabs/aws-cloudformation- templates/tree/master/aws/services/CloudFormation/MacrosExamples/S3Objects", "Type": "Target", "Required": true, "UpdateType": "Immutable" }, "Body": { "Documentation": "https://github.com/awslabs/aws-cloudformation- templates/tree/master/aws/services/CloudFormation/MacrosExamples/S3Object", "Required": true, "PrimitiveType": "String", "UpdateType": "Immutable" } } } } }

Slide 32

Slide 32 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Lab 3 Allotted time: around 30 minutes Objectives: Running a simple validation via command line Validating against a macro Overriding CloudFormation specifications Re-validating the macro with the new specifications Bonus! Validate the macro you worked with on Lab 2

Slide 33

Slide 33 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Post Lab 3 What did we learn? Use linter to validate a specific template Creating custom specifications to validate your own macros

Slide 34

Slide 34 text

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Wrap Up Leverage macros for many use cases Create a specification to validate your macro Set up optimal tools to improve your productivity

Slide 35

Slide 35 text

Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Luis Colon Senior Developer Advocate AWS CloudFormation Anuradha Garg Sw Development Engineer AWS CloudFormation Sayali Deshpande Sw Development Manager AWS CloudFormation Chelsey Salberg Web Development Engineer AWS CloudFormation

Slide 36

Slide 36 text

Please complete the session survey in the mobile app. ! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.