Slide 1

Slide 1 text

CloudFormation vs Ansible

Slide 2

Slide 2 text

Smackdown!

Slide 3

Slide 3 text

What is CloudFormation? “...an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion.” https://aws.amazon.com/cloudformation/

Slide 4

Slide 4 text

Ansible — Many platforms — Provisioning and configuring

Slide 5

Slide 5 text

CloudFormation — AWS platform only — Provisioning

Slide 6

Slide 6 text

Why CloudFormation then?

Slide 7

Slide 7 text

|--------------|---------------|----------------| | Feature | Ansible | CloudFormation | |==============|===============|================| | Written in | YAML | JSON | |--------------|---------------|----------------| | Grouped as | Playbook | Stack | |--------------|---------------|----------------| | State | Stateless | Keeps state | |--------------|---------------|----------------| | Deletion | Manual | Automatic | |--------------|---------------|----------------| | Dependencies | Manual | Automatic | |--------------|---------------|----------------| | Ordering | Top to bottom | Simultaneous | |--------------|---------------|----------------|

Slide 8

Slide 8 text

|--------------------|------------------|-----------------| | Feature | Ansible | CloudFormation | |====================|==================|=================| | UI | CLI/Tower | CLI/AWS Console | |--------------------|------------------|-----------------| | Reusability | Roles / includes | Nested Stacks | |--------------------|------------------|-----------------| | Rollbacks | No | Yes | |--------------------|------------------|-----------------| | Existing resources | Can incorporate | No | |--------------------|------------------|-----------------| | AWS Services | Limited | Most | |--------------------|------------------|-----------------| | Ease | Easier | Harder | |--------------------|------------------|-----------------| | Triggers/Policies | No | Yes | |--------------------|------------------|-----------------|

Slide 9

Slide 9 text

Writing CloudFormation templates

Slide 10

Slide 10 text

Don't!

Slide 11

Slide 11 text

Use your favourite language Generate!

Slide 12

Slide 12 text

https://github.com/ bazaarvoice/ cloudformation-ruby-dsl

Slide 13

Slide 13 text

Other generators — https://github.com/cloudtools/troposphere (Python) — https://github.com/tongueroo/lono (Ruby) — https://github.com/MonsantoCo/cloudformation- template-generator (Scala)

Slide 14

Slide 14 text

Organising

Slide 15

Slide 15 text

God stack — Single stack for all resources — Good for small deployments — Size limits — Reuse only by using a generator tool

Slide 16

Slide 16 text

Nested stacks ┌─────────────────┐ │ │ │ │ │ Solution │ │ │ │ │ └─────────────────┘ │ ┌───────────────────┬─────────┴─────────┬───────────────────┐ ▼ ▼ ▼ ▼ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ VPC │ │ Database │ │ Servers │ │ Security Groups │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └─────────────────┘ └─────────────────┘ └─────────────────┘ └─────────────────┘

Slide 17

Slide 17 text

Nested stacks — CloudFormation invokes child templates — Atomic execution — One fails, all fail — Pass parameters through "references" — Need to pass parameters for child template "through" the parent

Slide 18

Slide 18 text

Pipelined stacks Executor ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ 1 │ │ 2 │ │ 3 │ │ 4 │ │ │ │ │ │ │ │ │ │ VPC │ │ Database │ │ Servers │ │ Security Groups │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └─────────────────┘ └─────────────────┘ └─────────────────┘ └─────────────────┘

Slide 19

Slide 19 text

Pipelined stacks — Execute each template manually, requires orchestration — Non-atomic execution — One fails, only that template fails — Pass parameters directly — Chain outputs to inputs

Slide 20

Slide 20 text

Partitioning

Slide 21

Slide 21 text

Horizontal ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ │ │ Database │ │ Servers │ │ Security Groups │ │ │ ├─────────────────┤ ├─────────────────┤ ├─────────────────┤ │ │ │ Application 1 │ │ Application 1 │ │ Application 1 │ │ VPC │ │ │ ├─────────────────┤ ├─────────────────┤ │ │ ├─────────────────┤ │ Application 2 │ │ Application 2 │ │ │ │ Application 2 │ ├─────────────────┤ ├─────────────────┤ │ │ │ │ │ Application 3 │ │ Application 3 │ └─────────────────┘ └─────────────────┘ └─────────────────┘ └─────────────────┘

Slide 22

Slide 22 text

Vertical ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ │ │ Application 1 │ │ Application 2 │ │ Application 3 │ │ │ ├─────────────────┤ ├─────────────────┤ ├─────────────────┤ │ │ │ Servers │ │ Servers │ │ │ │ VPC │ ├─────────────────┤ ├─────────────────┤ │ Servers │ │ │ │ Database │ │ Database │ │ │ │ │ ├─────────────────┤ ├─────────────────┤ ├─────────────────┤ │ │ │ Security Groups │ │ Security Groups │ │ Security Groups │ └─────────────────┘ └─────────────────┘ └─────────────────┘ └─────────────────┘

Slide 23

Slide 23 text

Partitioning — Keep templates simple — Reuse parameters and mappings via a DSL — Choose vertical or horizontal scaling — Decide if you want to rollback the stack or not

Slide 24

Slide 24 text

Orchestration

Slide 25

Slide 25 text

Shell — Simple batch script — Use aws-cli to invoke CloudFormation templates: aws cloudformation create-stack... — Capture output variables for input variables for next stack — Returns immediately, therefore must check status: aws cloudformation create-describe- stacks...

Slide 26

Slide 26 text

Ansible - name: launch ansible cloudformation example cloudformation: stack_name: ansible-cloudformation state: present region: us-east-1 disable_rollback: true template: files/cloudformation-example.json template_parameters: KeyName: jmartin DiskType: ephemeral InstanceType: m1.small ClusterSize: 3 tags: Stack: ansible-cloudformation

Slide 27

Slide 27 text

Ansible - include: key_pairs.yml tags: always - include: compile.yml tags: always - include: vpc.yml tags: always - include: nat.yml tags: nat - include: titan.yml tags: titan

Slide 28

Slide 28 text

Ansible - name: delete old cloudformation templates file: path: "{{ item }}" state: absent with_fileglob: ../cloudformation/compiled/*.json when: clean is defined and clean|bool - name: compile the cloudformation templates shell: "{{ item }} expand --nopretty" with_fileglob: ../cloudformation/dsl/*.rb register: compiled - name: write and validate templates template: src: templates/cloudformation.json dest: cloudformation/compiled/{{ item.item | basename | regex_replace('\.rb$', '.json') }} validate: aws cloudformation validate-template --region {{ region }} --template-body file://%s environment: AWS_ACCESS_KEY_ID: "{{ aws_credentials[env].access_key }}" AWS_SECRET_ACCESS_KEY: "{{ aws_credentials[env].secret_key }}" with_items: compiled.results

Slide 29

Slide 29 text

Ansible - name: execute vpc cloudformation template cloudformation: aws_access_key: "{{ aws_credentials[env].access_key }}" aws_secret_key: "{{ aws_credentials[env].secret_key }}" stack_name: vpc-{{ env }} region: "{{ region }}" template: ./cloudformation/compiled/vpc.json disable_rollback: true template_parameters: Environment: "{{ env }}" tags: Environment: "{{ env }}" Region: "{{ region }}" register: vpc

Slide 30

Slide 30 text

Ansible - name: execute titan cloudformation template cloudformation: aws_access_key: "{{ aws_credentials[env].access_key }}" aws_secret_key: "{{ aws_credentials[env].secret_key }}" stack_name: titan-{{ env }} region: "{{ region }}" template: ./cloudformation/compiled/titan.json disable_rollback: true template_parameters: Environment: "{{ env }}" CochlearLinkHostedZone: "{{ vpc.stack_outputs.HostedZone }}" CochlearLinkVpc: "{{ vpc.stack_outputs.Vpc }}" TitanSubnets: "{{ vpc.stack_outputs.TitanSubnets }}" tags: Environment: "{{ env }}" Region: "{{ region }}" register: titan

Slide 31

Slide 31 text

Configuration

Slide 32

Slide 32 text

Configuration — Keep configuration in Git, S3 or DynamoDB — raw files from GitHub work well

Slide 33

Slide 33 text

{ "ap-southeast-2": { "dev": { "productVersion": "3.0.0", "installationId": "eeeeeeee-f057-40c7-b365-792cc7faeccb" }, "sit": { "productVersion": "3.1.8", "installationId": "17171717-f057-40c7-b365-792cc7faeccb" } }, "us-east-1": { "prd": { "productVersion": "2.1.0", "installationId": "e047ac6f-f057-40c7-b365-792cc7faeccb" } } }

Slide 34

Slide 34 text

{ "components": { "3.1.8": { "app": "3.1.8", "db": "1.0.1", } } }

Slide 35

Slide 35 text

{ "versions": { "3.1.8": { "ap-southeast-2": { "image": "ami-9f266aa5" }, "eu-west-1": { "image": "ami-0914307e" } }, "3.1.2": { "ap-southeast-2": { "image": "ami-170e422d" }, "eu-west-1": { "image": "ami-5dbf9a2a" } } } }

Slide 36

Slide 36 text

- name: get environments config uri: url: https://your-git-repo.com/application/config/raw/master/environments.json return_content: yes register: environments_meta_raw

Slide 37

Slide 37 text

Getting out of trouble

Slide 38

Slide 38 text

The dreaded _FAILED states CREATE_FAILED DELETE_FAILED ROLLBACK_FAILED UPDATE_ROLLBACK_FAILED

Slide 39

Slide 39 text

The dreaded _FAILED states — Resolve manually — Console will give you indication of which resource failed to update — Delete offending resource or its dependencies manually — Run delete stack again — When in real shit, call support

Slide 40

Slide 40 text

The first rule of Fight Club CloudFormation

Slide 41

Slide 41 text

Resources not provisioned by CloudFormation are not recognised by CloudFormation

Slide 42

Slide 42 text

Resources provisioned by CloudFormation but changed outside of CloudFormation have a high probability of failing in subsequent updates

Slide 43

Slide 43 text

Migrating to CloudFormation

Slide 44

Slide 44 text

Creating the templates — Use CloudFormer to instrospect your VPC. — http://docs.aws.amazon.com/AWSCloudFormation/ latest/UserGuide/cfn-using-cloudformer.html — http://aws.amazon.com/developertools/ 6460180344805680 — Uses a Micro instance inside your VPC to interrogate and dumps results to S3 — Parse with the ruby cloudformation-ruby-

Slide 45

Slide 45 text

Incorporating templates — Resources must be provisioned by CloudFormation — Use a phased approach. Use Ref to refer to existing resources, passed as parameters. — Usually in reverse of your pipeline — machines first, VPCs last — e.g. "VpcId" : { "Ref" : "VpcId" }

Slide 46

Slide 46 text

Policies — Use DependsOn and WaitConditions to control dependency creation — Use UpdatePolicy for ASG changes. — Use DeletionPolicy to ensure are not deleted by CloudFormation (instance termination protection on steroids) — Note the Update requires property in the documentation

Slide 47

Slide 47 text

Best practices — http://docs.aws.amazon.com/AWSCloudFormation/ latest/UserGuide/best-practices.html

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

Other — Test deploys in a sandbox environment that mirrors production. Smaller resource types can help reduce costs. — Parameterise everything. — Use the http://docs.aws.amazon.com/ AWSCloudFormation/latest/UserGuide/template- reference.html

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

Thanks :-) Steven Ringo [email protected]