Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Ansible vs CloudFormation Smackdown!

Steven Ringo
October 21, 2015

Ansible vs CloudFormation Smackdown!

A quick comparison of Ansible and CloudFormation followed by some best practices for CloudFormation.

Accompanying code at https://github.com/stevenringo/cloudformation-ruby-ansible-examples

Steven Ringo

October 21, 2015
Tweet

More Decks by Steven Ringo

Other Decks in Programming

Transcript

  1. CloudFormation
    vs
    Ansible

    View Slide

  2. Smackdown!

    View Slide

  3. What is CloudFormation?
    “...an easy way to create and manage a collection of
    related AWS resources, provisioning and updating them in
    an orderly and predictable fashion.”
    https://aws.amazon.com/cloudformation/

    View Slide

  4. Ansible
    — Many platforms
    — Provisioning and configuring

    View Slide

  5. CloudFormation
    — AWS platform only
    — Provisioning

    View Slide

  6. Why CloudFormation then?

    View Slide

  7. |--------------|---------------|----------------|
    | Feature | Ansible | CloudFormation |
    |==============|===============|================|
    | Written in | YAML | JSON |
    |--------------|---------------|----------------|
    | Grouped as | Playbook | Stack |
    |--------------|---------------|----------------|
    | State | Stateless | Keeps state |
    |--------------|---------------|----------------|
    | Deletion | Manual | Automatic |
    |--------------|---------------|----------------|
    | Dependencies | Manual | Automatic |
    |--------------|---------------|----------------|
    | Ordering | Top to bottom | Simultaneous |
    |--------------|---------------|----------------|

    View Slide

  8. |--------------------|------------------|-----------------|
    | Feature | Ansible | CloudFormation |
    |====================|==================|=================|
    | UI | CLI/Tower | CLI/AWS Console |
    |--------------------|------------------|-----------------|
    | Reusability | Roles / includes | Nested Stacks |
    |--------------------|------------------|-----------------|
    | Rollbacks | No | Yes |
    |--------------------|------------------|-----------------|
    | Existing resources | Can incorporate | No |
    |--------------------|------------------|-----------------|
    | AWS Services | Limited | Most |
    |--------------------|------------------|-----------------|
    | Ease | Easier | Harder |
    |--------------------|------------------|-----------------|
    | Triggers/Policies | No | Yes |
    |--------------------|------------------|-----------------|

    View Slide

  9. Writing CloudFormation
    templates

    View Slide

  10. Don't!

    View Slide

  11. Use your favourite language
    Generate!

    View Slide

  12. https://github.com/
    bazaarvoice/
    cloudformation-ruby-dsl

    View Slide

  13. Other generators
    — https://github.com/cloudtools/troposphere (Python)
    — https://github.com/tongueroo/lono (Ruby)
    — https://github.com/MonsantoCo/cloudformation-
    template-generator (Scala)

    View Slide

  14. Organising

    View Slide

  15. God stack
    — Single stack for all resources
    — Good for small deployments
    — Size limits
    — Reuse only by using a generator tool

    View Slide

  16. Nested stacks
    ┌─────────────────┐
    │ │
    │ │
    │ Solution │
    │ │
    │ │
    └─────────────────┘

    ┌───────────────────┬─────────┴─────────┬───────────────────┐
    ▼ ▼ ▼ ▼
    ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
    │ │ │ │ │ │ │ │
    │ │ │ │ │ │ │ │
    │ VPC │ │ Database │ │ Servers │ │ Security Groups │
    │ │ │ │ │ │ │ │
    │ │ │ │ │ │ │ │
    └─────────────────┘ └─────────────────┘ └─────────────────┘ └─────────────────┘

    View Slide

  17. Nested stacks
    — CloudFormation invokes child templates
    — Atomic execution
    — One fails, all fail
    — Pass parameters through "references"
    — Need to pass parameters for child template "through"
    the parent

    View Slide

  18. Pipelined stacks
    Executor
    ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
    │ 1 │ │ 2 │ │ 3 │ │ 4 │
    │ │ │ │ │ │ │ │
    │ VPC │ │ Database │ │ Servers │ │ Security Groups │
    │ │ │ │ │ │ │ │
    │ │ │ │ │ │ │ │
    └─────────────────┘ └─────────────────┘ └─────────────────┘ └─────────────────┘

    View Slide

  19. Pipelined stacks
    — Execute each template manually, requires
    orchestration
    — Non-atomic execution
    — One fails, only that template fails
    — Pass parameters directly
    — Chain outputs to inputs

    View Slide

  20. Partitioning

    View Slide

  21. Horizontal
    ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
    │ │ │ Database │ │ Servers │ │ Security Groups │
    │ │ ├─────────────────┤ ├─────────────────┤ ├─────────────────┤
    │ │ │ Application 1 │ │ Application 1 │ │ Application 1 │
    │ VPC │ │ │ ├─────────────────┤ ├─────────────────┤
    │ │ ├─────────────────┤ │ Application 2 │ │ Application 2 │
    │ │ │ Application 2 │ ├─────────────────┤ ├─────────────────┤
    │ │ │ │ │ Application 3 │ │ Application 3 │
    └─────────────────┘ └─────────────────┘ └─────────────────┘ └─────────────────┘

    View Slide

  22. Vertical
    ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
    │ │ │ Application 1 │ │ Application 2 │ │ Application 3 │
    │ │ ├─────────────────┤ ├─────────────────┤ ├─────────────────┤
    │ │ │ Servers │ │ Servers │ │ │
    │ VPC │ ├─────────────────┤ ├─────────────────┤ │ Servers │
    │ │ │ Database │ │ Database │ │ │
    │ │ ├─────────────────┤ ├─────────────────┤ ├─────────────────┤
    │ │ │ Security Groups │ │ Security Groups │ │ Security Groups │
    └─────────────────┘ └─────────────────┘ └─────────────────┘ └─────────────────┘

    View Slide

  23. Partitioning
    — Keep templates simple
    — Reuse parameters and mappings via a DSL
    — Choose vertical or horizontal scaling
    — Decide if you want to rollback the stack or not

    View Slide

  24. Orchestration

    View Slide

  25. Shell
    — Simple batch script
    — Use aws-cli to invoke CloudFormation templates:
    aws cloudformation create-stack...
    — Capture output variables for input variables for next
    stack
    — Returns immediately, therefore must check status:
    aws cloudformation create-describe-
    stacks...

    View Slide

  26. Ansible
    - name: launch ansible cloudformation example
    cloudformation:
    stack_name: ansible-cloudformation
    state: present
    region: us-east-1
    disable_rollback: true
    template: files/cloudformation-example.json
    template_parameters:
    KeyName: jmartin
    DiskType: ephemeral
    InstanceType: m1.small
    ClusterSize: 3
    tags:
    Stack: ansible-cloudformation

    View Slide

  27. Ansible
    - include: key_pairs.yml
    tags: always
    - include: compile.yml
    tags: always
    - include: vpc.yml
    tags: always
    - include: nat.yml
    tags: nat
    - include: titan.yml
    tags: titan

    View Slide

  28. Ansible
    - name: delete old cloudformation templates
    file:
    path: "{{ item }}"
    state: absent
    with_fileglob: ../cloudformation/compiled/*.json
    when: clean is defined and clean|bool
    - name: compile the cloudformation templates
    shell: "{{ item }} expand --nopretty"
    with_fileglob: ../cloudformation/dsl/*.rb
    register: compiled
    - name: write and validate templates
    template:
    src: templates/cloudformation.json
    dest: cloudformation/compiled/{{ item.item | basename | regex_replace('\.rb$', '.json') }}
    validate: aws cloudformation validate-template --region {{ region }} --template-body file://%s
    environment:
    AWS_ACCESS_KEY_ID: "{{ aws_credentials[env].access_key }}"
    AWS_SECRET_ACCESS_KEY: "{{ aws_credentials[env].secret_key }}"
    with_items: compiled.results

    View Slide

  29. Ansible
    - name: execute vpc cloudformation template
    cloudformation:
    aws_access_key: "{{ aws_credentials[env].access_key }}"
    aws_secret_key: "{{ aws_credentials[env].secret_key }}"
    stack_name: vpc-{{ env }}
    region: "{{ region }}"
    template: ./cloudformation/compiled/vpc.json
    disable_rollback: true
    template_parameters:
    Environment: "{{ env }}"
    tags:
    Environment: "{{ env }}"
    Region: "{{ region }}"
    register: vpc

    View Slide

  30. Ansible
    - name: execute titan cloudformation template
    cloudformation:
    aws_access_key: "{{ aws_credentials[env].access_key }}"
    aws_secret_key: "{{ aws_credentials[env].secret_key }}"
    stack_name: titan-{{ env }}
    region: "{{ region }}"
    template: ./cloudformation/compiled/titan.json
    disable_rollback: true
    template_parameters:
    Environment: "{{ env }}"
    CochlearLinkHostedZone: "{{ vpc.stack_outputs.HostedZone }}"
    CochlearLinkVpc: "{{ vpc.stack_outputs.Vpc }}"
    TitanSubnets: "{{ vpc.stack_outputs.TitanSubnets }}"
    tags:
    Environment: "{{ env }}"
    Region: "{{ region }}"
    register: titan

    View Slide

  31. Configuration

    View Slide

  32. Configuration
    — Keep configuration in Git, S3 or DynamoDB
    — raw files from GitHub work well

    View Slide

  33. {
    "ap-southeast-2": {
    "dev": {
    "productVersion": "3.0.0",
    "installationId": "eeeeeeee-f057-40c7-b365-792cc7faeccb"
    },
    "sit": {
    "productVersion": "3.1.8",
    "installationId": "17171717-f057-40c7-b365-792cc7faeccb"
    }
    },
    "us-east-1": {
    "prd": {
    "productVersion": "2.1.0",
    "installationId": "e047ac6f-f057-40c7-b365-792cc7faeccb"
    }
    }
    }

    View Slide

  34. {
    "components": {
    "3.1.8": {
    "app": "3.1.8",
    "db": "1.0.1",
    }
    }
    }

    View Slide

  35. {
    "versions": {
    "3.1.8": {
    "ap-southeast-2": {
    "image": "ami-9f266aa5"
    },
    "eu-west-1": {
    "image": "ami-0914307e"
    }
    },
    "3.1.2": {
    "ap-southeast-2": {
    "image": "ami-170e422d"
    },
    "eu-west-1": {
    "image": "ami-5dbf9a2a"
    }
    }
    }
    }

    View Slide

  36. - name: get environments config
    uri:
    url: https://your-git-repo.com/application/config/raw/master/environments.json
    return_content: yes
    register: environments_meta_raw

    View Slide

  37. Getting out of trouble

    View Slide

  38. The dreaded _FAILED states
    CREATE_FAILED
    DELETE_FAILED
    ROLLBACK_FAILED
    UPDATE_ROLLBACK_FAILED

    View Slide

  39. The dreaded _FAILED states
    — Resolve manually
    — Console will give you indication of which resource
    failed to update
    — Delete offending resource or its dependencies
    manually
    — Run delete stack again
    — When in real shit, call support

    View Slide

  40. The first rule of
    Fight Club CloudFormation

    View Slide

  41. Resources
    not provisioned by
    CloudFormation are
    not recognised
    by CloudFormation

    View Slide

  42. Resources provisioned by
    CloudFormation but
    changed outside of
    CloudFormation have a
    high probability of failing in
    subsequent updates

    View Slide

  43. Migrating to
    CloudFormation

    View Slide

  44. Creating the templates
    — Use CloudFormer to instrospect your VPC.
    — http://docs.aws.amazon.com/AWSCloudFormation/
    latest/UserGuide/cfn-using-cloudformer.html
    — http://aws.amazon.com/developertools/
    6460180344805680
    — Uses a Micro instance inside your VPC to interrogate
    and dumps results to S3
    — Parse with the ruby cloudformation-ruby-

    View Slide

  45. Incorporating templates
    — Resources must be provisioned by CloudFormation
    — Use a phased approach. Use Ref to refer to existing
    resources, passed as parameters.
    — Usually in reverse of your pipeline — machines first,
    VPCs last
    — e.g. "VpcId" : { "Ref" : "VpcId" }

    View Slide

  46. Policies
    — Use DependsOn and WaitConditions to control
    dependency creation
    — Use UpdatePolicy for ASG changes.
    — Use DeletionPolicy to ensure are not deleted by
    CloudFormation (instance termination protection on
    steroids)
    — Note the Update requires property in the
    documentation

    View Slide

  47. Best practices
    — http://docs.aws.amazon.com/AWSCloudFormation/
    latest/UserGuide/best-practices.html

    View Slide

  48. View Slide

  49. Other
    — Test deploys in a sandbox environment that mirrors
    production. Smaller resource types can help reduce
    costs.
    — Parameterise everything.
    — Use the http://docs.aws.amazon.com/
    AWSCloudFormation/latest/UserGuide/template-
    reference.html

    View Slide

  50. View Slide

  51. Thanks :-)
    Steven Ringo
    [email protected]

    View Slide