SparkleFormation: Build infrastructure with CloudFormation & keep your sanity.

This is an ignite talk for DevOpsDays Vancouver, 2014.

The following was the proposal for the full session.

Abstract:

AWS CloudFormation provides a way to create, provision, and update a collection of AWS resources in a predictable manner. Unfortunately, CloudFormation uses JSON “templates” to describe the desired infrastructure, which tend to become unwieldy, nearly impossible to maintain and reason about.

SparkleFormation is a Ruby DSL for programmatically composing CloudFormation and OpenStack Heat templates. In this talk, I will cover the DSL’s “building blocks”, provide working examples, and share neat tricks. I will demonstrate a working template, one currently being used to stress test Sensu, the open source monitoring framework (http://sensuapp.org).

Transcript

1. Build infrastructure with CloudFormation & keep your sanity. SparkleFormation Sean Porter Heavy Water Operations
2. Who am I? Sean Porter - @PorterTech Author of Sensu, the monitoring framework. github.com/portertech Work at Heavy Water Operations.
3. What is CloudFormation? • Amazon AWS service • Provides an easy way to create AWS resources • Provisions resources in an orderly and predictable fashion • Uses JSON templates to describe AWS resources
5. 528 lines of JSON
7. What are we working with? • Data - HASHES! • We can build HASHES programmatically! • Just need enough DSL ◦ compile & merge ◦ functions
8. SparkleFormation • Ruby DSL • Provides AWS intrinsic functions • Automatically compatible with future AWS resources • Small implementation ◦ Using AttributeStruct by Chris Roberts (@_spox) github.com/sparkleformation
9. Template Anatomy • Parameters ◦ RabbitMQ nodes: x (prompt) • Resources ◦ type: AWS::AutoScaling::LaunchConfiguration • Mappings ◦ key/value pairs referenced at runtime • Outputs
10. Functions • Ref ◦ Reference parameter & resource values ◦ eg. min_size ref!(:rabbitmq_nodes) • Attr ◦ Retrieve certain resources attributes directly ◦ eg. attr!(:cfn_user, :secret_access_key) • Join ◦ eg. join!(ref!(:environment), '-', map!(:region_map, ref!('AWS::Region'), :ami))
11. Building Blocks • Components ◦ static, reusable configuration • Dynamics ◦ arguments -> create unique resources ◦ eg. dynamic(:elb) do |_name, _config={}| • Registries
15. Documentation sparkleformation.github.io/sparkle_formation docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide **Resource Types Reference** LaunchConfigurationName -> launch_configuration_name
16. Tooling • SparkleFormation is just a library • knife-cloudformation ◦ Chef knife plugin (but no reliance on Chef) ◦ Builds CF templates from SparkleFormations ◦ Provides deep inspection into stacks ◦ github.com/heavywater/knife-cloudformation
17. $ knife cloudformation create sensu-42 --file \ templates/sensu_enterprise.rb --processing
18. What does it do? • Prompts for auto-scaling group node counts ◦ RabbitMQ, Redis, Sensu Enterprise, Uchiwa • Creates and uploads a CloudFormation template • Provisions auto-scaling groups, using Chef, in the defined order ◦ Bootstrap orchestration anyone?
20. Sean Porter Heavy Water Operations Thank you!