Upgrade to Pro — share decks privately, control downloads, hide ads and more …

SparkleFormation: Build infrastructure with CloudFormation & keep your sanity.

98f9dfc2e5e1318ac78b8c716582cd30?s=47 portertech
November 14, 2014

SparkleFormation: Build infrastructure with CloudFormation & keep your sanity.

This is an ignite talk for DevOpsDays Vancouver, 2014.

The following was the proposal for the full session.


AWS CloudFormation provides a way to create, provision, and update a collection of AWS resources in a predictable manner. Unfortunately, CloudFormation uses JSON “templates” to describe the desired infrastructure, which tend to become unwieldy, nearly impossible to maintain and reason about.

SparkleFormation is a Ruby DSL for programmatically composing CloudFormation and OpenStack Heat templates. In this talk, I will cover the DSL’s “building blocks”, provide working examples, and share neat tricks. I will demonstrate a working template, one currently being used to stress test Sensu, the open source monitoring framework (http://sensuapp.org).



November 14, 2014


  1. Build infrastructure with CloudFormation & keep your sanity. SparkleFormation Sean

    Porter Heavy Water Operations
  2. Who am I? Sean Porter - @PorterTech Author of Sensu,

    the monitoring framework. github.com/portertech Work at Heavy Water Operations.
  3. What is CloudFormation? • Amazon AWS service • Provides an

    easy way to create AWS resources • Provisions resources in an orderly and predictable fashion • Uses JSON templates to describe AWS resources
  4. None
  5. 528 lines of JSON

  6. None
  7. What are we working with? • Data - HASHES! •

    We can build HASHES programmatically! • Just need enough DSL ◦ compile & merge ◦ functions
  8. SparkleFormation • Ruby DSL • Provides AWS intrinsic functions •

    Automatically compatible with future AWS resources • Small implementation ◦ Using AttributeStruct by Chris Roberts (@_spox) github.com/sparkleformation
  9. Template Anatomy • Parameters ◦ RabbitMQ nodes: x (prompt) •

    Resources ◦ type: AWS::AutoScaling::LaunchConfiguration • Mappings ◦ key/value pairs referenced at runtime • Outputs
  10. Functions • Ref ◦ Reference parameter & resource values ◦

    eg. min_size ref!(:rabbitmq_nodes) • Attr ◦ Retrieve certain resources attributes directly ◦ eg. attr!(:cfn_user, :secret_access_key) • Join ◦ eg. join!(ref!(:environment), '-', map!(:region_map, ref!('AWS::Region'), :ami))
  11. Building Blocks • Components ◦ static, reusable configuration • Dynamics

    ◦ arguments -> create unique resources ◦ eg. dynamic(:elb) do |_name, _config={}| • Registries
  12. None
  13. None
  14. None
  15. Documentation sparkleformation.github.io/sparkle_formation docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide **Resource Types Reference** LaunchConfigurationName -> launch_configuration_name

  16. Tooling • SparkleFormation is just a library • knife-cloudformation ◦

    Chef knife plugin (but no reliance on Chef) ◦ Builds CF templates from SparkleFormations ◦ Provides deep inspection into stacks ◦ github.com/heavywater/knife-cloudformation
  17. $ knife cloudformation create sensu-42 --file \ templates/sensu_enterprise.rb --processing

  18. What does it do? • Prompts for auto-scaling group node

    counts ◦ RabbitMQ, Redis, Sensu Enterprise, Uchiwa • Creates and uploads a CloudFormation template • Provisions auto-scaling groups, using Chef, in the defined order ◦ Bootstrap orchestration anyone?
  19. None
  20. Sean Porter Heavy Water Operations Thank you!