Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Why your configuration needs a schema

Why your configuration needs a schema

Talk from Configuration Management Camp 2018, all about the high cost of current configuration management approaches, why that leads to serialisation formats like YAML and JSON being edited directly, and how schemas and auto-generation can help.

Gareth Rushgrove

February 06, 2018

More Decks by Gareth Rushgrove

Other Decks in Technology


  1. - The proliferation of config file formats - The high

    cost of config management - Why configuration needs a schema - Auto-generate everything
  2. Multiple configuration management tools is not a bad thing, but

    it does mean lots of reinventing the wheel
  3. sous-chef/apache2 1498 commits, 116 contributors, 46 releases puppetlabs/puppetlabs-apache 2992 commits,

    342 contributors, 39 releases Ansible Galaxy 298 results for apache
  4. Option 1: Templates None of the benefits of your chosen

    tool, and you’re exposed to all the configuration file formats directly
  5. How can you reason about a system when the configuration

    is spread across an explosion of templating languages, file formats and templates?
  6. Option 2: Format-specific resources You can now use your chosen

    tool, but the tool has no context for the application, it’s just data, and the format still bleeds through
  7. ini_setting { "sample setting": ensure => present, path => '/tmp/foo.ini',

    section => 'bar', setting => 'baz', value => 'quux', } Manage an INI file with Puppet
  8. Import-DscResource -ModuleName DSCR_IniFile cIniFile Apple { Path = "C:\Test.ini" Section

    = "" Key = "Fruit_A" Value = "Apple" } Manage an INI file with DSC
  9. Option 3: App-specific resources You get all the power of

    your chosen tool, but at the cost of bespoke development
  10. Most Chef cookbooks or Ansible or Puppet modules are not

    written by the developers of the application being managed
  11. That’s a lot of JSON PS> Get-Content -Path swagger.json |

    Measure-Object -line).Lines 85340 PS> (Get-ChildItem -Path v*/*.json -Recurse | Measure-Object).Count 26181 PS> (Get-ChildItem -Path v*/*.json -Recurse | Get-Content | Measure-Object -line).Lines 7296392
  12. { "definitions": {}, "$schema": "http://json-schema.org/draft-06/schema#", "id": "app_config", "title": "app_config", "type":

    "object", "additionalProperties": false, "required": [ "STATIC_URL_PATH", "MYSQL_DATABASE_USER" ], "properties": { "STATIC_URL_PATH": { "$id": "/properties/STATIC_URL_PATH", "type": "string", "title": "Static URL path", "description": "A filesystem path for static assets", Let’s write a (JSON) schema
  13. "STATIC_URL_PATH": { "$id": "/properties/STATIC_URL_PATH", "type": "string", "title": "Static URL path",

    "description": "A filesystem path for static assets", "examples": [ "/my_project/static" ] }, Describe each individual property
  14. Validate config using the schemas $ jsonschema -F "{error.message}" -i

    app.json schema.json u'STATIC_URL_PATH' is a required property
  15. Validate arbitrary structures with JSON Schema import json import fastjsonschema

    data = { "STATIC_URL_PATH": "/my_project/static", "MYSQL_DATABASE_USER": "project_user_name", "SHOW_SETTINGS_ROUTE": "/show-settings", "DEBUG": True, } validate = fastjsonschema.compile(json.load(open('schema.json'))) validate(data) print(json.dumps(data))
  16. The JSON in JSON Schema refers to the syntax for

    the schema. It can be used to validate data in other formats
  17. Quicktype generating Simple Types $ docker run -v ${PWD}:/pwd quicktype

    -l types -s schema /pwd/schemas/schema.json class Schema { staticURLPath: String mysqlDatabaseUser: String showSettingsRoute: Maybe<String> debug: Maybe<Bool> }
  18. Quicktype generating Go $ docker run -v ${PWD}:/pwd quicktype -l

    go -s schema /pwd/schemas/schema.json // To parse and unparse this JSON data, add this code to your project and do: // // r, err := UnmarshalSchema(bytes) // bytes, err = r.Marshal() package main import "encoding/json" func UnmarshalSchema(data []byte) (Schema, error) { var r Schema err := json.Unmarshal(data, &r) return r, err } func (r *Schema) Marshal() ([]byte, error) {
  19. Dynamically build objects from schemas import python_jsonschema_objects as pjs import

    json schema = json.load(open('schema.json')) builder = pjs.ObjectBuilder(schema) ns = builder.build_classes() Config = ns.AppConfig config = Config( STATIC_URL_PATH="/static", MYSQL_DATABASE_USER="db", )
  20. Generate Chef resource from schema $ ./to_chef.py resource_name :app_config property

    :path, String, name_property: true property :static_url_path, String, required: true property :mysql_database_user, String, required: true property :show_settings_route, String, default: '/settings' property :debug, Boolean action :create do file path do content "{ STATIC_URL_PATH: "#{static_url_path}", MYSQL_DATABASE_USER: "#{mysql_database_user}", SHOW_SETTINGS_ROUTE: "#{show_settings_route}", DEBUG: #{debug} }"
  21. Generate Libral provider from schema $ ./to_libral.py #! /usr/bin/python import

    json import sys import os METADATA=""" --- provider: type: app_config invoke: json actions: [get,set] suitable: true attributes: path: desc: The filepath for the configuration file
  22. Generate Puppet type from schema $ ./to_puppet.py Puppet::Type.newtype(:app_config) do ensurable

    validate do required_properties = [ :static_url_path, :mysql_database_user, ] required_properties.each do |property| if self[property].nil? and self.provider.send(property) == :absent fail "You must provide a #{property}" end end end newparam(:path, namevar: true) do