Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Let them configure! by Łukasz Langa

Let them configure! by Łukasz Langa

PyCon 2013

March 17, 2013
Tweet

More Decks by PyCon 2013

Other Decks in Technology

Transcript

  1. Click icon to add picture Robert M. Pirsig • „The

    solutions all are simple – after you have arrived at them. • But they're simple only when you know already what they are.”
  2. Easy-ish formats [user] email = [email protected] name = Łukasz Langa

    [push] default = current [color] branch = auto diff = auto interactive = auto status = auto [diff] external = git_vimdiff [pager] diff = [core] excludesfile = ~/.gitignore • INI • Apache • nginx • JSON
  3. Easy-ish formats Listen 80 User apache Group apache DocumentRoot "/var/www/vhosts/root"

    <Directory /> Options FollowSymLinks AllowOverride None </Directory> <Directory "/var/www/vhosts"> Options Indexes FollowSymLinks MultiViews ExecCGI AllowOverride All • INI • Apache • nginx • JSON
  4. Easy-ish formats Listen 80 User apache Group apache DocumentRoot "/var/www/vhosts/root"

    <Directory /> Options FollowSymLinks AllowOverride None </Directory> <Directory "/var/www/vhosts"> Options Indexes FollowSymLinks MultiViews ExecCGI AllowOverride All • INI • Apache • nginx • JSON What’s the difference: Options +Indexes Includes MultiViews Options Indexes Includes MultiViews
  5. Easy-ish formats server { listen 80; server_name d.com *.d.com; rewrite

    ^ http://www.d.com $request_uri? permanent; } server { listen 80 default; server_name www.d.com; index index.html; root /home/d.com; location /forum { rewrite forum(.*) http:// forum.d.com$1 • INI • Apache • nginx • JSON
  6. Easy-ish formats { "production":{ "phpSettings":{ "display_startup_errors": false, "display_errors": false },

    "includePaths":{ "library": "APPLICATION_PATH/../library" }, "bootstrap":{ "path": "APPLICATION_PATH/Bootstrap.php ", "class": "Bootstrap" }, • INI • Apache • nginx • JSON
  7. Easy-ish formats # This is a TOML document. Boom. title

    = "TOML Example" [owner] name = "Tom Preston-Werner" organization = "GitHub" bio = "GitHub Cofounder\n& CEO" dob = 1979-05-27T07:32:00Z [database] server = "192.168.1.1" ports = [ 8001, 8001, 8002 ] connection_max = 5000 enabled = true [servers] # You can indent as you please. [servers.alpha] ip = "10.0.0.1" # inline comments [servers.beta] ip = "10.0.0.2" [clients] data = [ ["gamma", "delta"], [1, 2] ] • INI • Apache • nginx • JSON • TOML?
  8. Easy-ish formats # This is a TOML document. Boom. title

    = "TOML Example" [owner] name = "Tom Preston-Werner" organization = "GitHub" bio = "GitHub Cofounder\n& CEO" dob = 1979-05-27T07:32:00Z [database] server = "192.168.1.1" ports = [ 8001, 8001, 8002 ] connection_max = 5000 enabled = true [servers] # You can indent as you please. [servers.alpha] ip = "10.0.0.1" # inline comments [servers.beta] ip = "10.0.0.2" [clients] data = [ ["gamma", "delta"], [1, 2] ] • INI • Apache • nginx • JSON • TOML? Strings are single-line values surrounded by double quotes encoded in UTF-8. vs. \xXX - byte (0x00-0xFF)
  9. Easy-ish formats # This is a TOML document. Boom. title

    = "TOML Example" [owner] name = "Tom Preston-Werner" organization = "GitHub" bio = "GitHub Cofounder\n& CEO" dob = 1979-05-27T07:32:00Z [database] server = "192.168.1.1" ports = [ 8001, 8001, 8002 ] connection_max = 5000 enabled = true [servers] # You can indent as you please. [servers.alpha] ip = "10.0.0.1" # inline comments [servers.beta] ip = "10.0.0.2" [clients] data = [ ["gamma", "delta"], [1, 2] ] • INI • Apache • nginx • JSON • TOML?
  10. Complex formats <configure xmlns="http://namespaces.zope.o rg/zope" xmlns:zmi="http://namespaces.zo pe.org/zmi" xmlns:browser="http://namespace s.zope.org/browser" >

    <browser:defaultView for="zope.i18n.interfaces.ITran slationService” name="index.html” • XML • YAML
  11. Complex formats application: myapp version: 1 runtime: python27 api_version: 1

    threadsafe: true handlers: - url: / script: home.app - url: /index\.html script: home.app - url: /stylesheets static_dir: stylesheets - url: /(.*\.(gif|png|jpg)) static_files: static/\1 upload: static/(.*\.(gif|png| • XML • YAML
  12. Turing complete formats DEBUG = True TEMPLATE_DEBUG = DEBUG DATABASES

    = { 'default': { 'ENGINE': 'django.db.backends.sqlite3', 'NAME': '/tmp/project.db', 'USER': '', 'PASSWORD': '', 'HOST': '', 'PORT': '', } } • Python
  13. Awkward formats Feature: Addition In order to avoid mistakes As

    a math slouch I want to be told the sum of two numbers Scenario: Add two numbers Given I have entered 50 into the calc And I have entered 70 into the calc When I press add Then the result should be 120 • Domain-Specific Languages • SQLite • Windows registry
  14. Awkward formats class davids_black_co_at { hosting::user { rztt: realname =>

    "Gerhard Schmitt", uid => 2001, admin => true; conny: realname => "Conny Schmitt", uid => 2002; oma: realname => "Oma Schmitt", uid => 2003; } # Install git.black.co.at include git::daemon • Domain-Specific Languages • SQLite • Windows registry
  15. Awkward formats # main.cf ... alias_maps = sqlite:/etc/postfix/sqlite- aliases.cf ...

    # sqlite config file for # local(8) aliases(5) lookups dbpath = /path/to/sqlite_database query = SELECT forw_addr FROM mxaliases WHERE alias='%s’ • Domain-Specific Languages • SQLite • Windows registry
  16. Awkward formats sqlite> .tables directory handler log proxy server statistic

    filter host mimetype route setting sqlite> select * from host; 1|1|0|localhost|localhost sqlite> select count(*) from mimetype; 850 sqlite> select * from server; • Domain-Specific Languages • SQLite • Windows registry
  17. Awkward formats sqlite> .tables directory handler log proxy server statistic

    filter host mimetype route setting sqlite> select * from host; 1|1|0|localhost|localhost sqlite> select count(*) from mimetype; 850 sqlite> select * from server; • Domain-Specific Languages • SQLite • Windows registry # get s list of the available servers to run m2sh servers -db tests/config.sqlite # see what hosts a server has m2sh hosts -db tests/config.sqlite -server test # find out if a server named 'test' is running m2sh running -db tests/config.sqlite -name test # start a server who's default host is 'localhost' m2sh start -db tests/config.sqlite -host localhost
  18. ambv@arrakis:~ $ wget -O - "https://www.dropbox.com/download? plat=lnx.x86_64" | tar xzf

    - --2012-10-20 11:24:22-- https://www.dropbox.com/download? plat=lnx.x86_64 Resolving www.dropbox.com... 199.47.217.171, 199.47.216.170, 199.47.216.171, ... Connecting to www.dropbox.com|199.47.217.171|:443... connected. HTTP request sent, awaiting response... 302 FOUND Location: https://dl-web.dropbox.com/u/17/dropbox-lnx.x86_64- 1.4.17.tar.gz [following] --2012-10-20 11:24:23-- https://dl-web.dropbox.com/u/17/dropbox- lnx.x86_64-1.4.17.tar.gz Resolving dl-web.dropbox.com... 174.129.197.250, 174.129.199.91, 107.20.174.220, ... Connecting to dl-web.dropbox.com|174.129.197.250|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 19090076 (18M) [application/x-tar] Saving to: `STDOUT' 100% [=============================================================>] 19,090,076 1.23M/s in 17s
  19. ambv@arrakis:~ $ wget -O - "https://www.dropbox.com/download? plat=lnx.x86_64" | tar xzf

    - 100%[======>] 19,090,076 1.23M/s in 17s ambv@arrakis:~ $ ./.dropbox- dist/dropboxd This client is not linked to any account... Please visit https://www.dropbox.com/cli_link?
  20. Excerpts from >>> import this Beautiful is better than ugly.

    Simple is better than complex. Flat is better than nested. Readability counts. There should be one (and preferably only one) obvious way to do it.
  21. Django settings.py Django mixes different kinds of settings • Framework

    behaviour • Application behaviour • Deployment-specific configuration • databases • caches • logging • paths • ADMINS • MANAGERS • DEBUG • Sensitive data • passwords • SECURE_KEY • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
  22. Django settings.py # settings.py # ... TIME_ZONE = 'Europe/Zurich' LANGUAGE_CODE

    = 'en-us' SECRET_KEY = 'secret' # ... from settings_local import * • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
  23. Django settings.py # settings/__init__.py # ... TIME_ZONE = 'Europe/Zurich' LANGUAGE_CODE

    = 'en-us' SECRET_KEY = 'secret' # ... try: from .local import * except ImportError: pass • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
  24. Django settings.py Alternative approach • settings.py is never used •

    settings_base.py defines the base configuration • settings_prod.py, settings_dev.py, etc. import settings_base.py at the beginning • run with DJANGO_SETTINGS_MODULE = proj.settings_prod • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
  25. Django settings.py # settings.py import os.path import glob conffiles =

    glob.glob( os.path.join( os.path.dirname(__file__), 'settings', '*.conf’ ) ) conffiles.sort() for f in conffiles: execfile(os.path.abspath(f )) • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
  26. Django settings.py # settings.py import os.path import glob conffiles =

    glob.glob( os.path.join( os.path.dirname(__file__), 'settings', '*.conf’ ) ) conffiles.sort() for f in conffiles: execfile(os.path.abspath(f )) • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration $ ls settings/ 10-base.conf 20-engines.conf 30-site.conf ...
  27. Django settings.py from configurations import Settings class Base(Settings): TIME_ZONE =

    'Europe/Berlin' class Dev(Base): DEBUG = True TEMPLATE_DEBUG = DEBUG class Prod(Base): TIME_ZONE = 'America/New_York’ • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
  28. Django settings.py from configurations import Settings class FullPageCaching(object): USE_ETAGS =

    True class Base(Settings): TIME_ZONE = 'Europe/Berlin' class Dev(Base): DEBUG = True TEMPLATE_DEBUG = DEBUG class Prod(Base, FullPageCaching): • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
  29. Django settings.py export DJANGO_CONFIGURATION=Dev or python manage.py runserver --settings=mysite.settings --configuration=Dev

    • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
  30. Django settings.py [database] DATABASE_USER = db-user DATABASE_PASSWORD = db-secret DATABASE_HOST

    = 127.0.0.1 DATABASE_PORT = 3306 DATABASE_ENGINE = mysql DATABASE_NAME = example TESTSUITE_DATABASE_NAME = test_example [secrets] SECRET_KEY = 12345678... CSRF_MIDDLEWARE_SECRET = 12345678... [debug] • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
  31. Django settings.py # production environment prod.ini [database] DATABASE_PASSWORD = db-secret

    [secrets] SECRET_KEY = 12345678... CSRF_MIDDLEWARE_SECRET = 12345678... [debug] DEBUG = false TEMPLATE_DEBUG = false VIEW_TEST = false INTERNAL_IPS = 127.0.0.1 SKIP_CSRF_MIDDLEWARE = false • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
  32. Django settings.py >>> c = ConfigParser() >>> c.read([‘config-base.ini', ‘config- prod.ini'])

    You can do a hierarchy like: 1. site-packages/project/config- defaults.ini 2. /etc/project/config.ini 3. /etc/project.d/10-config.ini 4. /etc/project.d/20-config.ini 5. ~/.project/config.ini 6. ENVIRONMENT_VARIABLE • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
  33. Mixing file-based configuration with command-line overrides • Use configglue •

    Checkout out ConfArgParse • Whatever you do, name your command-line arguments on par with the file configuration settings
  34. The old way cp.readfp(sio) if cp.has_section('user'): print(cp.get('user', 'email')) cp.set('user', 'password',

    'secret') cp.set('user', 'jabber', '% (email)s') print(cp.get('user', 'jabber'))
  35. The old way cp.readfp(sio) if cp.has_section('user'): print(cp.get('user', 'email')) cp.set('user', 'password',

    'secret') cp.set('user', 'jabber', '% (email)s') print(cp.get('user', 'jabber')) This reminds me of: >>> import string >>> string.split('hakuna matata') ['hakuna', 'matata'] >>> {}.has_key('to your heart') False >>> bool('Yuck') True
  36. The old way if cp.has_section('user'): print(cp.get('user', 'email')) cp.set('user', 'password', 'secret')

    cp.set('user', 'jabber', '% (email)s') print(cp.get('user', 'jabber'))
  37. The new way if 'user' in cp: user = cp['user']

    print(user['email']) user['password'] = 'secret' user['jabber'] = '%(email)s' print(cp['user']['jabber'])
  38. Configuration from a string cfg = """ [DEFAULT] host =

    localhost port = 8080 [user] name = ambv email = [email protected] """
  39. Configuration from a dictionary cfg = dict( DEFAULT = dict(

    host = 'localhost', port = '8080’, ), USER = dict( name = 'ambv', email = '[email protected]’, ), )
  40. Configuration from a dictionary cp = ConfigParser() cp.read_dict(cfg) cp2 =

    ConfigParser() cp2.read_dict(cp) assert cp == cp2
  41. Creating sections from a dictionary cp['ui'] = { 'askusername': 'yes',

    'fallbackencoding': 'utf-8', 'ignore': '~/.hgignore', } print(cp['ui'].items())
  42. Creating sections from a dictionary cp['ui'] = { 'askusername': 'yes',

    'fallbackencoding': 'utf-8', 'ignore': '~/.hgignore', } print(cp['ui'].items()) The output: [('ignore', '~/.hgignore'), ('askusername', 'yes'), ('fallbackencoding', 'utf-8'), ('host', 'localhost'), ('port', '8080')]
  43. Buildout-inspired interpolation from configparser import ConfigParser,\ ExtendedInterpolation cp = ConfigParser(interpolation=

    ExtendedInterpolation()) cp.read_string(""" [general] project_dir = /opt/epic_project debug = False [cms] path = ${general:project_dir}/cms debug = ${general:debug}
  44. Buildout-inspired interpolation from configparser import ConfigParser,\ ExtendedInterpolation cp = ConfigParser(interpolation=

    ExtendedInterpolation()) cp.read_string(""" [general] project_dir = /opt/epic_project debug = False [cms] path = ${general:project_dir}/cms debug = ${general:debug} Transitive interpolation: >>> cp['cms-plugin'].items() [('path', '/opt/epic_project/cms/plugin’), ('debug', 'False')]