Let them configure! by Łukasz Langa

Let them configure! by Łukasz Langa

Afcfefa1f067d10bd021de0cc2e5e806?s=128

PyCon 2013

March 17, 2013
Tweet

Transcript

  1. a sermon on reusability Let them configure!

  2. Łukasz Langa ambv at #python-dev (FreeNode IRC) @llanga on Twitter

    lukasz@langa.pl as a last resort
  3. None
  4. None
  5. None
  6. Click icon to add picture Robert M. Pirsig • „The

    solutions all are simple – after you have arrived at them. • But they're simple only when you know already what they are.”
  7. four desirable characteristics of configuration

  8. COMPOSABLE

  9. READABLE

  10. EXCHANGEAB LE

  11. DISCOVERAB LE

  12. COMPOSABLE READABLE EXCHANGEABLE DISCOVERABLE

  13. configuration format whirlwind tour

  14. Easy-ish formats [user] email = lukasz@langa.pl name = Łukasz Langa

    [push] default = current [color] branch = auto diff = auto interactive = auto status = auto [diff] external = git_vimdiff [pager] diff = [core] excludesfile = ~/.gitignore • INI • Apache • nginx • JSON
  15. Easy-ish formats Listen 80 User apache Group apache DocumentRoot "/var/www/vhosts/root"

    <Directory /> Options FollowSymLinks AllowOverride None </Directory> <Directory "/var/www/vhosts"> Options Indexes FollowSymLinks MultiViews ExecCGI AllowOverride All • INI • Apache • nginx • JSON
  16. Easy-ish formats Listen 80 User apache Group apache DocumentRoot "/var/www/vhosts/root"

    <Directory /> Options FollowSymLinks AllowOverride None </Directory> <Directory "/var/www/vhosts"> Options Indexes FollowSymLinks MultiViews ExecCGI AllowOverride All • INI • Apache • nginx • JSON What’s the difference: Options +Indexes Includes MultiViews Options Indexes Includes MultiViews
  17. Easy-ish formats server { listen 80; server_name d.com *.d.com; rewrite

    ^ http://www.d.com $request_uri? permanent; } server { listen 80 default; server_name www.d.com; index index.html; root /home/d.com; location /forum { rewrite forum(.*) http:// forum.d.com$1 • INI • Apache • nginx • JSON
  18. Easy-ish formats { "production":{ "phpSettings":{ "display_startup_errors": false, "display_errors": false },

    "includePaths":{ "library": "APPLICATION_PATH/../library" }, "bootstrap":{ "path": "APPLICATION_PATH/Bootstrap.php ", "class": "Bootstrap" }, • INI • Apache • nginx • JSON
  19. Easy-ish formats • INI • Apache • nginx • JSON

    WHAT ABOUT TOML?
  20. Easy-ish formats # This is a TOML document. Boom. title

    = "TOML Example" [owner] name = "Tom Preston-Werner" organization = "GitHub" bio = "GitHub Cofounder\n& CEO" dob = 1979-05-27T07:32:00Z [database] server = "192.168.1.1" ports = [ 8001, 8001, 8002 ] connection_max = 5000 enabled = true [servers] # You can indent as you please. [servers.alpha] ip = "10.0.0.1" # inline comments [servers.beta] ip = "10.0.0.2" [clients] data = [ ["gamma", "delta"], [1, 2] ] • INI • Apache • nginx • JSON • TOML?
  21. Easy-ish formats # This is a TOML document. Boom. title

    = "TOML Example" [owner] name = "Tom Preston-Werner" organization = "GitHub" bio = "GitHub Cofounder\n& CEO" dob = 1979-05-27T07:32:00Z [database] server = "192.168.1.1" ports = [ 8001, 8001, 8002 ] connection_max = 5000 enabled = true [servers] # You can indent as you please. [servers.alpha] ip = "10.0.0.1" # inline comments [servers.beta] ip = "10.0.0.2" [clients] data = [ ["gamma", "delta"], [1, 2] ] • INI • Apache • nginx • JSON • TOML? Strings are single-line values surrounded by double quotes encoded in UTF-8. vs. \xXX - byte (0x00-0xFF)
  22. Easy-ish formats # This is a TOML document. Boom. title

    = "TOML Example" [owner] name = "Tom Preston-Werner" organization = "GitHub" bio = "GitHub Cofounder\n& CEO" dob = 1979-05-27T07:32:00Z [database] server = "192.168.1.1" ports = [ 8001, 8001, 8002 ] connection_max = 5000 enabled = true [servers] # You can indent as you please. [servers.alpha] ip = "10.0.0.1" # inline comments [servers.beta] ip = "10.0.0.2" [clients] data = [ ["gamma", "delta"], [1, 2] ] • INI • Apache • nginx • JSON • TOML?
  23. Complex formats <configure xmlns="http://namespaces.zope.o rg/zope" xmlns:zmi="http://namespaces.zo pe.org/zmi" xmlns:browser="http://namespace s.zope.org/browser" >

    <browser:defaultView for="zope.i18n.interfaces.ITran slationService” name="index.html” • XML • YAML
  24. Complex formats application: myapp version: 1 runtime: python27 api_version: 1

    threadsafe: true handlers: - url: / script: home.app - url: /index\.html script: home.app - url: /stylesheets static_dir: stylesheets - url: /(.*\.(gif|png|jpg)) static_files: static/\1 upload: static/(.*\.(gif|png| • XML • YAML
  25. Turing complete formats DEBUG = True TEMPLATE_DEBUG = DEBUG DATABASES

    = { 'default': { 'ENGINE': 'django.db.backends.sqlite3', 'NAME': '/tmp/project.db', 'USER': '', 'PASSWORD': '', 'HOST': '', 'PORT': '', } } • Python
  26. Awkward formats Feature: Addition In order to avoid mistakes As

    a math slouch I want to be told the sum of two numbers Scenario: Add two numbers Given I have entered 50 into the calc And I have entered 70 into the calc When I press add Then the result should be 120 • Domain-Specific Languages • SQLite • Windows registry
  27. Awkward formats class davids_black_co_at { hosting::user { rztt: realname =>

    "Gerhard Schmitt", uid => 2001, admin => true; conny: realname => "Conny Schmitt", uid => 2002; oma: realname => "Oma Schmitt", uid => 2003; } # Install git.black.co.at include git::daemon • Domain-Specific Languages • SQLite • Windows registry
  28. Awkward formats # main.cf ... alias_maps = sqlite:/etc/postfix/sqlite- aliases.cf ...

    # sqlite config file for # local(8) aliases(5) lookups dbpath = /path/to/sqlite_database query = SELECT forw_addr FROM mxaliases WHERE alias='%s’ • Domain-Specific Languages • SQLite • Windows registry
  29. Awkward formats sqlite> .tables directory handler log proxy server statistic

    filter host mimetype route setting sqlite> select * from host; 1|1|0|localhost|localhost sqlite> select count(*) from mimetype; 850 sqlite> select * from server; • Domain-Specific Languages • SQLite • Windows registry
  30. Awkward formats sqlite> .tables directory handler log proxy server statistic

    filter host mimetype route setting sqlite> select * from host; 1|1|0|localhost|localhost sqlite> select count(*) from mimetype; 850 sqlite> select * from server; • Domain-Specific Languages • SQLite • Windows registry # get s list of the available servers to run m2sh servers -db tests/config.sqlite # see what hosts a server has m2sh hosts -db tests/config.sqlite -server test # find out if a server named 'test' is running m2sh running -db tests/config.sqlite -name test # start a server who's default host is 'localhost' m2sh start -db tests/config.sqlite -host localhost
  31. Awkward formats • Domain-Specific Languages • SQLite • Windows registry

  32. The worst format ever Your own format

  33. YOU WILL IMPOSE A LEARNING CURVE ON YOUR USERS

  34. YOUR DESIGN DECISIONS WILL BE UNINTUITIVE

  35. YOU WILL FAIL AT PARSING

  36. PEOPLE WILL START DEPENDING ON THE PARSER

  37. CONFIGURATION WRITTEN ONCE HAS TO BE SUPPORTED FOREVER

  38. Seeking balance one-size-fits-all vs. tweakable-beyond- recognition

  39. None
  40. ambv@arrakis:~ $ wget -O - "https://www.dropbox.com/download? plat=lnx.x86_64" | tar xzf

    - --2012-10-20 11:24:22-- https://www.dropbox.com/download? plat=lnx.x86_64 Resolving www.dropbox.com... 199.47.217.171, 199.47.216.170, 199.47.216.171, ... Connecting to www.dropbox.com|199.47.217.171|:443... connected. HTTP request sent, awaiting response... 302 FOUND Location: https://dl-web.dropbox.com/u/17/dropbox-lnx.x86_64- 1.4.17.tar.gz [following] --2012-10-20 11:24:23-- https://dl-web.dropbox.com/u/17/dropbox- lnx.x86_64-1.4.17.tar.gz Resolving dl-web.dropbox.com... 174.129.197.250, 174.129.199.91, 107.20.174.220, ... Connecting to dl-web.dropbox.com|174.129.197.250|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 19090076 (18M) [application/x-tar] Saving to: `STDOUT' 100% [=============================================================>] 19,090,076 1.23M/s in 17s
  41. ambv@arrakis:~ $ wget -O - "https://www.dropbox.com/download? plat=lnx.x86_64" | tar xzf

    - 100%[======>] 19,090,076 1.23M/s in 17s ambv@arrakis:~ $ ./.dropbox- dist/dropboxd This client is not linked to any account... Please visit https://www.dropbox.com/cli_link?
  42. COMPOSABLE READABLE EXCHANGEABLE DISCOVERABLE

  43. Excerpts from >>> import this Beautiful is better than ugly.

    Simple is better than complex. Flat is better than nested. Readability counts. There should be one (and preferably only one) obvious way to do it.
  44. None
  45. None
  46. Side note. Go to: http://make-everything-ok.com/

  47. CONFIG != DATA

  48. Hard coding • An anti-pattern • Like hardwiring circuits •

    Configuration embedded in source code
  49. Reverse hard coding • Source code embedded in configuration •

    Also an anti-pattern
  50. CONFIG != CODE

  51. None
  52. Practical configurability DJANGo: settings.py

  53. Django settings.py Django mixes different kinds of settings • Framework

    behaviour • Application behaviour • Deployment-specific configuration • databases • caches • logging • paths • ADMINS • MANAGERS • DEBUG • Sensitive data • passwords • SECURE_KEY • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
  54. Django settings.py # settings.py # ... TIME_ZONE = 'Europe/Zurich' LANGUAGE_CODE

    = 'en-us' SECRET_KEY = 'secret' # ... from settings_local import * • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
  55. Django settings.py # settings/__init__.py # ... TIME_ZONE = 'Europe/Zurich' LANGUAGE_CODE

    = 'en-us' SECRET_KEY = 'secret' # ... try: from .local import * except ImportError: pass • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
  56. Django settings.py Alternative approach • settings.py is never used •

    settings_base.py defines the base configuration • settings_prod.py, settings_dev.py, etc. import settings_base.py at the beginning • run with DJANGO_SETTINGS_MODULE = proj.settings_prod • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
  57. Django settings.py # settings.py import os.path import glob conffiles =

    glob.glob( os.path.join( os.path.dirname(__file__), 'settings', '*.conf’ ) ) conffiles.sort() for f in conffiles: execfile(os.path.abspath(f )) • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
  58. Django settings.py # settings.py import os.path import glob conffiles =

    glob.glob( os.path.join( os.path.dirname(__file__), 'settings', '*.conf’ ) ) conffiles.sort() for f in conffiles: execfile(os.path.abspath(f )) • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration $ ls settings/ 10-base.conf 20-engines.conf 30-site.conf ...
  59. Django settings.py from configurations import Settings class Base(Settings): TIME_ZONE =

    'Europe/Berlin' class Dev(Base): DEBUG = True TEMPLATE_DEBUG = DEBUG class Prod(Base): TIME_ZONE = 'America/New_York’ • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
  60. Django settings.py from configurations import Settings class FullPageCaching(object): USE_ETAGS =

    True class Base(Settings): TIME_ZONE = 'Europe/Berlin' class Dev(Base): DEBUG = True TEMPLATE_DEBUG = DEBUG class Prod(Base, FullPageCaching): • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
  61. Django settings.py export DJANGO_CONFIGURATION=Dev or python manage.py runserver --settings=mysite.settings --configuration=Dev

    • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
  62. Django settings.py [database] DATABASE_USER = db-user DATABASE_PASSWORD = db-secret DATABASE_HOST

    = 127.0.0.1 DATABASE_PORT = 3306 DATABASE_ENGINE = mysql DATABASE_NAME = example TESTSUITE_DATABASE_NAME = test_example [secrets] SECRET_KEY = 12345678... CSRF_MIDDLEWARE_SECRET = 12345678... [debug] • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
  63. Django settings.py # production environment prod.ini [database] DATABASE_PASSWORD = db-secret

    [secrets] SECRET_KEY = 12345678... CSRF_MIDDLEWARE_SECRET = 12345678... [debug] DEBUG = false TEMPLATE_DEBUG = false VIEW_TEST = false INTERNAL_IPS = 127.0.0.1 SKIP_CSRF_MIDDLEWARE = false • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
  64. Django settings.py >>> c = ConfigParser() >>> c.read([‘config-base.ini', ‘config- prod.ini'])

    You can do a hierarchy like: 1. site-packages/project/config- defaults.ini 2. /etc/project/config.ini 3. /etc/project.d/10-config.ini 4. /etc/project.d/20-config.ini 5. ~/.project/config.ini 6. ENVIRONMENT_VARIABLE • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
  65. Practical configurability mixing file-based configuration with command-line overrides

  66. Mixing file-based configuration with command-line overrides • Use configglue •

    Checkout out ConfArgParse • Whatever you do, name your command-line arguments on par with the file configuration settings
  67. configparser the real reason to switch to python 3

  68. An example file cfg = """ [DEFAULT] host = localhost

    port = 8080 [user] name = ambv email = lukasz@langa.pl """
  69. The old way import ConfigParser cp = ConfigParser.SafeConfigParser() from cStringIO

    import StringIO sio = StringIO(cfg) cp.readfp(sio)
  70. The old way import ConfigParser cp = ConfigParser.SafeConfigParser() from cStringIO

    import StringIO sio = StringIO(cfg) cp.readfp(sio)
  71. The old way import ConfigParser cp = ConfigParser.SafeConfigParser() from cStringIO

    import StringIO sio = StringIO(cfg) cp.readfp(sio)
  72. The new way from configparser import ConfigParser cp = ConfigParser()

    cp.read_string(cfg)
  73. The old way cp.readfp(sio) if cp.has_section('user'): print(cp.get('user', 'email')) cp.set('user', 'password',

    'secret') cp.set('user', 'jabber', '% (email)s') print(cp.get('user', 'jabber'))
  74. The old way cp.readfp(sio) if cp.has_section('user'): print(cp.get('user', 'email')) cp.set('user', 'password',

    'secret') cp.set('user', 'jabber', '% (email)s') print(cp.get('user', 'jabber')) This reminds me of: >>> import string >>> string.split('hakuna matata') ['hakuna', 'matata'] >>> {}.has_key('to your heart') False >>> bool('Yuck') True
  75. The old way if cp.has_section('user'): print(cp.get('user', 'email')) cp.set('user', 'password', 'secret')

    cp.set('user', 'jabber', '% (email)s') print(cp.get('user', 'jabber'))
  76. The new way if 'user' in cp: user = cp['user']

    print(user['email']) user['password'] = 'secret' user['jabber'] = '%(email)s' print(cp['user']['jabber'])
  77. The old way for option in cp.options('user'): print(option, end=", ")

    print()
  78. The new way for option in cp[‘user’]: print(option, end=", ")

    print()
  79. The old way if cp.has_option('user', 'host'): print(cp.get('user', 'host'))

  80. The new way user = cp[‘user’] if 'host' in user:

    print(user['host'])
  81. The new way print(user.get('host’, fallback=None))

  82. Configuration from a string cfg = """ [DEFAULT] host =

    localhost port = 8080 [user] name = ambv email = lukasz@langa.pl """
  83. Configuration from a dictionary cfg = dict( DEFAULT = dict(

    host = 'localhost', port = '8080’, ), USER = dict( name = 'ambv', email = 'lukasz@langa.pl’, ), )
  84. Configuration from a dictionary cp = ConfigParser() cp.read_dict(cfg) cp2 =

    ConfigParser() cp2.read_dict(cp) assert cp == cp2
  85. Creating sections from a dictionary cp['ui'] = { 'askusername': 'yes',

    'fallbackencoding': 'utf-8', 'ignore': '~/.hgignore', } print(cp['ui'].items())
  86. Creating sections from a dictionary cp['ui'] = { 'askusername': 'yes',

    'fallbackencoding': 'utf-8', 'ignore': '~/.hgignore', } print(cp['ui'].items()) The output: [('ignore', '~/.hgignore'), ('askusername', 'yes'), ('fallbackencoding', 'utf-8'), ('host', 'localhost'), ('port', '8080')]
  87. Buildout-inspired interpolation from configparser import ConfigParser,\ ExtendedInterpolation cp = ConfigParser(interpolation=

    ExtendedInterpolation()) cp.read_string(""" [general] project_dir = /opt/epic_project debug = False [cms] path = ${general:project_dir}/cms debug = ${general:debug}
  88. Buildout-inspired interpolation from configparser import ConfigParser,\ ExtendedInterpolation cp = ConfigParser(interpolation=

    ExtendedInterpolation()) cp.read_string(""" [general] project_dir = /opt/epic_project debug = False [cms] path = ${general:project_dir}/cms debug = ${general:debug} Transitive interpolation: >>> cp['cms-plugin'].items() [('path', '/opt/epic_project/cms/plugin’), ('debug', 'False')]
  89. Highly customizable!

  90. Highly customizable!

  91. Highly customizable!

  92. Highly customizable!

  93. Highly customizable!

  94. Highly customizable!

  95. Still stuck on Python 2.7?

  96. Still stuck on Python 2.6?

  97. Still stuck on Python 2.6? pip install configparser

  98. Questions? ambv at #python-dev (FreeNode IRC) @llanga on Twitter lukasz@langa.pl

    as a last resort