Let them configure! by Łukasz Langa

Transcript

1. a sermon on reusability Let them configure!
2. Łukasz Langa ambv at #python-dev (FreeNode IRC) @llanga on Twitter lukasz@langa.pl as a last resort
6. Click icon to add picture Robert M. Pirsig • „The solutions all are simple – after you have arrived at them. • But they're simple only when you know already what they are.”
7. four desirable characteristics of configuration
8. COMPOSABLE
9. READABLE
10. EXCHANGEAB LE
11. DISCOVERAB LE
12. COMPOSABLE READABLE EXCHANGEABLE DISCOVERABLE
13. configuration format whirlwind tour
14. Easy-ish formats [user] email = lukasz@langa.pl name = Łukasz Langa [push] default = current [color] branch = auto diff = auto interactive = auto status = auto [diff] external = git_vimdiff [pager] diff = [core] excludesfile = ~/.gitignore • INI • Apache • nginx • JSON
15. Easy-ish formats Listen 80 User apache Group apache DocumentRoot "/var/www/vhosts/root" <Directory /> Options FollowSymLinks AllowOverride None </Directory> <Directory "/var/www/vhosts"> Options Indexes FollowSymLinks MultiViews ExecCGI AllowOverride All • INI • Apache • nginx • JSON
16. Easy-ish formats Listen 80 User apache Group apache DocumentRoot "/var/www/vhosts/root" <Directory /> Options FollowSymLinks AllowOverride None </Directory> <Directory "/var/www/vhosts"> Options Indexes FollowSymLinks MultiViews ExecCGI AllowOverride All • INI • Apache • nginx • JSON What’s the difference: Options +Indexes Includes MultiViews Options Indexes Includes MultiViews
17. Easy-ish formats server { listen 80; server_name d.com *.d.com; rewrite ^ http://www.d.com $request_uri? permanent; } server { listen 80 default; server_name www.d.com; index index.html; root /home/d.com; location /forum { rewrite forum(.*) http:// forum.d.com$1 • INI • Apache • nginx • JSON
18. Easy-ish formats { "production":{ "phpSettings":{ "display_startup_errors": false, "display_errors": false }, "includePaths":{ "library": "APPLICATION_PATH/../library" }, "bootstrap":{ "path": "APPLICATION_PATH/Bootstrap.php ", "class": "Bootstrap" }, • INI • Apache • nginx • JSON
19. Easy-ish formats • INI • Apache • nginx • JSON WHAT ABOUT TOML?
20. Easy-ish formats # This is a TOML document. Boom. title = "TOML Example" [owner] name = "Tom Preston-Werner" organization = "GitHub" bio = "GitHub Cofounder\n& CEO" dob = 1979-05-27T07:32:00Z [database] server = "192.168.1.1" ports = [ 8001, 8001, 8002 ] connection_max = 5000 enabled = true [servers] # You can indent as you please. [servers.alpha] ip = "10.0.0.1" # inline comments [servers.beta] ip = "10.0.0.2" [clients] data = [ ["gamma", "delta"], [1, 2] ] • INI • Apache • nginx • JSON • TOML?
21. Easy-ish formats # This is a TOML document. Boom. title = "TOML Example" [owner] name = "Tom Preston-Werner" organization = "GitHub" bio = "GitHub Cofounder\n& CEO" dob = 1979-05-27T07:32:00Z [database] server = "192.168.1.1" ports = [ 8001, 8001, 8002 ] connection_max = 5000 enabled = true [servers] # You can indent as you please. [servers.alpha] ip = "10.0.0.1" # inline comments [servers.beta] ip = "10.0.0.2" [clients] data = [ ["gamma", "delta"], [1, 2] ] • INI • Apache • nginx • JSON • TOML? Strings are single-line values surrounded by double quotes encoded in UTF-8. vs. \xXX - byte (0x00-0xFF)
22. Easy-ish formats # This is a TOML document. Boom. title = "TOML Example" [owner] name = "Tom Preston-Werner" organization = "GitHub" bio = "GitHub Cofounder\n& CEO" dob = 1979-05-27T07:32:00Z [database] server = "192.168.1.1" ports = [ 8001, 8001, 8002 ] connection_max = 5000 enabled = true [servers] # You can indent as you please. [servers.alpha] ip = "10.0.0.1" # inline comments [servers.beta] ip = "10.0.0.2" [clients] data = [ ["gamma", "delta"], [1, 2] ] • INI • Apache • nginx • JSON • TOML?
23. Complex formats <configure xmlns="http://namespaces.zope.o rg/zope" xmlns:zmi="http://namespaces.zo pe.org/zmi" xmlns:browser="http://namespace s.zope.org/browser" > <browser:defaultView for="zope.i18n.interfaces.ITran slationService” name="index.html” • XML • YAML
24. Complex formats application: myapp version: 1 runtime: python27 api_version: 1 threadsafe: true handlers: - url: / script: home.app - url: /index\.html script: home.app - url: /stylesheets static_dir: stylesheets - url: /(.*\.(gif|png|jpg)) static_files: static/\1 upload: static/(.*\.(gif|png| • XML • YAML
25. Turing complete formats DEBUG = True TEMPLATE_DEBUG = DEBUG DATABASES = { 'default': { 'ENGINE': 'django.db.backends.sqlite3', 'NAME': '/tmp/project.db', 'USER': '', 'PASSWORD': '', 'HOST': '', 'PORT': '', } } • Python
26. Awkward formats Feature: Addition In order to avoid mistakes As a math slouch I want to be told the sum of two numbers Scenario: Add two numbers Given I have entered 50 into the calc And I have entered 70 into the calc When I press add Then the result should be 120 • Domain-Specific Languages • SQLite • Windows registry
27. Awkward formats class davids_black_co_at { hosting::user { rztt: realname => "Gerhard Schmitt", uid => 2001, admin => true; conny: realname => "Conny Schmitt", uid => 2002; oma: realname => "Oma Schmitt", uid => 2003; } # Install git.black.co.at include git::daemon • Domain-Specific Languages • SQLite • Windows registry
28. Awkward formats # main.cf ... alias_maps = sqlite:/etc/postfix/sqlite- aliases.cf ... # sqlite config file for # local(8) aliases(5) lookups dbpath = /path/to/sqlite_database query = SELECT forw_addr FROM mxaliases WHERE alias='%s’ • Domain-Specific Languages • SQLite • Windows registry
29. Awkward formats sqlite> .tables directory handler log proxy server statistic filter host mimetype route setting sqlite> select * from host; 1|1|0|localhost|localhost sqlite> select count(*) from mimetype; 850 sqlite> select * from server; • Domain-Specific Languages • SQLite • Windows registry
30. Awkward formats sqlite> .tables directory handler log proxy server statistic filter host mimetype route setting sqlite> select * from host; 1|1|0|localhost|localhost sqlite> select count(*) from mimetype; 850 sqlite> select * from server; • Domain-Specific Languages • SQLite • Windows registry # get s list of the available servers to run m2sh servers -db tests/config.sqlite # see what hosts a server has m2sh hosts -db tests/config.sqlite -server test # find out if a server named 'test' is running m2sh running -db tests/config.sqlite -name test # start a server who's default host is 'localhost' m2sh start -db tests/config.sqlite -host localhost
31. Awkward formats • Domain-Specific Languages • SQLite • Windows registry
32. The worst format ever Your own format
33. YOU WILL IMPOSE A LEARNING CURVE ON YOUR USERS
34. YOUR DESIGN DECISIONS WILL BE UNINTUITIVE
35. YOU WILL FAIL AT PARSING
36. PEOPLE WILL START DEPENDING ON THE PARSER
37. CONFIGURATION WRITTEN ONCE HAS TO BE SUPPORTED FOREVER
38. Seeking balance one-size-fits-all vs. tweakable-beyond- recognition
40. ambv@arrakis:~ $ wget -O - "https://www.dropbox.com/download? plat=lnx.x86_64" | tar xzf - --2012-10-20 11:24:22-- https://www.dropbox.com/download? plat=lnx.x86_64 Resolving www.dropbox.com... 199.47.217.171, 199.47.216.170, 199.47.216.171, ... Connecting to www.dropbox.com|199.47.217.171|:443... connected. HTTP request sent, awaiting response... 302 FOUND Location: https://dl-web.dropbox.com/u/17/dropbox-lnx.x86_64- 1.4.17.tar.gz [following] --2012-10-20 11:24:23-- https://dl-web.dropbox.com/u/17/dropbox- lnx.x86_64-1.4.17.tar.gz Resolving dl-web.dropbox.com... 174.129.197.250, 174.129.199.91, 107.20.174.220, ... Connecting to dl-web.dropbox.com|174.129.197.250|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 19090076 (18M) [application/x-tar] Saving to: `STDOUT' 100% [=============================================================>] 19,090,076 1.23M/s in 17s
41. ambv@arrakis:~ $ wget -O - "https://www.dropbox.com/download? plat=lnx.x86_64" | tar xzf - 100%[======>] 19,090,076 1.23M/s in 17s ambv@arrakis:~ $ ./.dropbox- dist/dropboxd This client is not linked to any account... Please visit https://www.dropbox.com/cli_link?
42. COMPOSABLE READABLE EXCHANGEABLE DISCOVERABLE
43. Excerpts from >>> import this Beautiful is better than ugly. Simple is better than complex. Flat is better than nested. Readability counts. There should be one (and preferably only one) obvious way to do it.
46. Side note. Go to: http://make-everything-ok.com/
47. CONFIG != DATA
48. Hard coding • An anti-pattern • Like hardwiring circuits • Configuration embedded in source code
49. Reverse hard coding • Source code embedded in configuration • Also an anti-pattern
50. CONFIG != CODE
52. Practical configurability DJANGo: settings.py
53. Django settings.py Django mixes different kinds of settings • Framework behaviour • Application behaviour • Deployment-specific configuration • databases • caches • logging • paths • ADMINS • MANAGERS • DEBUG • Sensitive data • passwords • SECURE_KEY • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
54. Django settings.py # settings.py # ... TIME_ZONE = 'Europe/Zurich' LANGUAGE_CODE = 'en-us' SECRET_KEY = 'secret' # ... from settings_local import * • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
55. Django settings.py # settings/__init__.py # ... TIME_ZONE = 'Europe/Zurich' LANGUAGE_CODE = 'en-us' SECRET_KEY = 'secret' # ... try: from .local import * except ImportError: pass • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
56. Django settings.py Alternative approach • settings.py is never used • settings_base.py defines the base configuration • settings_prod.py, settings_dev.py, etc. import settings_base.py at the beginning • run with DJANGO_SETTINGS_MODULE = proj.settings_prod • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
57. Django settings.py # settings.py import os.path import glob conffiles = glob.glob( os.path.join( os.path.dirname(__file__), 'settings', '*.conf’ ) ) conffiles.sort() for f in conffiles: execfile(os.path.abspath(f )) • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
58. Django settings.py # settings.py import os.path import glob conffiles = glob.glob( os.path.join( os.path.dirname(__file__), 'settings', '*.conf’ ) ) conffiles.sort() for f in conffiles: execfile(os.path.abspath(f )) • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration $ ls settings/ 10-base.conf 20-engines.conf 30-site.conf ...
59. Django settings.py from configurations import Settings class Base(Settings): TIME_ZONE = 'Europe/Berlin' class Dev(Base): DEBUG = True TEMPLATE_DEBUG = DEBUG class Prod(Base): TIME_ZONE = 'America/New_York’ • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
60. Django settings.py from configurations import Settings class FullPageCaching(object): USE_ETAGS = True class Base(Settings): TIME_ZONE = 'Europe/Berlin' class Dev(Base): DEBUG = True TEMPLATE_DEBUG = DEBUG class Prod(Base, FullPageCaching): • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
61. Django settings.py export DJANGO_CONFIGURATION=Dev or python manage.py runserver --settings=mysite.settings --configuration=Dev • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
62. Django settings.py [database] DATABASE_USER = db-user DATABASE_PASSWORD = db-secret DATABASE_HOST = 127.0.0.1 DATABASE_PORT = 3306 DATABASE_ENGINE = mysql DATABASE_NAME = example TESTSUITE_DATABASE_NAME = test_example [secrets] SECRET_KEY = 12345678... CSRF_MIDDLEWARE_SECRET = 12345678... [debug] • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
63. Django settings.py # production environment prod.ini [database] DATABASE_PASSWORD = db-secret [secrets] SECRET_KEY = 12345678... CSRF_MIDDLEWARE_SECRET = 12345678... [debug] DEBUG = false TEMPLATE_DEBUG = false VIEW_TEST = false INTERNAL_IPS = 127.0.0.1 SKIP_CSRF_MIDDLEWARE = false • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
64. Django settings.py >>> c = ConfigParser() >>> c.read([‘config-base.ini', ‘config- prod.ini']) You can do a hierarchy like: 1. site-packages/project/config- defaults.ini 2. /etc/project/config.ini 3. /etc/project.d/10-config.ini 4. /etc/project.d/20-config.ini 5. ~/.project/config.ini 6. ENVIRONMENT_VARIABLE • The problem • from settings import * • Ordered incremental execfile • django- configurations • INI based configuration
65. Practical configurability mixing file-based configuration with command-line overrides
66. Mixing file-based configuration with command-line overrides • Use configglue • Checkout out ConfArgParse • Whatever you do, name your command-line arguments on par with the file configuration settings
67. configparser the real reason to switch to python 3
68. An example file cfg = """ [DEFAULT] host = localhost port = 8080 [user] name = ambv email = lukasz@langa.pl """
69. The old way import ConfigParser cp = ConfigParser.SafeConfigParser() from cStringIO import StringIO sio = StringIO(cfg) cp.readfp(sio)
70. The old way import ConfigParser cp = ConfigParser.SafeConfigParser() from cStringIO import StringIO sio = StringIO(cfg) cp.readfp(sio)
71. The old way import ConfigParser cp = ConfigParser.SafeConfigParser() from cStringIO import StringIO sio = StringIO(cfg) cp.readfp(sio)
72. The new way from configparser import ConfigParser cp = ConfigParser() cp.read_string(cfg)
73. The old way cp.readfp(sio) if cp.has_section('user'): print(cp.get('user', 'email')) cp.set('user', 'password', 'secret') cp.set('user', 'jabber', '% (email)s') print(cp.get('user', 'jabber'))
74. The old way cp.readfp(sio) if cp.has_section('user'): print(cp.get('user', 'email')) cp.set('user', 'password', 'secret') cp.set('user', 'jabber', '% (email)s') print(cp.get('user', 'jabber')) This reminds me of: >>> import string >>> string.split('hakuna matata') ['hakuna', 'matata'] >>> {}.has_key('to your heart') False >>> bool('Yuck') True
75. The old way if cp.has_section('user'): print(cp.get('user', 'email')) cp.set('user', 'password', 'secret') cp.set('user', 'jabber', '% (email)s') print(cp.get('user', 'jabber'))
76. The new way if 'user' in cp: user = cp['user'] print(user['email']) user['password'] = 'secret' user['jabber'] = '%(email)s' print(cp['user']['jabber'])
77. The old way for option in cp.options('user'): print(option, end=", ") print()
78. The new way for option in cp[‘user’]: print(option, end=", ") print()
79. The old way if cp.has_option('user', 'host'): print(cp.get('user', 'host'))
80. The new way user = cp[‘user’] if 'host' in user: print(user['host'])
81. The new way print(user.get('host’, fallback=None))
82. Configuration from a string cfg = """ [DEFAULT] host = localhost port = 8080 [user] name = ambv email = lukasz@langa.pl """
83. Configuration from a dictionary cfg = dict( DEFAULT = dict( host = 'localhost', port = '8080’, ), USER = dict( name = 'ambv', email = 'lukasz@langa.pl’, ), )
84. Configuration from a dictionary cp = ConfigParser() cp.read_dict(cfg) cp2 = ConfigParser() cp2.read_dict(cp) assert cp == cp2
85. Creating sections from a dictionary cp['ui'] = { 'askusername': 'yes', 'fallbackencoding': 'utf-8', 'ignore': '~/.hgignore', } print(cp['ui'].items())
86. Creating sections from a dictionary cp['ui'] = { 'askusername': 'yes', 'fallbackencoding': 'utf-8', 'ignore': '~/.hgignore', } print(cp['ui'].items()) The output: [('ignore', '~/.hgignore'), ('askusername', 'yes'), ('fallbackencoding', 'utf-8'), ('host', 'localhost'), ('port', '8080')]
87. Buildout-inspired interpolation from configparser import ConfigParser,\ ExtendedInterpolation cp = ConfigParser(interpolation= ExtendedInterpolation()) cp.read_string(""" [general] project_dir = /opt/epic_project debug = False [cms] path = ${general:project_dir}/cms debug = ${general:debug}
88. Buildout-inspired interpolation from configparser import ConfigParser,\ ExtendedInterpolation cp = ConfigParser(interpolation= ExtendedInterpolation()) cp.read_string(""" [general] project_dir = /opt/epic_project debug = False [cms] path = ${general:project_dir}/cms debug = ${general:debug} Transitive interpolation: >>> cp['cms-plugin'].items() [('path', '/opt/epic_project/cms/plugin’), ('debug', 'False')]
89. Highly customizable!
90. Highly customizable!
91. Highly customizable!
92. Highly customizable!
93. Highly customizable!
94. Highly customizable!
95. Still stuck on Python 2.7?
96. Still stuck on Python 2.6?
97. Still stuck on Python 2.6? pip install configparser
98. Questions? ambv at #python-dev (FreeNode IRC) @llanga on Twitter lukasz@langa.pl as a last resort