Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Let them configure! by Łukasz Langa

Let them configure! by Łukasz Langa

PyCon 2013

March 17, 2013
Tweet

More Decks by PyCon 2013

Other Decks in Technology

Transcript

  1. a sermon on reusability
    Let them
    configure!

    View Slide

  2. Łukasz Langa
    ambv
    at #python-dev (FreeNode IRC)
    @llanga
    on Twitter
    [email protected]
    as a last resort

    View Slide

  3. View Slide

  4. View Slide

  5. View Slide

  6. Click icon to add picture
    Robert M.
    Pirsig

    „The solutions all
    are simple – after
    you have arrived at
    them.

    But they're simple
    only when you
    know already what
    they are.”

    View Slide

  7. four desirable
    characteristics of
    configuration

    View Slide

  8. COMPOSABLE

    View Slide

  9. READABLE

    View Slide

  10. EXCHANGEAB
    LE

    View Slide

  11. DISCOVERAB
    LE

    View Slide

  12. COMPOSABLE
    READABLE
    EXCHANGEABLE
    DISCOVERABLE

    View Slide

  13. configuration
    format whirlwind
    tour

    View Slide

  14. Easy-ish
    formats
    [user]
    email = [email protected]
    name = Łukasz Langa
    [push]
    default = current
    [color]
    branch = auto
    diff = auto
    interactive = auto
    status = auto
    [diff]
    external = git_vimdiff
    [pager]
    diff =
    [core]
    excludesfile = ~/.gitignore

    INI

    Apache

    nginx

    JSON

    View Slide

  15. Easy-ish
    formats
    Listen 80
    User apache
    Group apache
    DocumentRoot
    "/var/www/vhosts/root"

    Options FollowSymLinks
    AllowOverride None


    Options Indexes
    FollowSymLinks MultiViews
    ExecCGI
    AllowOverride All

    INI

    Apache

    nginx

    JSON

    View Slide

  16. Easy-ish
    formats
    Listen 80
    User apache
    Group apache
    DocumentRoot
    "/var/www/vhosts/root"

    Options FollowSymLinks
    AllowOverride None


    Options Indexes
    FollowSymLinks MultiViews
    ExecCGI
    AllowOverride All

    INI

    Apache

    nginx

    JSON What’s the difference:
    Options +Indexes Includes
    MultiViews
    Options Indexes Includes
    MultiViews

    View Slide

  17. Easy-ish
    formats
    server {
    listen 80;
    server_name d.com *.d.com;
    rewrite ^ http://www.d.com
    $request_uri? permanent;
    }
    server {
    listen 80 default;
    server_name www.d.com;
    index index.html;
    root /home/d.com;
    location /forum {
    rewrite forum(.*) http://
    forum.d.com$1

    INI

    Apache

    nginx

    JSON

    View Slide

  18. Easy-ish
    formats
    {
    "production":{
    "phpSettings":{
    "display_startup_errors":
    false,
    "display_errors": false
    },
    "includePaths":{
    "library":
    "APPLICATION_PATH/../library"
    },
    "bootstrap":{
    "path":
    "APPLICATION_PATH/Bootstrap.php
    ",
    "class": "Bootstrap"
    },

    INI

    Apache

    nginx

    JSON

    View Slide

  19. Easy-ish
    formats

    INI

    Apache

    nginx

    JSON
    WHAT
    ABOUT
    TOML?

    View Slide

  20. Easy-ish
    formats
    # This is a TOML document. Boom.
    title = "TOML Example"
    [owner]
    name = "Tom Preston-Werner"
    organization = "GitHub"
    bio = "GitHub Cofounder\n& CEO"
    dob = 1979-05-27T07:32:00Z
    [database]
    server = "192.168.1.1"
    ports = [ 8001, 8001, 8002 ]
    connection_max = 5000
    enabled = true
    [servers]
    # You can indent as you please.
    [servers.alpha]
    ip = "10.0.0.1" # inline comments
    [servers.beta]
    ip = "10.0.0.2"
    [clients]
    data = [ ["gamma", "delta"], [1, 2] ]

    INI

    Apache

    nginx

    JSON

    TOML?

    View Slide

  21. Easy-ish
    formats
    # This is a TOML document. Boom.
    title = "TOML Example"
    [owner]
    name = "Tom Preston-Werner"
    organization = "GitHub"
    bio = "GitHub Cofounder\n& CEO"
    dob = 1979-05-27T07:32:00Z
    [database]
    server = "192.168.1.1"
    ports = [ 8001, 8001, 8002 ]
    connection_max = 5000
    enabled = true
    [servers]
    # You can indent as you please.
    [servers.alpha]
    ip = "10.0.0.1" # inline comments
    [servers.beta]
    ip = "10.0.0.2"
    [clients]
    data = [ ["gamma", "delta"], [1, 2] ]

    INI

    Apache

    nginx

    JSON

    TOML?
    Strings are single-line values surrounded by
    double quotes encoded in UTF-8.
    vs.
    \xXX - byte (0x00-0xFF)

    View Slide

  22. Easy-ish
    formats
    # This is a TOML document. Boom.
    title = "TOML Example"
    [owner]
    name = "Tom Preston-Werner"
    organization = "GitHub"
    bio = "GitHub Cofounder\n& CEO"
    dob = 1979-05-27T07:32:00Z
    [database]
    server = "192.168.1.1"
    ports = [ 8001, 8001, 8002 ]
    connection_max = 5000
    enabled = true
    [servers]
    # You can indent as you please.
    [servers.alpha]
    ip = "10.0.0.1" # inline comments
    [servers.beta]
    ip = "10.0.0.2"
    [clients]
    data = [ ["gamma", "delta"], [1, 2] ]

    INI

    Apache

    nginx

    JSON

    TOML?

    View Slide

  23. Complex
    formats
    xmlns="http://namespaces.zope.o
    rg/zope"
    xmlns:zmi="http://namespaces.zo
    pe.org/zmi"
    xmlns:browser="http://namespace
    s.zope.org/browser"
    >
    for="zope.i18n.interfaces.ITran
    slationService”
    name="index.html”

    XML

    YAML

    View Slide

  24. Complex
    formats
    application: myapp
    version: 1
    runtime: python27
    api_version: 1
    threadsafe: true
    handlers:
    - url: /
    script: home.app
    - url: /index\.html
    script: home.app
    - url: /stylesheets
    static_dir: stylesheets
    - url: /(.*\.(gif|png|jpg))
    static_files: static/\1
    upload: static/(.*\.(gif|png|

    XML

    YAML

    View Slide

  25. Turing
    complete
    formats
    DEBUG = True
    TEMPLATE_DEBUG = DEBUG
    DATABASES = {
    'default': {
    'ENGINE':
    'django.db.backends.sqlite3',
    'NAME':
    '/tmp/project.db',
    'USER': '',
    'PASSWORD': '',
    'HOST': '',
    'PORT': '',
    }
    }

    Python

    View Slide

  26. Awkward
    formats
    Feature: Addition
    In order to avoid mistakes
    As a math slouch
    I want to be told the sum of two
    numbers
    Scenario: Add two numbers
    Given I have entered 50 into the
    calc
    And I have entered 70 into the calc
    When I press add
    Then the result should be 120

    Domain-Specific
    Languages

    SQLite

    Windows registry

    View Slide

  27. Awkward
    formats
    class davids_black_co_at {
    hosting::user {
    rztt: realname =>
    "Gerhard Schmitt",
    uid => 2001, admin =>
    true;
    conny: realname => "Conny
    Schmitt",
    uid => 2002;
    oma: realname => "Oma
    Schmitt",
    uid => 2003;
    }
    # Install git.black.co.at
    include git::daemon

    Domain-Specific
    Languages

    SQLite

    Windows registry

    View Slide

  28. Awkward
    formats
    # main.cf
    ...
    alias_maps =
    sqlite:/etc/postfix/sqlite-
    aliases.cf
    ...
    # sqlite config file for
    # local(8) aliases(5) lookups
    dbpath =
    /path/to/sqlite_database
    query = SELECT forw_addr FROM
    mxaliases WHERE alias='%s’

    Domain-Specific
    Languages

    SQLite

    Windows registry

    View Slide

  29. Awkward
    formats
    sqlite> .tables
    directory handler log
    proxy server statistic
    filter host mimetype
    route setting
    sqlite> select * from host;
    1|1|0|localhost|localhost
    sqlite> select count(*) from
    mimetype;
    850
    sqlite> select * from server;

    Domain-Specific
    Languages

    SQLite

    Windows registry

    View Slide

  30. Awkward
    formats
    sqlite> .tables
    directory handler log
    proxy server statistic
    filter host mimetype
    route setting
    sqlite> select * from host;
    1|1|0|localhost|localhost
    sqlite> select count(*) from
    mimetype;
    850
    sqlite> select * from server;

    Domain-Specific
    Languages

    SQLite

    Windows registry
    # get s list of the available servers to run
    m2sh servers -db tests/config.sqlite
    # see what hosts a server has
    m2sh hosts -db tests/config.sqlite -server test
    # find out if a server named 'test' is running
    m2sh running -db tests/config.sqlite -name test
    # start a server who's default host is 'localhost'
    m2sh start -db tests/config.sqlite -host localhost

    View Slide

  31. Awkward
    formats

    Domain-Specific
    Languages

    SQLite

    Windows registry

    View Slide

  32. The worst format ever
    Your own format

    View Slide

  33. YOU WILL IMPOSE
    A LEARNING
    CURVE ON YOUR
    USERS

    View Slide

  34. YOUR DESIGN
    DECISIONS WILL
    BE UNINTUITIVE

    View Slide

  35. YOU WILL FAIL AT
    PARSING

    View Slide

  36. PEOPLE WILL
    START
    DEPENDING ON
    THE PARSER

    View Slide

  37. CONFIGURATION
    WRITTEN ONCE
    HAS TO BE
    SUPPORTED
    FOREVER

    View Slide

  38. Seeking balance
    one-size-fits-all vs.
    tweakable-beyond-
    recognition

    View Slide

  39. View Slide

  40. [email protected]:~ $ wget -O - "https://www.dropbox.com/download?
    plat=lnx.x86_64" | tar xzf -
    --2012-10-20 11:24:22-- https://www.dropbox.com/download?
    plat=lnx.x86_64
    Resolving www.dropbox.com... 199.47.217.171, 199.47.216.170,
    199.47.216.171, ...
    Connecting to www.dropbox.com|199.47.217.171|:443... connected.
    HTTP request sent, awaiting response... 302 FOUND
    Location: https://dl-web.dropbox.com/u/17/dropbox-lnx.x86_64-
    1.4.17.tar.gz [following]
    --2012-10-20 11:24:23-- https://dl-web.dropbox.com/u/17/dropbox-
    lnx.x86_64-1.4.17.tar.gz
    Resolving dl-web.dropbox.com... 174.129.197.250, 174.129.199.91,
    107.20.174.220, ...
    Connecting to dl-web.dropbox.com|174.129.197.250|:443...
    connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 19090076 (18M) [application/x-tar]
    Saving to: `STDOUT'
    100%
    [=============================================================>]
    19,090,076 1.23M/s in 17s

    View Slide

  41. [email protected]:~ $ wget -O -
    "https://www.dropbox.com/download?
    plat=lnx.x86_64" | tar xzf -
    100%[======>] 19,090,076 1.23M/s in
    17s
    [email protected]:~ $ ./.dropbox-
    dist/dropboxd
    This client is not linked to any
    account...
    Please visit
    https://www.dropbox.com/cli_link?

    View Slide

  42. COMPOSABLE
    READABLE
    EXCHANGEABLE
    DISCOVERABLE

    View Slide

  43. Excerpts from >>> import this
    Beautiful is better than ugly.
    Simple is better than complex.
    Flat is better than nested.
    Readability counts.
    There should be one (and
    preferably only one) obvious
    way to do it.

    View Slide

  44. View Slide

  45. View Slide

  46. Side note. Go to: http://make-everything-ok.com/

    View Slide

  47. CONFIG !=
    DATA

    View Slide

  48. Hard coding

    An anti-pattern

    Like hardwiring circuits

    Configuration embedded in
    source code

    View Slide

  49. Reverse hard
    coding

    Source code embedded in
    configuration

    Also an anti-pattern

    View Slide

  50. CONFIG !=
    CODE

    View Slide

  51. View Slide

  52. Practical configurability
    DJANGo: settings.py

    View Slide

  53. Django
    settings.py
    Django mixes different
    kinds of settings

    Framework behaviour

    Application behaviour

    Deployment-specific configuration
    • databases
    • caches
    • logging
    • paths
    • ADMINS
    • MANAGERS
    • DEBUG

    Sensitive data
    • passwords
    • SECURE_KEY

    The problem

    from settings import
    *

    Ordered
    incremental execfile

    django-
    configurations

    INI based
    configuration

    View Slide

  54. Django
    settings.py
    # settings.py
    # ...
    TIME_ZONE = 'Europe/Zurich'
    LANGUAGE_CODE = 'en-us'
    SECRET_KEY = 'secret'
    # ...
    from settings_local import *

    The problem

    from settings
    import *

    Ordered
    incremental execfile

    django-
    configurations

    INI based
    configuration

    View Slide

  55. Django
    settings.py
    # settings/__init__.py
    # ...
    TIME_ZONE = 'Europe/Zurich'
    LANGUAGE_CODE = 'en-us'
    SECRET_KEY = 'secret'
    # ...
    try:
    from .local import *
    except ImportError:
    pass

    The problem

    from settings
    import *

    Ordered
    incremental execfile

    django-
    configurations

    INI based
    configuration

    View Slide

  56. Django
    settings.py
    Alternative
    approach

    settings.py is never used

    settings_base.py defines
    the base configuration

    settings_prod.py,
    settings_dev.py, etc. import
    settings_base.py at the
    beginning

    run with
    DJANGO_SETTINGS_MODULE
    = proj.settings_prod

    The problem

    from settings
    import *

    Ordered
    incremental execfile

    django-
    configurations

    INI based
    configuration

    View Slide

  57. Django
    settings.py
    # settings.py
    import os.path
    import glob
    conffiles = glob.glob(
    os.path.join(
    os.path.dirname(__file__),
    'settings', '*.conf’
    )
    )
    conffiles.sort()
    for f in conffiles:
    execfile(os.path.abspath(f
    ))

    The problem

    from settings import
    *

    Ordered
    incremental
    execfile

    django-
    configurations

    INI based
    configuration

    View Slide

  58. Django
    settings.py
    # settings.py
    import os.path
    import glob
    conffiles = glob.glob(
    os.path.join(
    os.path.dirname(__file__),
    'settings', '*.conf’
    )
    )
    conffiles.sort()
    for f in conffiles:
    execfile(os.path.abspath(f
    ))

    The problem

    from settings import
    *

    Ordered
    incremental
    execfile

    django-
    configurations

    INI based
    configuration
    $ ls settings/
    10-base.conf
    20-engines.conf
    30-site.conf
    ...

    View Slide

  59. Django
    settings.py
    from configurations import
    Settings
    class Base(Settings):
    TIME_ZONE =
    'Europe/Berlin'
    class Dev(Base):
    DEBUG = True
    TEMPLATE_DEBUG = DEBUG
    class Prod(Base):
    TIME_ZONE =
    'America/New_York’

    The problem

    from settings import
    *

    Ordered
    incremental execfile

    django-
    configurations

    INI based
    configuration

    View Slide

  60. Django
    settings.py
    from configurations import
    Settings
    class FullPageCaching(object):
    USE_ETAGS = True
    class Base(Settings):
    TIME_ZONE =
    'Europe/Berlin'
    class Dev(Base):
    DEBUG = True
    TEMPLATE_DEBUG = DEBUG
    class Prod(Base,
    FullPageCaching):

    The problem

    from settings import
    *

    Ordered
    incremental execfile

    django-
    configurations

    INI based
    configuration

    View Slide

  61. Django
    settings.py
    export
    DJANGO_CONFIGURATION=Dev
    or
    python manage.py runserver
    --settings=mysite.settings
    --configuration=Dev

    The problem

    from settings import
    *

    Ordered
    incremental execfile

    django-
    configurations

    INI based
    configuration

    View Slide

  62. Django
    settings.py
    [database]
    DATABASE_USER = db-user
    DATABASE_PASSWORD = db-secret
    DATABASE_HOST = 127.0.0.1
    DATABASE_PORT = 3306
    DATABASE_ENGINE = mysql
    DATABASE_NAME = example
    TESTSUITE_DATABASE_NAME =
    test_example
    [secrets]
    SECRET_KEY = 12345678...
    CSRF_MIDDLEWARE_SECRET =
    12345678...
    [debug]

    The problem

    from settings import
    *

    Ordered
    incremental execfile

    django-
    configurations

    INI based
    configuration

    View Slide

  63. Django
    settings.py
    # production environment prod.ini
    [database]
    DATABASE_PASSWORD = db-secret
    [secrets]
    SECRET_KEY = 12345678...
    CSRF_MIDDLEWARE_SECRET =
    12345678...
    [debug]
    DEBUG = false
    TEMPLATE_DEBUG = false
    VIEW_TEST = false
    INTERNAL_IPS = 127.0.0.1
    SKIP_CSRF_MIDDLEWARE = false

    The problem

    from settings import
    *

    Ordered
    incremental execfile

    django-
    configurations

    INI based
    configuration

    View Slide

  64. Django
    settings.py
    >>> c = ConfigParser()
    >>> c.read([‘config-base.ini', ‘config-
    prod.ini'])
    You can do a hierarchy like:
    1.
    site-packages/project/config-
    defaults.ini
    2.
    /etc/project/config.ini
    3.
    /etc/project.d/10-config.ini
    4.
    /etc/project.d/20-config.ini
    5.
    ~/.project/config.ini
    6.
    ENVIRONMENT_VARIABLE

    The problem

    from settings import
    *

    Ordered
    incremental execfile

    django-
    configurations

    INI based
    configuration

    View Slide

  65. Practical configurability
    mixing file-based
    configuration with
    command-line overrides

    View Slide

  66. Mixing file-based configuration with
    command-line overrides

    Use configglue

    Checkout out ConfArgParse

    Whatever you do, name your command-line
    arguments on par with the file configuration
    settings

    View Slide

  67. configparser
    the real reason to
    switch to python 3

    View Slide

  68. An example file
    cfg = """
    [DEFAULT]
    host = localhost
    port = 8080
    [user]
    name = ambv
    email = [email protected]
    """

    View Slide

  69. The old way
    import ConfigParser
    cp = ConfigParser.SafeConfigParser()
    from cStringIO import StringIO
    sio = StringIO(cfg)
    cp.readfp(sio)

    View Slide

  70. The old way
    import ConfigParser
    cp = ConfigParser.SafeConfigParser()
    from cStringIO import StringIO
    sio = StringIO(cfg)
    cp.readfp(sio)

    View Slide

  71. The old way
    import ConfigParser
    cp = ConfigParser.SafeConfigParser()
    from cStringIO import StringIO
    sio = StringIO(cfg)
    cp.readfp(sio)

    View Slide

  72. The new way
    from configparser import ConfigParser
    cp = ConfigParser()
    cp.read_string(cfg)

    View Slide

  73. The old way
    cp.readfp(sio)
    if cp.has_section('user'):
    print(cp.get('user', 'email'))
    cp.set('user', 'password',
    'secret')
    cp.set('user', 'jabber', '%
    (email)s')
    print(cp.get('user', 'jabber'))

    View Slide

  74. The old way
    cp.readfp(sio)
    if cp.has_section('user'):
    print(cp.get('user', 'email'))
    cp.set('user', 'password',
    'secret')
    cp.set('user', 'jabber', '%
    (email)s')
    print(cp.get('user', 'jabber'))
    This reminds me of:
    >>> import string
    >>> string.split('hakuna matata')
    ['hakuna', 'matata']
    >>> {}.has_key('to your heart')
    False
    >>> bool('Yuck')
    True

    View Slide

  75. The old way
    if cp.has_section('user'):
    print(cp.get('user', 'email'))
    cp.set('user', 'password',
    'secret')
    cp.set('user', 'jabber', '%
    (email)s')
    print(cp.get('user', 'jabber'))

    View Slide

  76. The new way
    if 'user' in cp:
    user = cp['user']
    print(user['email'])
    user['password'] = 'secret'
    user['jabber'] = '%(email)s'
    print(cp['user']['jabber'])

    View Slide

  77. The old way
    for option in cp.options('user'):
    print(option, end=", ")
    print()

    View Slide

  78. The new way
    for option in cp[‘user’]:
    print(option, end=", ")
    print()

    View Slide

  79. The old way
    if cp.has_option('user', 'host'):
    print(cp.get('user', 'host'))

    View Slide

  80. The new way
    user = cp[‘user’]
    if 'host' in user:
    print(user['host'])

    View Slide

  81. The new way
    print(user.get('host’, fallback=None))

    View Slide

  82. Configuration from a string
    cfg = """
    [DEFAULT]
    host = localhost
    port = 8080
    [user]
    name = ambv
    email = [email protected]
    """

    View Slide

  83. Configuration from a dictionary
    cfg = dict(
    DEFAULT = dict(
    host = 'localhost',
    port = '8080’,
    ),
    USER = dict(
    name = 'ambv',
    email = '[email protected]’,
    ),
    )

    View Slide

  84. Configuration from a dictionary
    cp = ConfigParser()
    cp.read_dict(cfg)
    cp2 = ConfigParser()
    cp2.read_dict(cp)
    assert cp == cp2

    View Slide

  85. Creating sections from a dictionary
    cp['ui'] = {
    'askusername': 'yes',
    'fallbackencoding': 'utf-8',
    'ignore': '~/.hgignore',
    }
    print(cp['ui'].items())

    View Slide

  86. Creating sections from a dictionary
    cp['ui'] = {
    'askusername': 'yes',
    'fallbackencoding': 'utf-8',
    'ignore': '~/.hgignore',
    }
    print(cp['ui'].items())
    The output:
    [('ignore', '~/.hgignore'),
    ('askusername', 'yes'),
    ('fallbackencoding', 'utf-8'),
    ('host', 'localhost'),
    ('port', '8080')]

    View Slide

  87. Buildout-inspired interpolation
    from configparser import ConfigParser,\
    ExtendedInterpolation
    cp = ConfigParser(interpolation=
    ExtendedInterpolation())
    cp.read_string("""
    [general]
    project_dir = /opt/epic_project
    debug = False
    [cms]
    path = ${general:project_dir}/cms
    debug = ${general:debug}

    View Slide

  88. Buildout-inspired interpolation
    from configparser import ConfigParser,\
    ExtendedInterpolation
    cp = ConfigParser(interpolation=
    ExtendedInterpolation())
    cp.read_string("""
    [general]
    project_dir = /opt/epic_project
    debug = False
    [cms]
    path = ${general:project_dir}/cms
    debug = ${general:debug}
    Transitive interpolation:
    >>> cp['cms-plugin'].items()
    [('path', '/opt/epic_project/cms/plugin’),
    ('debug', 'False')]

    View Slide

  89. Highly
    customizable!

    View Slide

  90. Highly
    customizable!

    View Slide

  91. Highly
    customizable!

    View Slide

  92. Highly
    customizable!

    View Slide

  93. Highly
    customizable!

    View Slide

  94. Highly
    customizable!

    View Slide

  95. Still stuck on Python 2.7?

    View Slide

  96. Still stuck on Python 2.6?

    View Slide

  97. Still stuck on Python 2.6?
    pip install
    configparser

    View Slide

  98. Questions?
    ambv
    at #python-dev (FreeNode IRC)
    @llanga
    on Twitter
    [email protected]
    as a last resort

    View Slide