Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The classification problems: challenges and solutions

Marco Marongiu
February 01, 2014

The classification problems: challenges and solutions

I have 20 minutes to try to give you an idea of why classification in configuration management is critical, why it is difficult, and how you can make it sane.

Marco Marongiu

February 01, 2014
Tweet

More Decks by Marco Marongiu

Other Decks in Technology

Transcript

  1. The classification problem:
    challenges and solutions
    Marco Marongiu (@brontolinux)

    View full-size slide

  2. Acknowledgements
    All the images and logos used in this presentation are shown
    for didactic purposes only, in the belief that it constitutes a
    fair use of these resources. I don't claim any right on the
    images and products shown in this presentation.
    If you are the owner of any material used in this presentation
    and you believe that I am abusing it, please let me know and
    I'll promptly remove it from all my sources.

    View full-size slide

  3. Agenda

    Why classification is critical

    Why classification is difficult

    How we can make it sane

    Final take-aways

    View full-size slide

  4. Part 1:
    the problem

    View full-size slide

  5. !!!
    !!!
    !!!
    !!!
    !!!
    !!!
    !!!

    View full-size slide

  6. Configuration management
    «Configuration Management is the process of
    identifying and defining the items in the system,
    controlling the change of these items throughout
    their life-cycle, recording and reporting the status
    of items and change requests, and verifying the
    completeness and correctness of items»
    -- IEEE Glossary of Software Engineering Terminology (Standard
    729-1983)

    View full-size slide

  7. Configuration management
    «Configuration Management is the process of
    identifying and defining the items in the system,
    controlling the change of these items throughout
    their life-cycle, recording and reporting the status
    of items and change requests, and verifying the
    completeness and correctness of items»
    -- IEEE Glossary of Software Engineering Terminology (Standard
    729-1983)

    View full-size slide

  8. In configuration management
    you don't apply a
    configuration to
    a system, you
    rather apply it
    to classes
    of systems!

    View full-size slide

  9. Part 2:
    challenges

    View full-size slide

  10. Exceptions
    are the rule

    View full-size slide

  11. Generic settings, the same for all systems
    Amsterdam
    DNS server NTP server Syslog server SSH configs
    Oslo
    Other random exceptions
    Seattle Iceland

    View full-size slide

  12. Internal classification doesn't scale
    definitions
    explosion
    difficult
    reporting
    «The business should define what systems
    belong in which classes. The Cfengine
    administrator should build policy. Once
    the Cfengine administrator is left to
    manually defining classes within policy,
    you become a bottleneck»
    -- M.Svoboda, LinkedIn
    human
    bottleneck

    View full-size slide

  13. Part 3:
    Solutions

    View full-size slide

  14. Infrastructure complexity
    low complexity high complexity
    Technical skills
    well versed
    non technical
    Basic interface
    &
    basic backend
    Basic interface
    &
    sophisticated
    backend
    Sophisticated
    interface
    &
    sophisticated
    backend
    Sophisticated
    interface
    &
    basic backend

    View full-size slide

  15. The LinkedIn solution
    M.Svoboda - LinkedIn
    "Leveraging In-Memory Key Value Stores for Large-Scale Operations"

    View full-size slide

  16. our implementation: hENC
    power &
    simplicity
    config info
    in plain
    text files
    module
    protocol
    a simple
    Perl script

    View full-size slide

  17. CFEngine's module protocol
    +activated_class
    -cancelled_class
    =my_var=my value
    =@my_list={ 'list','of','4','values'}
    =my_array[element]=value
    @my_array[list]={ 'list','of','4','values'}

    View full-size slide

  18. #!/bin/sh
    /bin/egrep -h ^[=@+-] $* 2> /dev/null

    View full-size slide

  19. #!/usr/bin/perl
    use strict ;
    use warnings ;
    my %class ; # classes container
    my %variable ; # variables container
    # Silence errors (e.g.: missing files)
    close STDERR ;
    while (my $line = <>) {
    chomp $line ;
    my ($setting,$id) =
    ( $line =~ m{^\s*([=\@+-])(.+)\s*$} ) ;
    # line didn't match the module protocol
    next if not defined $setting ;
    # add a class
    if ($setting eq '+') {
    # $id is a class name, or should be.
    $class{$id} = 1 ;
    }
    # undefine a class
    if ($setting eq '-') {
    # $id is a class name, or should be.
    $class{$id} = -1 ;
    }
    # define a variable/list
    if ($setting eq '=' or $setting eq '@') {
    # $id is "variable = something", or should be
    my ($varname) = ( $id =~ m{^(.+?)=} ) ;
    $variable{$varname} = $line ;
    }
    # discard the rest
    }
    # print out classes
    foreach my $classname (keys %class) {
    print "+$classname\n" if $class{$classname} > 0 ;
    print "-$classname\n" if $class{$classname} < 0 ;
    }
    # print variable/list assignments, the last one wins
    foreach my $assignment (values %variable) {
    print "$assignment\n" ;
    }

    View full-size slide

  20. read
    general
    defaults
    read
    location
    defaults
    read
    environment
    defaults
    read
    node
    settings

    View full-size slide

  21. oslo_public::
    "enc_subdir"
    policy => "overridable",
    string => "$(enc_basedir)/pub" ;
    on_private_net_only::
    "enc_subdir"
    policy => "overridable",
    string => "$(enc_basedir)/priv" ;
    oslo::
    "henclist"
    policy => "overridable",
    slist => {
    "$(enc_basedir)/_default_",
    "$(enc_basedir)/_oslo_",
    "$(enc_subdir)/_oslo_",
    "$(enc_subdir)/$(sys.domain)/$(sys.fqhost)",
    } ;
    }environment-
    dependent
    settings
    } general defaults
    } location defaults
    } environ. defaults

    View full-size slide

  22. methods:
    "ENC"
    comment => "External node classification",
    usebundle => henc("site.henclist") ;
    vars:
    "motd_file"
    string => "$(henc.motd_file)",
    policy => "overridable" ;

    View full-size slide

  23. final take-aways
    smart
    classification
    is crucial...
    ...but doesn't
    need to be
    complicated...
    ...unless your
    infrastructure is.
    sometimes,
    even plain text
    is good enough!

    View full-size slide

  24. comments?
    @brontolinux
    [email protected]
    http://syslog.me
    http://no.linkedin.com/in/marcomarongiu/

    View full-size slide