External node classification, the CFEngine way

I’ll try to give you an idea of why classification in configuration management is critical, why it is difficult, and how you can make it sane. It’s the same seminar as FOSDEM; however, I’ll be able to show some examples of what makes classification difficult when you try to do it only “internally” with your CM tool of choice, and I’ll be able to show analogies and differences between Puppet and CFEngine when it comes to external node classification.

Transcript

1. External node classification,
   the CFEngine way
   The classification problem:
   challenges and solutions
   Marco Marongiu (@brontolinux)
   

   View Slide

2. Acknowledgements
   All the images and logos used in this presentation are shown
   for didactic purposes only, in the belief that it constitutes a
   fair use of these resources. I don't claim any right on the
   images and products shown in this presentation.
   If you are the owner of any material used in this presentation
   and you believe that I am abusing it, please let me know and
   I'll promptly remove it from all my sources.
   

   View Slide

3. Agenda
   ➔
   Why classification is critical
   ➔
   Why classification is difficult
   ➔
   How we can make it sane
   ➔
   Final take-aways
   

   View Slide

4. Part 1:
   the problem
   

   View Slide

5. !!!
   !!!
   !!!
   !!!
   !!!
   !!!
   !!!
   

   View Slide

6. Configuration management
   «Configuration Management is the process of
   identifying and defining the items in the system,
   controlling the change of these items throughout
   their life-cycle, recording and reporting the status
   of items and change requests, and verifying the
   completeness and correctness of items»
   -- IEEE Glossary of Software Engineering Terminology (Standard
   729-1983)
   

   View Slide

7. Configuration management
   «Configuration Management is the process of
   identifying and defining the items in the system,
   controlling the change of these items throughout
   their life-cycle, recording and reporting the status
   of items and change requests, and verifying the
   completeness and correctness of items»
   -- IEEE Glossary of Software Engineering Terminology (Standard
   729-1983)
   

   View Slide

8. In configuration management
   you don't apply a
   configuration to
   a system, you
   rather apply it
   to classes
   of systems!
   

   View Slide

9. Part 2:
   challenges
   

   View Slide

10. Exceptions
    are the rule
    

    View Slide

11. Generic settings, the same for all systems
    Amsterdam
    DNS server NTP server Syslog server SSH configs
    Oslo
    Other random exceptions
    Seattle Iceland
    

    View Slide

12. Internal classification doesn't scale
    definitions
    explosion
    difficult
    reporting
    «The business should define what systems
    belong in which classes. The Cfengine
    administrator should build policy. Once
    the Cfengine administrator is left to
    manually defining classes within policy,
    you become a bottleneck»
    -- M.Svoboda, LinkedIn
    human
    bottleneck
    

    View Slide

13. Definitions explosion
    class1
    exception1 exception2
    

    View Slide

14. Definitions explosion
    class1
    exception1
    exception3
    exception2
    c1&!(ex1|ex2|ex3)
    ex1&!(ex2|ex3) ex2&!(ex1|ex3)
    ex3&!(ex1|ex2)
    (ex1&ex2)&!ex3
    (ex2&ex3)&!ex1
    ex1&ex2&ex3
    (ex1&ex3)&!ex2
    

    View Slide

15. Reporting
    ✔ take an infrastructure of 30000 nodes (LinkedIn-size!)
    ✔ try to report on something using SSH in a loop
    ✔ if no problem happens, and it takes ~3s per server on
    average, it takes 90000 seconds (>24h)
    ✔ if you do it with 10 parallel proceses, it's still 2.5h
    ✗ ...and you haven't started analysing the data yet
    ✗ ...and you haven't made it available to management yet
    ✗ ...and you need an up-to-date inventory of all nodes to
    provide accurate results
    ✗ ...and since not everybody has SSH access to the nodes or
    to the inventory, you are the only one (and the bottleneck)
    who can make that report!
    ✗ ...and it's an ideal case!
    

    View Slide

16. Your task is building policies,
    not defining classes
    

    View Slide

17. What if you had the ability to write a policy and
    use a class within that policy -- but the business
    could control which machines were associated
    with that class external to Cfengine? [...] The
    business can deploy an application to a machine,
    and Cfengine automatically responds in its policy
    because new classes have been set on the client
    automatically. Other business units can create
    classes of machines, and file a ticket for you to
    implement XYZ against the class they created.
    You are allowing your business to work the way it
    wants to.
    -- M.Svoboda, LinkedIn
    

    View Slide

18. Part 3:
    Solutions
    

    View Slide

19. Infrastructure complexity
    low complexity high complexity
    Technical skills
    well versed
    non technical
    Basic interface
    &
    basic backend
    Basic interface
    &
    sophisticated
    backend
    Sophisticated
    interface
    &
    sophisticated
    backend
    Sophisticated
    interface
    &
    basic backend
    

    View Slide

20. Puppet and ENC
    An external node classifier is an arbitrary script
    or application which can tell Puppet which
    classes a node should have [...] that can be called
    by puppet master; it doesn’t have to be written
    in Ruby. Its only argument is the name of the
    node to be classified, and it returns a YAML
    document describing the node. [...] To tell puppet
    master to use an ENC, you need to set two
    configuration options: node_terminus has to be
    set to “exec”, and external_nodes should have
    the path to the executable.
    

    View Slide

21. modules!
    ENC in CFEngine means...
    

    View Slide

22. The LinkedIn solution
    M.Svoboda - LinkedIn
    "Leveraging In-Memory Key Value Stores for Large-Scale Operations"
    

    View Slide

23. our implementation: hENC
    power &
    simplicity
    config info
    in plain
    text files
    module
    protocol
    a simple
    Perl script
    

    View Slide

24. View Slide

25. CFEngine's module protocol
    +activated_class
    -cancelled_class
    =my_var=my value
    =@my_list={ 'list','of','4','values'}
    =my_array[element]=value
    @my_array[list]={ 'list','of','4','values'}
    

    View Slide

26. #!/bin/sh
    /bin/egrep -h ^[=@+-] $* 2> /dev/null
    

    View Slide

27. #!/usr/bin/perl
    use strict ;
    use warnings ;
    my %class ; # classes container
    my %variable ; # variables container
    # Silence errors (e.g.: missing files)
    close STDERR ;
    while (my $line = <>) {
    chomp $line ;
    my ($setting,$id) =
    ( $line =~ m{^\s*([=\@+-])(.+)\s*$} ) ;
    # line didn't match the module protocol
    next if not defined $setting ;
    # add a class
    if ($setting eq '+') {
    # $id is a class name, or should be.
    $class{$id} = 1 ;
    }
    # undefine a class
    if ($setting eq '-') {
    # $id is a class name, or should be.
    $class{$id} = -1 ;
    }
    # define a variable/list
    if ($setting eq '=' or $setting eq '@') {
    # $id is "variable = something", or should be
    my ($varname) = ( $id =~ m{^(.+?)=} ) ;
    $variable{$varname} = $line ;
    }
    # discard the rest
    }
    # print out classes
    foreach my $classname (keys %class) {
    print "+$classname\n" if $class{$classname} > 0 ;
    print "-$classname\n" if $class{$classname} < 0 ;
    }
    # print variable/list assignments, the last one wins
    foreach my $assignment (values %variable) {
    print "$assignment\n" ;
    }
    

    View Slide

28. read
    general
    defaults
    read
    location
    defaults
    read
    environment
    defaults
    read
    node
    settings
    

    View Slide

29. oslo_public::
    "enc_subdir"
    policy => "overridable",
    string => "$(enc_basedir)/pub" ;
    on_private_net_only::
    "enc_subdir"
    policy => "overridable",
    string => "$(enc_basedir)/priv" ;
    oslo::
    "henclist"
    policy => "overridable",
    slist => {
    "$(enc_basedir)/_default_",
    "$(enc_basedir)/_oslo_",
    "$(enc_subdir)/_oslo_",
    "$(enc_subdir)/$(sys.domain)/$(sys.fqhost)",
    } ;
    }environment-
    dependent
    settings
    } general defaults
    } location defaults
    } environ. defaults
    

    View Slide

30. methods:
    "ENC"
    comment => "External node classification",
    usebundle => henc("site.henclist") ;
    vars:
    "motd_file"
    string => "$(henc.motd_file)",
    policy => "overridable" ;
    

    View Slide

31. bundle agent henc(enclist_name) {
    vars:
    henc_has_list::
    "enclist" slist => { "@($(enclist_name))" } ;
    "enc_fullpath" slist => maplist("$(site.inputs)/$(this)","enclist") ;
    "encargs" string => join(" ","enc_fullpath") ;
    classes:
    "henc_has_list" expression => isvariable("enclist_name") ;
    "henc_has_args" expression => isvariable("encargs") ;
    "henc_can_classify" and => { "henc_has_list","henc_has_args" } ;
    files:
    "$(site.lmodules)/henc"
    comment => "Copy/update hierarchical merger",
    copy_from => digest_cp("$(site.modules)/henc"),
    perms => mog("0755","root","root") ;
    henc_has_list::
    "$(site.inputs)/$(enclist)"
    comment => "Cache henc files locally",
    copy_from => digest_cp("$(site.masterfiles)/$(enclist)") ;
    commands:
    henc_can_classify.!henc_classes_activated::
    "$(site.lmodules)/henc"
    comment => "Hierarchical classification for $(sys.fqhost)",
    args => "$(encargs)",
    classes => always("henc_classes_activated"),
    module => "true" ;
    }
    

    View Slide

32. final take-aways
    smart
    classification
    is crucial...
    ...but doesn't
    need to be
    complicated...
    ...unless your
    infrastructure is.
    sometimes,
    even plain text
    is good enough!
    

    View Slide

33. View Slide

34. comments?
    @brontolinux
    mmarongiu@tiscali.it
    http://syslog.me
    http://no.linkedin.com/in/marcomarongiu/
    

    View Slide

35. THANK
    you
    

    View Slide