Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Configuration Management 101

someara
February 04, 2014

Configuration Management 101

Given at FOSDEM 2014

someara

February 04, 2014
Tweet

More Decks by someara

Other Decks in Programming

Transcript

  1. Configuration Management 101!
    FOSDEM 2014

    View full-size slide

  2. The Dawn of Configuration
    Management

    View full-size slide

  3. • What is configuration management?!
    • Strategies and techniques for managing
    configuration and its complexity!
    • The art of change management

    View full-size slide

  4. Manual Configuration

    View full-size slide

  5. • Intuitive!
    • How we all start out!
    • Log into machine, manipulate with fingers!
    • Make with the clicky clicky!
    • Long tradition

    View full-size slide

  6. • Somehow feels the “safest"!
    • First instinct in emergencies!
    • This is an illusion!
    • Do not do this

    View full-size slide

  7. • Labor intensive!
    • Error prone!
    • Difficult to reproduce!
    • Obviously unsustainable

    View full-size slide

  8. • setup.sh!
    • setup.pl!
    • setup.py!
    • setup.rb

    View full-size slide

  9. • doit.sh!
    • doit.pl!
    • doit.py!
    • doit.rb

    View full-size slide

  10. • Ad-hoc in nature!
    • Loss of history!
    • Lacks testing methodology!
    • A step in the right direction

    View full-size slide

  11. File Distribution

    View full-size slide

  12. • NFS!
    • SMB!
    • AFS!
    • SSHFS!
    • GlusterFS

    View full-size slide

  13. • uucp!
    • rcp!
    • ftp!
    • http!
    • scp

    View full-size slide

  14. •Distributed systems!
    •Shares often managed manually or with
    scripts!
    •Package repositories!
    •Pull is better than push!
    •Scp on a cron *

    View full-size slide

  15. Execution Management

    View full-size slide

  16. • Image management!
    • Snapshots and cloning!
    • Containers

    View full-size slide

  17. • SSH on a for loop!
    • Func!
    • Commands on message queues!
    • ISConf

    View full-size slide

  18. • Loss of history!
    • Image sprawl!
    • Easy to order change across nodes

    View full-size slide

  19. Convergent Operators!
    (promises)

    View full-size slide

  20. • CFEngine!
    • Bcfg2!
    • Puppet!
    • Chef!
    • Salt!
    • Ansible

    View full-size slide

  21. Policy
    http://www.flickr.com/photos/sfllaw/222795669/

    View full-size slide

  22. • /etc/passwd should be mode 0644!
    • /etc/shadow should be mode 0600

    View full-size slide

  23. • user ‘kermit’ should exist!
    • user ‘fonzi’ should exist!
    • group ‘muppets’ should exist!
    • group ‘muppets’ should contain kermit
    and fonzi

    View full-size slide

  24. • package ‘ntpd’ should be installed!
    • ntpd should sync with our AD service!
    • service ‘ntpd’ should be running

    View full-size slide

  25. • package ‘httpd’ should be installed!
    • httpd should be expose /mnt/software/java!
    • service ‘httpd’ should be running

    View full-size slide

  26. • The Java JDK, version 7u45, found on an
    internally hosted web server, should be
    installed into /usr/local/jdk-7u45/

    View full-size slide

  27. Polices are declarations
    about the state of things
    in a system

    View full-size slide

  28. Polices are applied
    repeatedly and repair the
    system when needed

    View full-size slide

  29. Policies often change

    View full-size slide

  30. • package ‘widget-factory’ should be
    installed at version 1.2.3

    View full-size slide

  31. • package ‘widget-factory’ should be
    installed at version 1.3.0

    View full-size slide

  32. http://www.flickr.com/photos/jakepjohnson/4937767595
    Repeatability

    View full-size slide

  33. Repeatable -> Idempotent ->
    Convergent

    View full-size slide

  34. • Scripts are not generally repeatable

    View full-size slide

  35. • But they can be!

    View full-size slide

  36. !
    Idempotent operations can be
    applied infinite times and will
    yield the same result every time

    View full-size slide

  37. Idempotent
    http://www.flickr.com/photos/ian_munroe/4758240536/

    View full-size slide

  38. http://www.flickr.com/photos/ian_munroe/4758240536/
    Idempotent
    NOT GOOD
    ENOUGH

    View full-size slide

  39. !
    Convergent operations
    test state and repair if
    needed

    View full-size slide

  40. !
    A control loop keeps the
    system stable and allows for
    change when policy is updated

    View full-size slide

  41. Autonomous agent
    Policy: The box should be closed

    View full-size slide

  42. Converging with Bash

    View full-size slide

  43. git clone [email protected]:someara/
    cbash.git

    View full-size slide

  44. Convergence and Iteration

    View full-size slide

  45. Does order matter?

    View full-size slide

  46. Promises
    http://www.flickr.com/photos/nazzen9009/6809694353/

    View full-size slide

  47. • Agents are autonomous!
    • A promise is a signal or message perceived by
    an observer.!
    • Promises may or may not be kept.!
    • Agents can observe other agents!
    • Agents only have local information *!
    • Inner workings of agents are assumed to be
    unknown
    http://markburgess.org/BookOfPromises.pdf

    View full-size slide

  48. • Agents have intentions (possible
    behaviors)!
    • Agents can make assessments about
    other agents
    http://markburgess.org/BookOfPromises.pdf

    View full-size slide

  49. • Configuration Management tools embody
    tenants of Promise Theory intentionally or
    not

    View full-size slide

  50. Domain Specific Languages

    View full-size slide

  51. !
    DSLs restrict machine
    instructions to
    convergent operations

    View full-size slide

  52. !
    DSLs manage
    ordering

    View full-size slide

  53. type subject intentions

    View full-size slide

  54. type
    subject intentions

    View full-size slide

  55. type subject
    intention

    View full-size slide

  56. type
    subject
    intentions

    View full-size slide

  57. type
    intention
    subject

    View full-size slide

  58. Intermission

    View full-size slide

  59. resource one
    resource two
    resource three

    View full-size slide

  60. {
    testable

    intent

    View full-size slide

  61. recipe[http::server]

    View full-size slide

  62. recipe[http::server]

    View full-size slide

  63. recipes
    supporting files

    View full-size slide

  64. interface
    implementation

    View full-size slide

  65. intentions
    parameters

    View full-size slide

  66. new scope
    intention implementation

    View full-size slide

  67. new scope
    intention implementation

    View full-size slide

  68. http v0.1.0
    chef-server api
    yum v3.0.0

    View full-size slide

  69. • nodes request their own initial run_list

    View full-size slide

  70. recipe[httpd::server]
    chef-server api
    run_list:
    http v0.1.0

    View full-size slide

  71. recipe[httpd::server]
    chef-server api
    run_list:
    http v0.1.0
    recipe[openssh::server]
    openssh v3.2.1

    View full-size slide

  72. recipe[ntp::client]
    chef-server api run_list:
    http v0.1.0
    recipe[openssh::server]
    openssh v3.2.1
    recipe[httpd::server]
    ntp v1.0.0

    View full-size slide

  73. • Push vs Pull!
    • Networking considerations!
    • Machines down for maintenance!
    • Machines that don’t exist yet

    View full-size slide

  74. Dependencies

    View full-size slide

  75. recipe[widgetfactory]
    chef-server api
    run_list:
    http v0.1.0
    yum v3.0.0
    widgetfactory v1.0.0

    View full-size slide

  76. Integration testing

    View full-size slide

  77. • Test that a set of agents has achieved
    their combined goal

    View full-size slide

  78. • lsof -i :80!
    • ps -ef | grep httpd!
    • curl localhost 2>&1 > /dev/null

    View full-size slide

  79. • Berkshelf!
    • Vagrant!
    • Kitchen.ci!
    • Bats!
    • Serverspec

    View full-size slide

  80. Environments

    View full-size slide

  81. • Environments constrain cookbook
    versions!
    • Environments can set data

    View full-size slide

  82. • Environments can be used to test
    branches!
    • Environments can be used to segregate
    machines!
    • Environments can be manipulated
    programatically

    View full-size slide

  83. http v0.1.0
    chef-server api
    http v0.2.0
    openssh v1.2.3
    postgresql v3.2.1

    View full-size slide

  84. recipe[widgetfactory]
    run_list:
    http v0.1.0
    yum v3.0.0
    widgetfactory v1.0.0
    chef_environment: production

    View full-size slide

  85. recipe[widgetfactory]
    run_list:
    http v0.2.0
    yum v3.0.0
    widgetfactory v1.0.0
    chef_environment: staging

    View full-size slide

  86. Clusters
    http://www.flickr.com/photos/youraccount/5938852370/

    View full-size slide

  87. Typical Cluster

    View full-size slide

  88. loadbalancer
    application
    db-slave
    db-master

    View full-size slide

  89. Production
    httpd 0.1.0

    View full-size slide

  90. Production Staging
    httpd 0.1.0 httpd 0.1.0

    View full-size slide

  91. Production Staging UUID
    httpd 0.1.0 httpd 0.1.0 httpd 0.2.0

    View full-size slide

  92. Production Staging UUID
    httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

    View full-size slide

  93. Production Staging UUID
    httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

    View full-size slide

  94. Production Staging UUID
    httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

    View full-size slide

  95. Production Staging UUID
    httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

    View full-size slide

  96. Production Staging UUID
    httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

    View full-size slide

  97. Production Staging UUID
    httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

    View full-size slide

  98. Production Staging UUID
    httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

    View full-size slide

  99. Production Staging UUID
    httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

    View full-size slide

  100. Production Staging UUID
    httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

    View full-size slide

  101. Production Staging UUID
    httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

    View full-size slide

  102. Production Staging UUID
    httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

    View full-size slide

  103. Production Staging
    httpd 0.2.0 httpd 0.2.0

    View full-size slide

  104. Production
    httpd 0.2.0

    View full-size slide

  105. An Ordering Problem

    View full-size slide

  106. loadbalancer
    application

    View full-size slide

  107. • Take a machine out of the pool!
    • Drain the connections!
    • Modify configuration!
    • Insert it back into the pool

    View full-size slide

  108. loadbalancer
    application

    View full-size slide

  109. loadbalancer
    application

    View full-size slide

  110. loadbalancer
    application

    View full-size slide

  111. loadbalancer
    application

    View full-size slide

  112. loadbalancer
    application

    View full-size slide

  113. loadbalancer
    application

    View full-size slide

  114. loadbalancer
    application

    View full-size slide

  115. loadbalancer
    application

    View full-size slide

  116. loadbalancer
    application

    View full-size slide

  117. loadbalancer
    application

    View full-size slide

  118. Orchestration

    View full-size slide

  119. • Conductor showing signals to
    autonomous agents (creative policy
    manipulation)!
    • External actor controlling sequencing
    (execution management)!
    • Application level sequencing (vector
    clocks, etc)

    View full-size slide

  120. • Infrastructures are snowflakes!
    • Solutions are unique to applications by
    nature!
    • Configuration Management 201

    View full-size slide

  121. • There is no separation between
    ‘infrastructure’ and ‘application’!
    • Distributed systems are hard!
    • Specialists need to work together

    View full-size slide

  122. • Study Promise Theory!
    • Study distributed systems!
    • Develop high quality primitives!
    • Be excellent to each other

    View full-size slide