Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Configuration Management 101 @ Scale12x

someara
February 28, 2014

Configuration Management 101 @ Scale12x

someara

February 28, 2014
Tweet

More Decks by someara

Other Decks in Technology

Transcript

  1. Configuration Management 101!
    Scale 12x

    View Slide

  2. AKA

    View Slide

  3. Sean Drops the Fucking
    Science

    View Slide

  4. Sean OMeara!
    [email protected]!
    @someara

    View Slide

  5. Sean OMeara!
    [email protected]!
    @someara

    View Slide

  6. Sean OMeara!
    [email protected]!
    @someara

    View Slide

  7. whoami

    View Slide

  8. Part 1

    View Slide

  9. The Dawn of Configuration
    Management

    View Slide

  10. • What is configuration management?!
    • Strategies and techniques for managing
    configuration and its complexity!
    • The art of change management

    View Slide

  11. Manual Configuration

    View Slide

  12. • Intuitive!
    • How we all start out!
    • Log into machine, manipulate with fingers!
    • Make with the clicky clicky!
    • Long tradition

    View Slide

  13. • Somehow feels the “safest"!
    • First instinct in emergencies!
    • This is an illusion!
    • Do not do this

    View Slide

  14. • Labor intensive!
    • Error prone!
    • Difficult to reproduce!
    • Obviously unsustainable

    View Slide

  15. Scripting

    View Slide

  16. • setup.sh!
    • setup.pl!
    • setup.py!
    • setup.rb

    View Slide

  17. • doit.sh!
    • doit.pl!
    • doit.py!
    • doit.rb

    View Slide

  18. • Ad-hoc in nature!
    • Loss of history!
    • Lacks testing methodology!
    • A step in the right direction

    View Slide

  19. File Distribution

    View Slide

  20. • NFS!
    • SMB!
    • AFS!
    • SSHFS!
    • GlusterFS

    View Slide

  21. • uucp!
    • rcp!
    • ftp!
    • http!
    • scp

    View Slide

  22. •Distributed systems!
    •Shares often managed manually or with
    scripts!
    •Package repositories!
    •Pull is better than push!
    •Scp on a cron *

    View Slide

  23. Execution Management

    View Slide

  24. • Image management!
    • Snapshots and cloning!
    • Containers

    View Slide

  25. • SSH on a for loop!
    • Func!
    • Commands on message queues!
    • ISConf

    View Slide

  26. • Loss of history!
    • Image sprawl!
    • Easy to order change across nodes

    View Slide

  27. Convergent Operators!
    (promises)

    View Slide

  28. View Slide

  29. Mark Burgess

    View Slide

  30. The rest of us

    View Slide

  31. Tools

    View Slide

  32. • CFEngine!
    • Bcfg2!
    • Puppet!
    • Chef!
    • Salt!
    • Ansible

    View Slide

  33. View Slide

  34. Part 2

    View Slide

  35. Policy
    http://www.flickr.com/photos/sfllaw/222795669/

    View Slide

  36. • /etc/passwd should be mode 0644!
    • /etc/shadow should be mode 0600

    View Slide

  37. • user ‘kermit’ should exist!
    • user ‘fonzi’ should exist!
    • group ‘muppets’ should exist!
    • group ‘muppets’ should contain kermit
    and fonzi

    View Slide

  38. • package ‘ntpd’ should be installed!
    • ntpd should sync with our AD service!
    • service ‘ntpd’ should be running

    View Slide

  39. • package ‘httpd’ should be installed!
    • httpd should be expose /mnt/software/java!
    • service ‘httpd’ should be running

    View Slide

  40. • The Java JDK, version 7u45, found on an
    internally hosted web server, should be
    installed into /usr/local/jdk-7u45/

    View Slide

  41. Polices are declarations
    about the state of things
    in a system

    View Slide

  42. Polices are applied
    repeatedly and repair the
    system when needed

    View Slide

  43. Policies often change

    View Slide

  44. • package ‘widget-factory’ should be
    installed at version 1.2.3

    View Slide

  45. • package ‘widget-factory’ should be
    installed at version 1.3.0

    View Slide

  46. http://www.flickr.com/photos/jakepjohnson/4937767595
    Repeatability

    View Slide

  47. Repeatable -> Idempotent ->
    Convergent

    View Slide

  48. • Scripts are not generally repeatable

    View Slide

  49. View Slide

  50. View Slide

  51. • But they can be!

    View Slide

  52. View Slide

  53. !
    Idempotent operations can be
    applied infinite times and will
    yield the same result every time

    View Slide

  54. View Slide

  55. Idempotent
    http://www.flickr.com/photos/ian_munroe/4758240536/

    View Slide

  56. http://www.flickr.com/photos/ian_munroe/4758240536/
    Idempotent
    NOT GOOD
    ENOUGH

    View Slide

  57. !
    Convergent operations
    test state and repair if
    needed

    View Slide

  58. View Slide

  59. View Slide

  60. !
    A control loop keeps the
    system stable and allows for
    change when policy is updated

    View Slide

  61. Autonomous agent
    Policy: The box should be closed

    View Slide

  62. Convergence

    View Slide

  63. View Slide

  64. View Slide

  65. View Slide

  66. Converging with Bash

    View Slide

  67. git clone [email protected]:someara/
    cbash.git

    View Slide

  68. View Slide

  69. View Slide

  70. View Slide

  71. View Slide

  72. View Slide

  73. View Slide

  74. View Slide

  75. View Slide

  76. View Slide

  77. View Slide

  78. View Slide

  79. View Slide

  80. View Slide

  81. Convergence and Iteration

    View Slide

  82. View Slide

  83. View Slide

  84. View Slide

  85. View Slide

  86. View Slide

  87. View Slide

  88. Does order matter?

    View Slide

  89. YES

    View Slide

  90. View Slide

  91. Promises
    http://www.flickr.com/photos/nazzen9009/6809694353/

    View Slide

  92. • Agents are autonomous!
    • A promise is a signal or message perceived by
    an observer.!
    • Promises may or may not be kept.!
    • Agents can observe other agents!
    • Agents only have local information *!
    • Inner workings of agents are assumed to be
    unknown
    http://markburgess.org/BookOfPromises.pdf

    View Slide

  93. • Agents have intentions (possible
    behaviors)!
    • Agents can make assessments about
    other agents
    http://markburgess.org/BookOfPromises.pdf

    View Slide

  94. • Configuration Management tools embody
    tenants of Promise Theory intentionally or
    not

    View Slide

  95. Domain Specific Languages

    View Slide

  96. !
    DSLs restrict machine
    instructions to
    convergent operations

    View Slide

  97. !
    DSLs manage
    ordering

    View Slide

  98. View Slide

  99. type subject intentions

    View Slide

  100. View Slide

  101. type
    subject intentions

    View Slide

  102. signal

    View Slide

  103. View Slide

  104. type subject
    intention

    View Slide

  105. observation

    View Slide

  106. View Slide

  107. type
    subject
    intentions

    View Slide

  108. View Slide

  109. type
    intention
    subject

    View Slide

  110. signal

    View Slide

  111. Intermission

    View Slide

  112. View Slide

  113. Part 3

    View Slide

  114. Composition

    View Slide

  115. View Slide

  116. Recipes

    View Slide

  117. resource one
    resource two
    resource three

    View Slide

  118. {
    testable

    intent

    View Slide

  119. recipe[http::server]

    View Slide

  120. recipe[http::server]

    View Slide

  121. recipes
    supporting files

    View Slide

  122. Types

    View Slide

  123. View Slide

  124. interface
    implementation

    View Slide

  125. View Slide

  126. intentions
    parameters

    View Slide

  127. View Slide

  128. new scope
    intention implementation

    View Slide

  129. new scope
    intention implementation

    View Slide

  130. Artifacts

    View Slide

  131. metadata

    View Slide

  132. View Slide

  133. metadata

    View Slide

  134. View Slide

  135. http v0.1.0
    chef-server api
    yum v3.0.0

    View Slide

  136. Delivery

    View Slide

  137. • nodes request their own initial run_list

    View Slide

  138. recipe[httpd::server]
    chef-server api
    run_list:
    http v0.1.0

    View Slide

  139. recipe[httpd::server]
    chef-server api
    run_list:
    http v0.1.0
    recipe[openssh::server]
    openssh v3.2.1

    View Slide

  140. recipe[ntp::client]
    chef-server api run_list:
    http v0.1.0
    recipe[openssh::server]
    openssh v3.2.1
    recipe[httpd::server]
    ntp v1.0.0

    View Slide

  141. • Push vs Pull!
    • Networking considerations!
    • Machines down for maintenance!
    • Machines that don’t exist yet

    View Slide

  142. Dependencies

    View Slide

  143. View Slide

  144. View Slide

  145. View Slide

  146. View Slide

  147. View Slide

  148. View Slide

  149. recipe[widgetfactory]
    chef-server api
    run_list:
    http v0.1.0
    yum v3.0.0
    widgetfactory v1.0.0

    View Slide

  150. Integration testing

    View Slide

  151. • Test that a set of agents has achieved
    their combined goal

    View Slide

  152. • lsof -i :80!
    • ps -ef | grep httpd!
    • curl localhost 2>&1 > /dev/null

    View Slide

  153. • Berkshelf!
    • Vagrant!
    • Kitchen.ci!
    • Bats!
    • Serverspec

    View Slide

  154. Environments

    View Slide

  155. • Environments constrain cookbook
    versions!
    • Environments can set data

    View Slide

  156. View Slide

  157. View Slide

  158. • Environments can be used to test
    branches!
    • Environments can be used to segregate
    machines!
    • Environments can be manipulated
    programatically

    View Slide

  159. View Slide

  160. View Slide

  161. http v0.1.0
    chef-server api
    http v0.2.0
    openssh v1.2.3
    postgresql v3.2.1

    View Slide

  162. recipe[widgetfactory]
    run_list:
    http v0.1.0
    yum v3.0.0
    widgetfactory v1.0.0
    chef_environment: production

    View Slide

  163. recipe[widgetfactory]
    run_list:
    http v0.2.0
    yum v3.0.0
    widgetfactory v1.0.0
    chef_environment: staging

    View Slide

  164. Part 4

    View Slide

  165. Clusters
    http://www.flickr.com/photos/youraccount/5938852370/

    View Slide

  166. Typical Cluster

    View Slide

  167. loadbalancer
    application
    db-slave
    db-master

    View Slide

  168. Production
    httpd 0.1.0

    View Slide

  169. Production Staging
    httpd 0.1.0 httpd 0.1.0

    View Slide

  170. Production Staging UUID
    httpd 0.1.0 httpd 0.1.0 httpd 0.2.0

    View Slide

  171. Production Staging UUID
    httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

    View Slide

  172. Production Staging UUID
    httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

    View Slide

  173. Production Staging UUID
    httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

    View Slide

  174. Production Staging UUID
    httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

    View Slide

  175. Production Staging UUID
    httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

    View Slide

  176. Production Staging UUID
    httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

    View Slide

  177. Production Staging UUID
    httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

    View Slide

  178. Production Staging UUID
    httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

    View Slide

  179. Production Staging UUID
    httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

    View Slide

  180. Production Staging UUID
    httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

    View Slide

  181. Production Staging UUID
    httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

    View Slide

  182. Production Staging
    httpd 0.2.0 httpd 0.2.0

    View Slide

  183. Production
    httpd 0.2.0

    View Slide

  184. An Ordering Problem

    View Slide

  185. loadbalancer
    application

    View Slide

  186. • Take a machine out of the pool!
    • Drain the connections!
    • Modify configuration!
    • Insert it back into the pool

    View Slide

  187. loadbalancer
    application

    View Slide

  188. loadbalancer
    application

    View Slide

  189. loadbalancer
    application

    View Slide

  190. loadbalancer
    application

    View Slide

  191. loadbalancer
    application

    View Slide

  192. loadbalancer
    application

    View Slide

  193. loadbalancer
    application

    View Slide

  194. loadbalancer
    application

    View Slide

  195. loadbalancer
    application

    View Slide

  196. loadbalancer
    application

    View Slide

  197. Orchestration

    View Slide

  198. • Conductor showing signals to
    autonomous agents (creative policy
    manipulation)!
    • External actor controlling sequencing
    (execution management)!
    • Application level sequencing (vector
    clocks, etc)

    View Slide

  199. • Infrastructures are snowflakes!
    • Solutions are unique to applications by
    nature!
    • Configuration Management 201

    View Slide

  200. • There is no separation between
    ‘infrastructure’ and ‘application’!
    • Distributed systems are hard!
    • Specialists need to work together

    View Slide

  201. Devops

    View Slide

  202. • Study Promise Theory!
    • Study distributed systems!
    • Develop high quality primitives!
    • Be excellent to each other

    View Slide

  203. Fin

    View Slide