Configuration Management 101 @ Scale12x

Transcript

1. Configuration Management 101! Scale 12x
2. AKA
3. Sean Drops the Fucking Science
4. Sean OMeara! someara@opscode.com! @someara
5. Sean OMeara! someara@opscode.com! @someara
6. Sean OMeara! someara@getchef.com! @someara
7. whoami
8. Part 1
9. The Dawn of Configuration Management
10. • What is configuration management?! • Strategies and techniques for managing configuration and its complexity! • The art of change management
11. Manual Configuration
12. • Intuitive! • How we all start out! • Log into machine, manipulate with fingers! • Make with the clicky clicky! • Long tradition
13. • Somehow feels the “safest"! • First instinct in emergencies! • This is an illusion! • Do not do this
14. • Labor intensive! • Error prone! • Difficult to reproduce! • Obviously unsustainable
15. Scripting
16. • setup.sh! • setup.pl! • setup.py! • setup.rb
17. • doit.sh! • doit.pl! • doit.py! • doit.rb
18. • Ad-hoc in nature! • Loss of history! • Lacks testing methodology! • A step in the right direction
19. File Distribution
20. • NFS! • SMB! • AFS! • SSHFS! • GlusterFS
21. • uucp! • rcp! • ftp! • http! • scp
22. •Distributed systems! •Shares often managed manually or with scripts! •Package repositories! •Pull is better than push! •Scp on a cron *
23. Execution Management
24. • Image management! • Snapshots and cloning! • Containers
25. • SSH on a for loop! • Func! • Commands on message queues! • ISConf
26. • Loss of history! • Image sprawl! • Easy to order change across nodes
27. Convergent Operators! (promises)
29. Mark Burgess
30. The rest of us
31. Tools
32. • CFEngine! • Bcfg2! • Puppet! • Chef! • Salt! • Ansible
34. Part 2
35. Policy http://www.flickr.com/photos/sfllaw/222795669/
36. • /etc/passwd should be mode 0644! • /etc/shadow should be mode 0600
37. • user ‘kermit’ should exist! • user ‘fonzi’ should exist! • group ‘muppets’ should exist! • group ‘muppets’ should contain kermit and fonzi
38. • package ‘ntpd’ should be installed! • ntpd should sync with our AD service! • service ‘ntpd’ should be running
39. • package ‘httpd’ should be installed! • httpd should be expose /mnt/software/java! • service ‘httpd’ should be running
40. • The Java JDK, version 7u45, found on an internally hosted web server, should be installed into /usr/local/jdk-7u45/
41. Polices are declarations about the state of things in a system
42. Polices are applied repeatedly and repair the system when needed
43. Policies often change
44. • package ‘widget-factory’ should be installed at version 1.2.3
45. • package ‘widget-factory’ should be installed at version 1.3.0
46. http://www.flickr.com/photos/jakepjohnson/4937767595 Repeatability
47. Repeatable -> Idempotent -> Convergent
48. • Scripts are not generally repeatable
51. • But they can be!
53. ! Idempotent operations can be applied infinite times and will yield the same result every time
55. Idempotent http://www.flickr.com/photos/ian_munroe/4758240536/
56. http://www.flickr.com/photos/ian_munroe/4758240536/ Idempotent NOT GOOD ENOUGH
57. ! Convergent operations test state and repair if needed
60. ! A control loop keeps the system stable and allows for change when policy is updated
61. Autonomous agent Policy: The box should be closed
62. Convergence
66. Converging with Bash
67. git clone git@github.com:someara/ cbash.git
81. Convergence and Iteration
88. Does order matter?
89. YES
91. Promises http://www.flickr.com/photos/nazzen9009/6809694353/
92. • Agents are autonomous! • A promise is a signal or message perceived by an observer.! • Promises may or may not be kept.! • Agents can observe other agents! • Agents only have local information *! • Inner workings of agents are assumed to be unknown http://markburgess.org/BookOfPromises.pdf
93. • Agents have intentions (possible behaviors)! • Agents can make assessments about other agents http://markburgess.org/BookOfPromises.pdf
94. • Configuration Management tools embody tenants of Promise Theory intentionally or not
95. Domain Specific Languages
96. ! DSLs restrict machine instructions to convergent operations
97. ! DSLs manage ordering
99. type subject intentions
101. type subject intentions
102. signal
104. type subject intention
105. observation
107. type subject intentions
109. type intention subject
110. signal
111. Intermission
113. Part 3
114. Composition
116. Recipes
117. resource one resource two resource three
118. { testable intent
119. recipe[http::server]
120. recipe[http::server]
121. recipes supporting files
122. Types
124. interface implementation
126. intentions parameters
128. new scope intention implementation
129. new scope intention implementation
130. Artifacts
131. metadata
133. metadata
135. http v0.1.0 chef-server api yum v3.0.0
136. Delivery
137. • nodes request their own initial run_list
138. recipe[httpd::server] chef-server api run_list: http v0.1.0
139. recipe[httpd::server] chef-server api run_list: http v0.1.0 recipe[openssh::server] openssh v3.2.1
140. recipe[ntp::client] chef-server api run_list: http v0.1.0 recipe[openssh::server] openssh v3.2.1 recipe[httpd::server] ntp v1.0.0
141. • Push vs Pull! • Networking considerations! • Machines down for maintenance! • Machines that don’t exist yet
142. Dependencies
149. recipe[widgetfactory] chef-server api run_list: http v0.1.0 yum v3.0.0 widgetfactory v1.0.0
150. Integration testing
151. • Test that a set of agents has achieved their combined goal
152. • lsof -i :80! • ps -ef | grep httpd! • curl localhost 2>&1 > /dev/null
153. • Berkshelf! • Vagrant! • Kitchen.ci! • Bats! • Serverspec
154. Environments
155. • Environments constrain cookbook versions! • Environments can set data
158. • Environments can be used to test branches! • Environments can be used to segregate machines! • Environments can be manipulated programatically
161. http v0.1.0 chef-server api http v0.2.0 openssh v1.2.3 postgresql v3.2.1
162. recipe[widgetfactory] run_list: http v0.1.0 yum v3.0.0 widgetfactory v1.0.0 chef_environment: production
163. recipe[widgetfactory] run_list: http v0.2.0 yum v3.0.0 widgetfactory v1.0.0 chef_environment: staging
164. Part 4
165. Clusters http://www.flickr.com/photos/youraccount/5938852370/
166. Typical Cluster
167. loadbalancer application db-slave db-master
168. Production httpd 0.1.0
169. Production Staging httpd 0.1.0 httpd 0.1.0
170. Production Staging UUID httpd 0.1.0 httpd 0.1.0 httpd 0.2.0
171. Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
172. Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
173. Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
174. Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
175. Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
176. Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
177. Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
178. Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
179. Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
180. Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
181. Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
182. Production Staging httpd 0.2.0 httpd 0.2.0
183. Production httpd 0.2.0
184. An Ordering Problem
185. loadbalancer application
186. • Take a machine out of the pool! • Drain the connections! • Modify configuration! • Insert it back into the pool
187. loadbalancer application
188. loadbalancer application
189. loadbalancer application
190. loadbalancer application
191. loadbalancer application
192. loadbalancer application
193. loadbalancer application
194. loadbalancer application
195. loadbalancer application
196. loadbalancer application
197. Orchestration
198. • Conductor showing signals to autonomous agents (creative policy manipulation)! • External actor controlling sequencing (execution management)! • Application level sequencing (vector clocks, etc)
199. • Infrastructures are snowflakes! • Solutions are unique to applications by nature! • Configuration Management 201
200. • There is no separation between ‘infrastructure’ and ‘application’! • Distributed systems are hard! • Specialists need to work together
201. Devops
202. • Study Promise Theory! • Study distributed systems! • Develop high quality primitives! • Be excellent to each other
203. Fin