Configuration Management 101 @ Scale12x

Transcript

1. Configuration Management 101! Scale 12x

2. AKA

3. Sean Drops the Fucking Science

4. Sean OMeara! someara@opscode.com! @someara

5. Sean OMeara! someara@opscode.com! @someara

6. Sean OMeara! someara@getchef.com! @someara

7. whoami

8. Part 1

9. The Dawn of Configuration Management

10. • What is configuration management?! • Strategies and techniques for

    managing configuration and its complexity! • The art of change management

11. Manual Configuration

12. • Intuitive! • How we all start out! • Log

    into machine, manipulate with fingers! • Make with the clicky clicky! • Long tradition

13. • Somehow feels the “safest"! • First instinct in emergencies!

    • This is an illusion! • Do not do this

14. • Labor intensive! • Error prone! • Difficult to reproduce!

    • Obviously unsustainable

15. Scripting

16. • setup.sh! • setup.pl! • setup.py! • setup.rb

17. • doit.sh! • doit.pl! • doit.py! • doit.rb

18. • Ad-hoc in nature! • Loss of history! • Lacks

    testing methodology! • A step in the right direction

19. File Distribution

20. • NFS! • SMB! • AFS! • SSHFS! • GlusterFS

21. • uucp! • rcp! • ftp! • http! • scp

22. •Distributed systems! •Shares often managed manually or with scripts! •Package

    repositories! •Pull is better than push! •Scp on a cron *

23. Execution Management

24. • Image management! • Snapshots and cloning! • Containers

25. • SSH on a for loop! • Func! • Commands

    on message queues! • ISConf

26. • Loss of history! • Image sprawl! • Easy to

    order change across nodes

27. Convergent Operators! (promises)

28. None

29. Mark Burgess

30. The rest of us

31. Tools

32. • CFEngine! • Bcfg2! • Puppet! • Chef! • Salt!

    • Ansible
33. None

34. Part 2

35. Policy http://www.flickr.com/photos/sfllaw/222795669/

36. • /etc/passwd should be mode 0644! • /etc/shadow should be

    mode 0600

37. • user ‘kermit’ should exist! • user ‘fonzi’ should exist!

    • group ‘muppets’ should exist! • group ‘muppets’ should contain kermit and fonzi

38. • package ‘ntpd’ should be installed! • ntpd should sync

    with our AD service! • service ‘ntpd’ should be running

39. • package ‘httpd’ should be installed! • httpd should be

    expose /mnt/software/java! • service ‘httpd’ should be running

40. • The Java JDK, version 7u45, found on an internally

    hosted web server, should be installed into /usr/local/jdk-7u45/

41. Polices are declarations about the state of things in a

    system

42. Polices are applied repeatedly and repair the system when needed

43. Policies often change

44. • package ‘widget-factory’ should be installed at version 1.2.3

45. • package ‘widget-factory’ should be installed at version 1.3.0

46. http://www.flickr.com/photos/jakepjohnson/4937767595 Repeatability

47. Repeatable -> Idempotent -> Convergent

48. • Scripts are not generally repeatable

49. None
50. None

51. • But they can be!

52. None

53. ! Idempotent operations can be applied infinite times and will

    yield the same result every time
54. None

55. Idempotent http://www.flickr.com/photos/ian_munroe/4758240536/

56. http://www.flickr.com/photos/ian_munroe/4758240536/ Idempotent NOT GOOD ENOUGH

57. ! Convergent operations test state and repair if needed

58. None
59. None

60. ! A control loop keeps the system stable and allows

    for change when policy is updated

61. Autonomous agent Policy: The box should be closed

62. Convergence

63. None
64. None
65. None

66. Converging with Bash

67. git clone git@github.com:someara/ cbash.git

68. None
69. None
70. None
71. None
72. None
73. None
74. None
75. None
76. None
77. None
78. None
79. None
80. None

81. Convergence and Iteration

82. None
83. None
84. None
85. None
86. None
87. None

88. Does order matter?

89. YES

90. None

91. Promises http://www.flickr.com/photos/nazzen9009/6809694353/

92. • Agents are autonomous! • A promise is a signal

    or message perceived by an observer.! • Promises may or may not be kept.! • Agents can observe other agents! • Agents only have local information *! • Inner workings of agents are assumed to be unknown http://markburgess.org/BookOfPromises.pdf

93. • Agents have intentions (possible behaviors)! • Agents can make

    assessments about other agents http://markburgess.org/BookOfPromises.pdf

94. • Configuration Management tools embody tenants of Promise Theory intentionally

    or not

95. Domain Specific Languages

96. ! DSLs restrict machine instructions to convergent operations

97. ! DSLs manage ordering

98. None

99. type subject intentions

100. None

101. type subject intentions

102. signal

103. None

104. type subject intention

105. observation

106. None

107. type subject intentions

108. None

109. type intention subject

110. signal

111. Intermission

112. None

113. Part 3

114. Composition

115. None

116. Recipes

117. resource one resource two resource three

118. { testable intent

119. recipe[http::server]

120. recipe[http::server]

121. recipes supporting files

122. Types

123. None

124. interface implementation

125. None

126. intentions parameters

127. None

128. new scope intention implementation

129. new scope intention implementation

130. Artifacts

131. metadata

132. None

133. metadata

134. None

135. http v0.1.0 chef-server api yum v3.0.0

136. Delivery

137. • nodes request their own initial run_list

138. recipe[httpd::server] chef-server api run_list: http v0.1.0

139. recipe[httpd::server] chef-server api run_list: http v0.1.0 recipe[openssh::server] openssh v3.2.1

140. recipe[ntp::client] chef-server api run_list: http v0.1.0 recipe[openssh::server] openssh v3.2.1 recipe[httpd::server]

     ntp v1.0.0

141. • Push vs Pull! • Networking considerations! • Machines down

     for maintenance! • Machines that don’t exist yet

142. Dependencies

143. None
144. None
145. None
146. None
147. None
148. None

149. recipe[widgetfactory] chef-server api run_list: http v0.1.0 yum v3.0.0 widgetfactory v1.0.0

150. Integration testing

151. • Test that a set of agents has achieved their

     combined goal

152. • lsof -i :80! • ps -ef | grep httpd!

     • curl localhost 2>&1 > /dev/null

153. • Berkshelf! • Vagrant! • Kitchen.ci! • Bats! • Serverspec

154. Environments

155. • Environments constrain cookbook versions! • Environments can set data

156. None
157. None

158. • Environments can be used to test branches! • Environments

     can be used to segregate machines! • Environments can be manipulated programatically
159. None
160. None

161. http v0.1.0 chef-server api http v0.2.0 openssh v1.2.3 postgresql v3.2.1

162. recipe[widgetfactory] run_list: http v0.1.0 yum v3.0.0 widgetfactory v1.0.0 chef_environment: production

163. recipe[widgetfactory] run_list: http v0.2.0 yum v3.0.0 widgetfactory v1.0.0 chef_environment: staging

164. Part 4

165. Clusters http://www.flickr.com/photos/youraccount/5938852370/

166. Typical Cluster

167. loadbalancer application db-slave db-master

168. Production httpd 0.1.0

169. Production Staging httpd 0.1.0 httpd 0.1.0

170. Production Staging UUID httpd 0.1.0 httpd 0.1.0 httpd 0.2.0

171. Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

172. Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

173. Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

174. Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

175. Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

176. Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

177. Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

178. Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

179. Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

180. Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

181. Production Staging UUID httpd 0.1.0 httpd 0.2.0 httpd 0.2.0

182. Production Staging httpd 0.2.0 httpd 0.2.0

183. Production httpd 0.2.0

184. An Ordering Problem

185. loadbalancer application

186. • Take a machine out of the pool! • Drain

     the connections! • Modify configuration! • Insert it back into the pool

187. loadbalancer application

188. loadbalancer application

189. loadbalancer application

190. loadbalancer application

191. loadbalancer application

192. loadbalancer application

193. loadbalancer application

194. loadbalancer application

195. loadbalancer application

196. loadbalancer application

197. Orchestration

198. • Conductor showing signals to autonomous agents (creative policy manipulation)!

     • External actor controlling sequencing (execution management)! • Application level sequencing (vector clocks, etc)

199. • Infrastructures are snowflakes! • Solutions are unique to applications

     by nature! • Configuration Management 201

200. • There is no separation between ‘infrastructure’ and ‘application’! •

     Distributed systems are hard! • Specialists need to work together

201. Devops

202. • Study Promise Theory! • Study distributed systems! • Develop

     high quality primitives! • Be excellent to each other

203. Fin