Configuration Management 101 @ Scale12x

Transcript

1. Configuration Management 101!
   Scale 12x
   

   View Slide

2. AKA
   

   View Slide

3. Sean Drops the Fucking
   Science
   

   View Slide

4. Sean OMeara!
   someara@opscode.com!
   @someara
   

   View Slide

5. Sean OMeara!
   someara@opscode.com!
   @someara
   

   View Slide

6. Sean OMeara!
   someara@getchef.com!
   @someara
   

   View Slide

7. whoami
   

   View Slide

8. Part 1
   

   View Slide

9. The Dawn of Configuration
   Management
   

   View Slide

10. • What is configuration management?!
    • Strategies and techniques for managing
    configuration and its complexity!
    • The art of change management
    

    View Slide

11. Manual Configuration
    

    View Slide

12. • Intuitive!
    • How we all start out!
    • Log into machine, manipulate with fingers!
    • Make with the clicky clicky!
    • Long tradition
    

    View Slide

13. • Somehow feels the “safest"!
    • First instinct in emergencies!
    • This is an illusion!
    • Do not do this
    

    View Slide

14. • Labor intensive!
    • Error prone!
    • Difficult to reproduce!
    • Obviously unsustainable
    

    View Slide

15. Scripting
    

    View Slide

16. • setup.sh!
    • setup.pl!
    • setup.py!
    • setup.rb
    

    View Slide

17. • doit.sh!
    • doit.pl!
    • doit.py!
    • doit.rb
    

    View Slide

18. • Ad-hoc in nature!
    • Loss of history!
    • Lacks testing methodology!
    • A step in the right direction
    

    View Slide

19. File Distribution
    

    View Slide

20. • NFS!
    • SMB!
    • AFS!
    • SSHFS!
    • GlusterFS
    

    View Slide

21. • uucp!
    • rcp!
    • ftp!
    • http!
    • scp
    

    View Slide

22. •Distributed systems!
    •Shares often managed manually or with
    scripts!
    •Package repositories!
    •Pull is better than push!
    •Scp on a cron *
    

    View Slide

23. Execution Management
    

    View Slide

24. • Image management!
    • Snapshots and cloning!
    • Containers
    

    View Slide

25. • SSH on a for loop!
    • Func!
    • Commands on message queues!
    • ISConf
    

    View Slide

26. • Loss of history!
    • Image sprawl!
    • Easy to order change across nodes
    

    View Slide

27. Convergent Operators!
    (promises)
    

    View Slide

28. View Slide

29. Mark Burgess
    

    View Slide

30. The rest of us
    

    View Slide

31. Tools
    

    View Slide

32. • CFEngine!
    • Bcfg2!
    • Puppet!
    • Chef!
    • Salt!
    • Ansible
    

    View Slide

33. View Slide

34. Part 2
    

    View Slide

35. Policy
    http://www.flickr.com/photos/sfllaw/222795669/
    

    View Slide

36. • /etc/passwd should be mode 0644!
    • /etc/shadow should be mode 0600
    

    View Slide

37. • user ‘kermit’ should exist!
    • user ‘fonzi’ should exist!
    • group ‘muppets’ should exist!
    • group ‘muppets’ should contain kermit
    and fonzi
    

    View Slide

38. • package ‘ntpd’ should be installed!
    • ntpd should sync with our AD service!
    • service ‘ntpd’ should be running
    

    View Slide

39. • package ‘httpd’ should be installed!
    • httpd should be expose /mnt/software/java!
    • service ‘httpd’ should be running
    

    View Slide

40. • The Java JDK, version 7u45, found on an
    internally hosted web server, should be
    installed into /usr/local/jdk-7u45/
    

    View Slide

41. Polices are declarations
    about the state of things
    in a system
    

    View Slide

42. Polices are applied
    repeatedly and repair the
    system when needed
    

    View Slide

43. Policies often change
    

    View Slide

44. • package ‘widget-factory’ should be
    installed at version 1.2.3
    

    View Slide

45. • package ‘widget-factory’ should be
    installed at version 1.3.0
    

    View Slide

46. http://www.flickr.com/photos/jakepjohnson/4937767595
    Repeatability
    

    View Slide

47. Repeatable -> Idempotent ->
    Convergent
    

    View Slide

48. • Scripts are not generally repeatable
    

    View Slide

49. View Slide

50. View Slide

51. • But they can be!
    

    View Slide

52. View Slide

53. !
    Idempotent operations can be
    applied infinite times and will
    yield the same result every time
    

    View Slide

54. View Slide

55. Idempotent
    http://www.flickr.com/photos/ian_munroe/4758240536/
    

    View Slide

56. http://www.flickr.com/photos/ian_munroe/4758240536/
    Idempotent
    NOT GOOD
    ENOUGH
    

    View Slide

57. !
    Convergent operations
    test state and repair if
    needed
    

    View Slide

58. View Slide

59. View Slide

60. !
    A control loop keeps the
    system stable and allows for
    change when policy is updated
    

    View Slide

61. Autonomous agent
    Policy: The box should be closed
    

    View Slide

62. Convergence
    

    View Slide

63. View Slide

64. View Slide

65. View Slide

66. Converging with Bash
    

    View Slide

67. git clone git@github.com:someara/
    cbash.git
    

    View Slide

68. View Slide

69. View Slide

70. View Slide

71. View Slide

72. View Slide

73. View Slide

74. View Slide

75. View Slide

76. View Slide

77. View Slide

78. View Slide

79. View Slide

80. View Slide

81. Convergence and Iteration
    

    View Slide

82. View Slide

83. View Slide

84. View Slide

85. View Slide

86. View Slide

87. View Slide

88. Does order matter?
    

    View Slide

89. YES
    

    View Slide

90. View Slide

91. Promises
    http://www.flickr.com/photos/nazzen9009/6809694353/
    

    View Slide

92. • Agents are autonomous!
    • A promise is a signal or message perceived by
    an observer.!
    • Promises may or may not be kept.!
    • Agents can observe other agents!
    • Agents only have local information *!
    • Inner workings of agents are assumed to be
    unknown
    http://markburgess.org/BookOfPromises.pdf
    

    View Slide

93. • Agents have intentions (possible
    behaviors)!
    • Agents can make assessments about
    other agents
    http://markburgess.org/BookOfPromises.pdf
    

    View Slide

94. • Configuration Management tools embody
    tenants of Promise Theory intentionally or
    not
    

    View Slide

95. Domain Specific Languages
    

    View Slide

96. !
    DSLs restrict machine
    instructions to
    convergent operations
    

    View Slide

97. !
    DSLs manage
    ordering
    

    View Slide

98. View Slide

99. type subject intentions
    

    View Slide

100. View Slide

101. type
     subject intentions
     

     View Slide

102. signal
     

     View Slide

103. View Slide

104. type subject
     intention
     

     View Slide

105. observation
     

     View Slide

106. View Slide

107. type
     subject
     intentions
     

     View Slide

108. View Slide

109. type
     intention
     subject
     

     View Slide

110. signal
     

     View Slide

111. Intermission
     

     View Slide

112. View Slide

113. Part 3
     

     View Slide

114. Composition
     

     View Slide

115. View Slide

116. Recipes
     

     View Slide

117. resource one
     resource two
     resource three
     

     View Slide

118. {
     testable
     
     intent
     

     View Slide

119. recipe[http::server]
     

     View Slide

120. recipe[http::server]
     

     View Slide

121. recipes
     supporting files
     

     View Slide

122. Types
     

     View Slide

123. View Slide

124. interface
     implementation
     

     View Slide

125. View Slide

126. intentions
     parameters
     

     View Slide

127. View Slide

128. new scope
     intention implementation
     

     View Slide

129. new scope
     intention implementation
     

     View Slide

130. Artifacts
     

     View Slide

131. metadata
     

     View Slide

132. View Slide

133. metadata
     

     View Slide

134. View Slide

135. http v0.1.0
     chef-server api
     yum v3.0.0
     

     View Slide

136. Delivery
     

     View Slide

137. • nodes request their own initial run_list
     

     View Slide

138. recipe[httpd::server]
     chef-server api
     run_list:
     http v0.1.0
     

     View Slide

139. recipe[httpd::server]
     chef-server api
     run_list:
     http v0.1.0
     recipe[openssh::server]
     openssh v3.2.1
     

     View Slide

140. recipe[ntp::client]
     chef-server api run_list:
     http v0.1.0
     recipe[openssh::server]
     openssh v3.2.1
     recipe[httpd::server]
     ntp v1.0.0
     

     View Slide

141. • Push vs Pull!
     • Networking considerations!
     • Machines down for maintenance!
     • Machines that don’t exist yet
     

     View Slide

142. Dependencies
     

     View Slide

143. View Slide

144. View Slide

145. View Slide

146. View Slide

147. View Slide

148. View Slide

149. recipe[widgetfactory]
     chef-server api
     run_list:
     http v0.1.0
     yum v3.0.0
     widgetfactory v1.0.0
     

     View Slide

150. Integration testing
     

     View Slide

151. • Test that a set of agents has achieved
     their combined goal
     

     View Slide

152. • lsof -i :80!
     • ps -ef | grep httpd!
     • curl localhost 2>&1 > /dev/null
     

     View Slide

153. • Berkshelf!
     • Vagrant!
     • Kitchen.ci!
     • Bats!
     • Serverspec
     

     View Slide

154. Environments
     

     View Slide

155. • Environments constrain cookbook
     versions!
     • Environments can set data
     

     View Slide

156. View Slide

157. View Slide

158. • Environments can be used to test
     branches!
     • Environments can be used to segregate
     machines!
     • Environments can be manipulated
     programatically
     

     View Slide

159. View Slide

160. View Slide

161. http v0.1.0
     chef-server api
     http v0.2.0
     openssh v1.2.3
     postgresql v3.2.1
     

     View Slide

162. recipe[widgetfactory]
     run_list:
     http v0.1.0
     yum v3.0.0
     widgetfactory v1.0.0
     chef_environment: production
     

     View Slide

163. recipe[widgetfactory]
     run_list:
     http v0.2.0
     yum v3.0.0
     widgetfactory v1.0.0
     chef_environment: staging
     

     View Slide

164. Part 4
     

     View Slide

165. Clusters
     http://www.flickr.com/photos/youraccount/5938852370/
     

     View Slide

166. Typical Cluster
     

     View Slide

167. loadbalancer
     application
     db-slave
     db-master
     

     View Slide

168. Production
     httpd 0.1.0
     

     View Slide

169. Production Staging
     httpd 0.1.0 httpd 0.1.0
     

     View Slide

170. Production Staging UUID
     httpd 0.1.0 httpd 0.1.0 httpd 0.2.0
     

     View Slide

171. Production Staging UUID
     httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
     

     View Slide

172. Production Staging UUID
     httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
     

     View Slide

173. Production Staging UUID
     httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
     

     View Slide

174. Production Staging UUID
     httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
     

     View Slide

175. Production Staging UUID
     httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
     

     View Slide

176. Production Staging UUID
     httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
     

     View Slide

177. Production Staging UUID
     httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
     

     View Slide

178. Production Staging UUID
     httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
     

     View Slide

179. Production Staging UUID
     httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
     

     View Slide

180. Production Staging UUID
     httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
     

     View Slide

181. Production Staging UUID
     httpd 0.1.0 httpd 0.2.0 httpd 0.2.0
     

     View Slide

182. Production Staging
     httpd 0.2.0 httpd 0.2.0
     

     View Slide

183. Production
     httpd 0.2.0
     

     View Slide

184. An Ordering Problem
     

     View Slide

185. loadbalancer
     application
     

     View Slide

186. • Take a machine out of the pool!
     • Drain the connections!
     • Modify configuration!
     • Insert it back into the pool
     

     View Slide

187. loadbalancer
     application
     

     View Slide

188. loadbalancer
     application
     

     View Slide

189. loadbalancer
     application
     

     View Slide

190. loadbalancer
     application
     

     View Slide

191. loadbalancer
     application
     

     View Slide

192. loadbalancer
     application
     

     View Slide

193. loadbalancer
     application
     

     View Slide

194. loadbalancer
     application
     

     View Slide

195. loadbalancer
     application
     

     View Slide

196. loadbalancer
     application
     

     View Slide

197. Orchestration
     

     View Slide

198. • Conductor showing signals to
     autonomous agents (creative policy
     manipulation)!
     • External actor controlling sequencing
     (execution management)!
     • Application level sequencing (vector
     clocks, etc)
     

     View Slide

199. • Infrastructures are snowflakes!
     • Solutions are unique to applications by
     nature!
     • Configuration Management 201
     

     View Slide

200. • There is no separation between
     ‘infrastructure’ and ‘application’!
     • Distributed systems are hard!
     • Specialists need to work together
     

     View Slide

201. Devops
     

     View Slide

202. • Study Promise Theory!
     • Study distributed systems!
     • Develop high quality primitives!
     • Be excellent to each other
     

     View Slide

203. Fin
     

     View Slide