Infrastructure is development

Presentation from the 2009 Red Hat Summit. This was the first presentation I did to use Sharpie slides in some cases, mostly because I got mad at OpenOffice. I don't agree with everything in there anymore, but on the whole, I still like it.

This covers agile infrastructure. Today, some people call that Devops.

This also was the fun because I did 300+ slides in about 40 minutes.

Transcript

1. Red Hat Summit 2009 | Michael Stahnke 1 Infrastructure is Development Michael Stahnke (stahnma) 02-SEP-2009
2. Red Hat Summit 2009 | Michael Stahnke 2 The ideas presented today are not representative of my employer, business success, past jobs and do not offer endorsement to any particular products brands or companies. Heck, a lot of this stuff isn't even my idea to begin with. http://www.flickr.com/photos/sea-turtle/3049443478/ http://www.flickr.com/photos/jjze/726682393/
3. Red Hat Summit 2009 | Michael Stahnke 3 “There are only two hard problems in Computer Science: cache invalidation and naming things.” --Phil Karlton
4. Red Hat Summit 2009 | Michael Stahnke 4 Who Am I?
5. Red Hat Summit 2009 | Michael Stahnke 5 Who Am I?
6. Red Hat Summit 2009 | Michael Stahnke 6 Who Am I?
7. Red Hat Summit 2009 | Michael Stahnke 7 Who Am I?
8. Red Hat Summit 2009 | Michael Stahnke 8 Who Am I? <Insert Large Company Name Here>
9. Red Hat Summit 2009 | Michael Stahnke 9 Where Am I? • http://stahnma.fedorapeople.org • http://www.stahnkage.com • @stahnma on Twitter • @stahnma on identi.ca • stahnma on github • stahnma@fedoraproject.org • stahnma on Freenode IRC
10. Red Hat Summit 2009 | Michael Stahnke 10 Where Am I? If you google stahnma, it's probably me.
11. Red Hat Summit 2009 | Michael Stahnke 11 Baseline Infrastructure – the collection of all components that make up the non-external facing IT realm. Primarily, in this case, servers connected to storage and network. It can include network/SAN devices in some cases.
12. Red Hat Summit 2009 | Michael Stahnke 12 What's Coming Up The Tao of the Agile Infrastructure
13. Red Hat Summit 2009 | Michael Stahnke 13 What's Coming Up Inherit Problems with System Administration
14. Red Hat Summit 2009 | Michael Stahnke 14 What's Coming Up Some solutions to problems with System Administration
15. Red Hat Summit 2009 | Michael Stahnke 15 Three Domains 1. Infrastructure Practices
16. Red Hat Summit 2009 | Michael Stahnke 16 Three Domains 1. Infrastructure Practices 2. People
17. Red Hat Summit 2009 | Michael Stahnke 17 Three Domains 1. Infrastructure Practices 2. People 3. Technology Choices
18. Red Hat Summit 2009 | Michael Stahnke 18 The Problem
19. Red Hat Summit 2009 | Michael Stahnke 19 System Administration isn't a science
20. Red Hat Summit 2009 | Michael Stahnke 20 You can't get a degree in System Administration
21. Red Hat Summit 2009 | Michael Stahnke 21 Heterogeneous Systems
22. Red Hat Summit 2009 | Michael Stahnke 22 No Clear Job Description
23. Red Hat Summit 2009 | Michael Stahnke 23 Expectations not Clear
24. Red Hat Summit 2009 | Michael Stahnke 24 Can't keep up with demand
25. Red Hat Summit 2009 | Michael Stahnke 25 Admins fight management http://www.flickr.com/photos/jjze/726682393/
26. Red Hat Summit 2009 | Michael Stahnke 26 Solutions
27. Red Hat Summit 2009 | Michael Stahnke 27 Are delivered Solutions
28. Red Hat Summit 2009 | Michael Stahnke 28 Through Ideas Solutions
29. Red Hat Summit 2009 | Michael Stahnke 29 Put together by Good People Solutions
30. Red Hat Summit 2009 | Michael Stahnke 30 Using some awesome technologies Solutions
31. Red Hat Summit 2009 | Michael Stahnke 31 Topic 1 Infrastructure Practices
32. Red Hat Summit 2009 | Michael Stahnke 32 Why is your infrastructure special?
33. Red Hat Summit 2009 | Michael Stahnke 33 Steal Ideas http://www.flickr.com/photos/purplemattfish/3508761485/
34. Red Hat Summit 2009 | Michael Stahnke 34 Open your infrastructure
35. Red Hat Summit 2009 | Michael Stahnke 35 Opening your infrastructure includes your issues
36. Red Hat Summit 2009 | Michael Stahnke 36 Open your infrastructure and your solutions
37. Red Hat Summit 2009 | Michael Stahnke 37 Well, what do you mean?
38. Red Hat Summit 2009 | Michael Stahnke 38 To Have a High Performing Team : • You must know what you manage.
39. Red Hat Summit 2009 | Michael Stahnke 39 What do you have? • Asset Database • LDAP Directory • Hardware Management Tools • Power Management Tools • Monitoring Tools • Provisioning Tools • Storage Management Tools • Backup Tools • Policy Engines • Patch Tools • Security Scanning Tools • Virtualization Management Tools • Log Management Tools
40. Red Hat Summit 2009 | Michael Stahnke 40 Common Example: • Asset Database – Who knows? • LDAP Directory – RHDS, 389, SunOne, AD, OpenLDAP • Hardware Management Tools – IBM Director, DRAC • Power Management Tools – APC • Monitoring Tools – Nagios, Tivoli, OpenView • Provisioning Tools – Cobbler, Vmware, • Storage Management Tools – IBM Whatever, Some Custom Stuff • Backup Tools – Netbackup, Tivoli, Networker, Tar, Gzip, Rsync, Cron • Policy Engines – Cfengine, Puppet, Scripts • Patch Tools – RHN, NIM, Custom Repos • Security Scanning Tools – Lots of stuff • Virtualization Management Tools – vCenter, Virt-Manager, Spacewalk, RHN • Log Management Tools – Syslog Server • DNS – Bind
41. Red Hat Summit 2009 | Michael Stahnke 41
42. Red Hat Summit 2009 | Michael Stahnke 42 What now? A. You can cry about it
43. Red Hat Summit 2009 | Michael Stahnke 43 What now? A. You can cry about it B. You can remove data sources
44. Red Hat Summit 2009 | Michael Stahnke 44 What now? A. You can cry about it B. You can remove data sources C. You can integrate/federate them
45. Red Hat Summit 2009 | Michael Stahnke 45 What now? A. You can cry about it B. You can remove data sources C. You can integrate/federate them D. All of the Above
46. Red Hat Summit 2009 | Michael Stahnke 46 What now? A. You can cry about it B. You can remove data sources C. You can integrate/federate them D. All of the Above The correct answers are both B and C; however it is very likely you will encounter A, so the likely answer is, in fact, D.
47. Red Hat Summit 2009 | Michael Stahnke 47 Infrastructure is Development
48. Red Hat Summit 2009 | Michael Stahnke 48 The Infrastructure is the Application
49. Red Hat Summit 2009 | Michael Stahnke 49 Application == Infrastructure
50. Red Hat Summit 2009 | Michael Stahnke 50 Infrastructure Goals • Deliver results to the business
51. Red Hat Summit 2009 | Michael Stahnke 51 Infrastructure Goals • Deliver results to the business • Make the infrastructure an investment, not a cost
52. Red Hat Summit 2009 | Michael Stahnke 52 Infrastructure Direction • Solving Problems
53. Red Hat Summit 2009 | Michael Stahnke 53 Infrastructure Direction • Solving Problems • Automation
54. Red Hat Summit 2009 | Michael Stahnke 54 Infrastructure Direction • Solving Problems • Automation • The right mix of people
55. Red Hat Summit 2009 | Michael Stahnke 55 Infrastructure Direction • Solving Problems • Automation • The right mix of people • The right decision processes
56. Red Hat Summit 2009 | Michael Stahnke 56 Infrastructure Direction • Solving Problems • Automation • The right mix of people • The right decision processes • Vision
57. Red Hat Summit 2009 | Michael Stahnke 57 Infrastructure Direction • Solving Problems • Automation • The right mix of people • The right decision processes • Vision • Known State
58. Red Hat Summit 2009 | Michael Stahnke 58 Where do we start?
59. Red Hat Summit 2009 | Michael Stahnke 59 Axiom 1 Reuse before building or purchasing
60. Red Hat Summit 2009 | Michael Stahnke 60 Your Infrastructure isn't a secret
61. Red Hat Summit 2009 | Michael Stahnke 61 Your Infrastructure isn't the secret sauce
62. Red Hat Summit 2009 | Michael Stahnke 62 Your Infrastructure isn't differentiating
63. Red Hat Summit 2009 | Michael Stahnke 63 Your Infrastructure isn't a secret • Everybody has servers, a network, some storage • Somebody has probably solved this problem • Check some common places: • Google • IRC • Sourceforge • Ohloh.net • Amazon book selection • What solutions do you see?
64. Red Hat Summit 2009 | Michael Stahnke 64 So when I say re-use??? Reuse the code and tools you have
65. Red Hat Summit 2009 | Michael Stahnke 65 So when I say re-use??? Enable features you are currently not utilizing
66. Red Hat Summit 2009 | Michael Stahnke 66 So when I say re-use??? Search for a solution built on an open platform
67. Red Hat Summit 2009 | Michael Stahnke 67 So when I say re-use??? Find other organizations successes on an open infrastructure
68. Red Hat Summit 2009 | Michael Stahnke 68 You can't reuse? Ask why.
69. Red Hat Summit 2009 | Michael Stahnke 69 You can't reuse? Is your problem new?
70. Red Hat Summit 2009 | Michael Stahnke 70 You can't reuse? Is your problem special?
71. Red Hat Summit 2009 | Michael Stahnke 71 You can't reuse? Ok, you can look at building or purchasing... http://www.flickr.com:80/photos/iambrad/289462494/
72. Red Hat Summit 2009 | Michael Stahnke 72 I have/need a purchased proprietary solution for Problem X
73. Red Hat Summit 2009 | Michael Stahnke 73 I have/need a purchased proprietary solution for Problem X So do I, and I'm sorry. http://www.stahnkage.com/outage
74. Red Hat Summit 2009 | Michael Stahnke 74 Software/Tool Selection
75. Red Hat Summit 2009 | Michael Stahnke 75 Purchased Software Selection Criteria There a few things to evaluate
76. Red Hat Summit 2009 | Michael Stahnke 76 Purchased Software Selection Criteria There a few things to evaluate • Price
77. Red Hat Summit 2009 | Michael Stahnke 77 Purchased Software Selection Criteria There a few things to evaluate • Price • Performance
78. Red Hat Summit 2009 | Michael Stahnke 78 Purchased Software Selection Criteria There a few things to evaluate • Price • Performance • Functionality
79. Red Hat Summit 2009 | Michael Stahnke 79 Purchased Software Selection Criteria There a few things to evaluate • Price • Performance • Functionality • But really....
80. Red Hat Summit 2009 | Michael Stahnke 80 Meatcloud Manifesto
81. Red Hat Summit 2009 | Michael Stahnke 81 Cloud Computing is all the rage
82. Red Hat Summit 2009 | Michael Stahnke 82 Cloud Computing is all the rage Scaling through people is not
83. Red Hat Summit 2009 | Michael Stahnke 83 The GUI is for what some user interface designer thought you wanted to do. The CLI is for what you actually need to get done.
84. Red Hat Summit 2009 | Michael Stahnke 84 The GUI is for what some user interface designer thought you wanted to do. The CLI is for what you actually need to get done. -- Mike Stahnke
85. Red Hat Summit 2009 | Michael Stahnke 85 Axiom 2 Don't Leverage the Meatcloud
86. Red Hat Summit 2009 | Michael Stahnke 86 Choose your technologies wisely
87. Red Hat Summit 2009 | Michael Stahnke 87 Very wisely
88. Red Hat Summit 2009 | Michael Stahnke 88 Rules for Software Evaluation • Do not implement any product that does not provide an API.
89. Red Hat Summit 2009 | Michael Stahnke 89 Rules for Software Evaluation • Do not implement any product that does not provide an API. • The provided API must have all functionality that the application provides.
90. Red Hat Summit 2009 | Michael Stahnke 90 Rules for Software Evaluation • Do not implement any product that does not provide an API. • The provided API must have all functionality that the application provides. • The provided API must be tailored to more than one language and platform.
91. Red Hat Summit 2009 | Michael Stahnke 91 Rules for Software Evaluation • Do not implement any product that does not provide an API. • The provided API must have all functionality that the application provides. • The provided API must be tailored to more than one language and platform. • Source code counts as an API, and may be restricted to one language or platform.
92. Red Hat Summit 2009 | Michael Stahnke 92 Rules for Software Evaluation • Do not implement any product that does not provide an API. • The provided API must have all functionality that the application provides. • The provided API must be tailored to more than one language and platform. • Source code counts as an API, and may be restricted to one language or platform. • The API must include functional examples and not require someone to be an expert on the product to use.
93. Red Hat Summit 2009 | Michael Stahnke 93 Rules for Software Evaluation • Do not use any product with configurations that are not machine parseable and machine writable
94. Red Hat Summit 2009 | Michael Stahnke 94 Rules for Software Evaluation • Do not use any product with configurations that are not machine parseable and machine writable • All data stored in the product must be machine readable and writable by applications other than the product itself.
95. Red Hat Summit 2009 | Michael Stahnke 95 Rules for Software Evaluation • Do not use any product with configurations that are not machine parseable and machine writable • All data stored in the product must be machine readable and writable by applications other than the product itself. • Writing work-arounds to cover the deficiencies in a product should be less work than writing the product’s designed functionality.
96. Red Hat Summit 2009 | Michael Stahnke 96 But I don't have automation specialists
97. Red Hat Summit 2009 | Michael Stahnke 97 That's a problem.
98. Red Hat Summit 2009 | Michael Stahnke 98 Do you have developers?
99. Red Hat Summit 2009 | Michael Stahnke 99
100. Red Hat Summit 2009 | Michael Stahnke 100 What's the difference between development and automation?
101. Red Hat Summit 2009 | Michael Stahnke 101 Process?
102. Red Hat Summit 2009 | Michael Stahnke 102 Requirements Gathering?
103. Red Hat Summit 2009 | Michael Stahnke 103 Cost?
104. Red Hat Summit 2009 | Michael Stahnke 104 Testing?
105. Red Hat Summit 2009 | Michael Stahnke 105 Egos?
106. Red Hat Summit 2009 | Michael Stahnke 106 Funky Lava Lamps?
107. Red Hat Summit 2009 | Michael Stahnke 107 Design Patterns?
108. Red Hat Summit 2009 | Michael Stahnke 108 Code portability?
109. Red Hat Summit 2009 | Michael Stahnke 109 What's the difference between development and automation?
110. Red Hat Summit 2009 | Michael Stahnke 110 Developers write code for use by somebody else http://www.flickr.com:80/photos/davesag/71441707/
111. Red Hat Summit 2009 | Michael Stahnke 111 System Admins write code for use by themselves, and hopefully somebody else http://www.flickr.com:80/photos/herzogbr/2274372747/
112. Red Hat Summit 2009 | Michael Stahnke 112 People make the Infrastructure Agile
113. Red Hat Summit 2009 | Michael Stahnke 113 Attract Talent People
114. Red Hat Summit 2009 | Michael Stahnke 114 Retain Talent People
115. Red Hat Summit 2009 | Michael Stahnke 115 People Hire rock stars
116. Red Hat Summit 2009 | Michael Stahnke 116 People Treat them like rock stars
117. Red Hat Summit 2009 | Michael Stahnke 117 People Observe their organizational behavior
118. Red Hat Summit 2009 | Michael Stahnke 118 People “...organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations.” -- Conway's Law (1968)
119. Red Hat Summit 2009 | Michael Stahnke 119 People Test their knowledge
120. Red Hat Summit 2009 | Michael Stahnke 120 Hiring Questions
121. Red Hat Summit 2009 | Michael Stahnke 121 You want rock stars remember?
122. Red Hat Summit 2009 | Michael Stahnke 122 Some helpful hiring questions Describe your home network setup.
123. Red Hat Summit 2009 | Michael Stahnke 123 Some helpful hiring questions What cool technology projects didn't make your resume?
124. Red Hat Summit 2009 | Michael Stahnke 124 Some helpful hiring questions Do you know what version control is?
125. Red Hat Summit 2009 | Michael Stahnke 125 Some helpful hiring questions Do you know what version control is? Are you beyond CVS?
126. Red Hat Summit 2009 | Michael Stahnke 126 Some helpful hiring questions Security or Availability?
127. Red Hat Summit 2009 | Michael Stahnke 127 Some helpful hiring questions Apt or Yum?
128. Red Hat Summit 2009 | Michael Stahnke 128 Some helpful hiring questions Provide three uses for a towel.
129. Red Hat Summit 2009 | Michael Stahnke 129 Some helpful hiring questions Pirates or Ninjas?
130. Red Hat Summit 2009 | Michael Stahnke 130 Some helpful hiring questions Icanhascheezburger or ihasahotdog ?
131. Red Hat Summit 2009 | Michael Stahnke 131 Two Types of People Breadth vs Depth
132. Red Hat Summit 2009 | Michael Stahnke 132 Breadth vs Depth Breadth • Systems Automation People.
133. Red Hat Summit 2009 | Michael Stahnke 133 Breadth vs Depth Breadth • Systems Automation People. • Big Picture thinkers.
134. Red Hat Summit 2009 | Michael Stahnke 134 Breadth vs Depth Breadth • Systems Automation People. • Big Picture thinkers. • Visionaries.
135. Red Hat Summit 2009 | Michael Stahnke 135 Breadth vs Depth Breadth • Systems Automation People. • Big Picture thinkers. • Visionaries. • Holistic View.
136. Red Hat Summit 2009 | Michael Stahnke 136 Breadth vs Depth Depth • Solve the problem for this exact situation.
137. Red Hat Summit 2009 | Michael Stahnke 137 Breadth vs Depth Depth • Solve the problem for this exact situation. • Tune like it's going out of style.
138. Red Hat Summit 2009 | Michael Stahnke 138 Breadth vs Depth Depth • Solve the problem for this exact situation. • Tune like it's going out of style. • Maximum ROI.
139. Red Hat Summit 2009 | Michael Stahnke 139 Breadth vs Depth Depth • Solve the problem for this exact situation. • Tune like it's going out of style. • Maximum ROI. • Deep understanding of the technology.
140. Red Hat Summit 2009 | Michael Stahnke 140 Breadth vs Depth Turns out, you need both.
141. Red Hat Summit 2009 | Michael Stahnke 141 People Specialist vs Generalist
142. Red Hat Summit 2009 | Michael Stahnke 142 People Specialist vs Generalist.^H^H^H^H^...not so much.
143. Red Hat Summit 2009 | Michael Stahnke 143 How many people is the right amount?
144. Red Hat Summit 2009 | Michael Stahnke 144 50:1?
145. Red Hat Summit 2009 | Michael Stahnke 145 100:1?
146. Red Hat Summit 2009 | Michael Stahnke 146 200:1?
147. Red Hat Summit 2009 | Michael Stahnke 147 2:1?
148. Red Hat Summit 2009 | Michael Stahnke 148 Gartner Says:
149. Red Hat Summit 2009 | Michael Stahnke 149 Gartner Says: “Yes”
150. Red Hat Summit 2009 | Michael Stahnke 150 There is one rule for Server/Admin Ratio:
151. Red Hat Summit 2009 | Michael Stahnke 151
152. Red Hat Summit 2009 | Michael Stahnke 152 Axiom 2 Don't Leverage the Meatcloud
153. Red Hat Summit 2009 | Michael Stahnke 153 We've Covered Technology Selection
154. Red Hat Summit 2009 | Michael Stahnke 154 We've Covered People Decisions
155. Red Hat Summit 2009 | Michael Stahnke 155 Let's cover Community
156. Red Hat Summit 2009 | Michael Stahnke 156 Axiom 3 Decouple your Infrastructure.
157. Red Hat Summit 2009 | Michael Stahnke 157 Code for the generic case
158. Red Hat Summit 2009 | Michael Stahnke 158 Stop reinventing the wheel, srsly.
159. Red Hat Summit 2009 | Michael Stahnke 159 Moving Forward Can you implement an Open Strategy to solve this issue?
160. Red Hat Summit 2009 | Michael Stahnke 160 Moving Forward If you have to solve it yourself, can it be Open for others?
161. Red Hat Summit 2009 | Michael Stahnke 161 Moving Forward Have an Open Infrastructure
162. Red Hat Summit 2009 | Michael Stahnke 162 Moving Forward Driving on hexagonal wheels isn't fun http://www.flickr.com:80/photos/andrewmbutler/2762480367/
163. Red Hat Summit 2009 | Michael Stahnke 163 Moving Forward Driving on hexagonal wheels isn't fun http://www.flickr.com:80/photos/andrewmbutler/2762480367/ Ask this guy.
164. Red Hat Summit 2009 | Michael Stahnke 164 Moving Forward Driving on hexagonal wheels isn't fun Quit reinventing the wheel...poorly
165. Red Hat Summit 2009 | Michael Stahnke 165 Steal it all Practical Examples of Open Infrastructure
166. Red Hat Summit 2009 | Michael Stahnke 166 Steal it all Practical Examples The Fedora Infrastructure Project https://fedoraproject.org/wiki/Infrastructure
167. Red Hat Summit 2009 | Michael Stahnke 167 Steal it all Practical Examples The Community Services Infrastructure Standards http://infrastructure.fedoraproject.org/csi/free-software-policy/en- US/html-single/ https://fedorahosted.org/csi/
168. Red Hat Summit 2009 | Michael Stahnke 168 De-coupled Infrastructure Benefits • Commonly accepted solutions to problem http://www.flickr.com:80/photos/thomas-merton/255204957
169. Red Hat Summit 2009 | Michael Stahnke 169 De-coupled Infrastructure Benefits • Commonly accepted solutions to problem • Portability of solutions http://www.flickr.com:80/photos/thomas-merton/255204957
170. Red Hat Summit 2009 | Michael Stahnke 170 De-coupled Infrastructure Benefits • Commonly accepted solutions to problem • Portability of solutions • Ability to hire knowledgeable individuals http://www.flickr.com:80/photos/thomas-merton/255204957
171. Red Hat Summit 2009 | Michael Stahnke 171 De-coupled Infrastructure Benefits • Commonly accepted solutions to problem • Portability of solutions • Ability to hire knowledgeable individuals • Able to retain rock stars http://www.flickr.com:80/photos/thomas-merton/255204957
172. Red Hat Summit 2009 | Michael Stahnke 172 De-coupled Infrastructure Benefits • Commonly accepted solutions to problem • Portability of solutions • Ability to hire knowledgeable individuals • Able to retain rock stars • Community built around tools and support http://www.flickr.com:80/photos/thomas-merton/255204957
173. Red Hat Summit 2009 | Michael Stahnke 173 3 Axioms 1. Reuse before building or purchasing 2. Don't leverage the meatcloud 3. Decouple your infrastructure
174. Red Hat Summit 2009 | Michael Stahnke 174
175. Red Hat Summit 2009 | Michael Stahnke 175 Your Time is Valuable
176. Red Hat Summit 2009 | Michael Stahnke 176 Waste less of it
177. Red Hat Summit 2009 | Michael Stahnke 177 Your team's time is valuable
178. Red Hat Summit 2009 | Michael Stahnke 178 Manage it
179. Red Hat Summit 2009 | Michael Stahnke 179 Your Time is Valuable Identify tasks on which the team spends the most time. Commonly • (Growth) Deployment • Account Management • Ad-Hoc File Transfer type activity • Patches http://www.flickr.com:80/photos/denaldo/3326738675/
180. Red Hat Summit 2009 | Michael Stahnke 180 Time Evaluation • Evaluate your team's time spent on “Displacement Activities” • Fund Raising • Parties/Showers • Volunteer Stuff • Charity • This is normally done because the person can't actually meet the requirements of $DAYJOB See Also: People
181. Red Hat Summit 2009 | Michael Stahnke 181 Pick a Task • Focus on one thing • Focus on one thing only • Seriously, this will help
182. Red Hat Summit 2009 | Michael Stahnke 182 You're ready to automate.
183. Red Hat Summit 2009 | Michael Stahnke 183 OK, maybe you're not ready to automate.
184. Red Hat Summit 2009 | Michael Stahnke 184 Don't automate bad process. http://www.flickr.com:80/photos/houseofcards/94935538/in/photostream/
185. Red Hat Summit 2009 | Michael Stahnke 185 You get bad results.....faster.
186. Red Hat Summit 2009 | Michael Stahnke 186 Bad processes deliver bad results.
187. Red Hat Summit 2009 | Michael Stahnke 187 You don't need a panflute.
188. Red Hat Summit 2009 | Michael Stahnke 188
189. Red Hat Summit 2009 | Michael Stahnke 189 It starts with requirements
190. Red Hat Summit 2009 | Michael Stahnke 190 Pick a Task • Let's start with system Deployment • You have some great options for tools • Do you need a tool? • Do you need it automated?
191. Red Hat Summit 2009 | Michael Stahnke 191 Pick a Task • Let's start with system Deployment • You have some great options for tools • Do you need a tool? • Do you need it automated? • Do you need a pan-flute?
192. Red Hat Summit 2009 | Michael Stahnke 192 Process Evaluation Who consumes the process?
193. Red Hat Summit 2009 | Michael Stahnke 193 Process Evaluation What are the outputs?
194. Red Hat Summit 2009 | Michael Stahnke 194 Process Evaluation What are the inputs?
195. Red Hat Summit 2009 | Michael Stahnke 195 Process Evaluation What's unknown?
196. Red Hat Summit 2009 | Michael Stahnke 196 Process Evaluation What's that rare case that you should just leave out?
197. Red Hat Summit 2009 | Michael Stahnke 197 Process Evaluation What's the gotcha you didn't put in because it was hard?
198. Red Hat Summit 2009 | Michael Stahnke 198 Gather Requirements Ask your process consumers about their needs
199. Red Hat Summit 2009 | Michael Stahnke 199 Gather Requirements Ask them to think about outputs, not the process
200. Red Hat Summit 2009 | Michael Stahnke 200 Gather Requirements How do they provide the information?
201. Red Hat Summit 2009 | Michael Stahnke 201 Requirements Formal Specification Document?
202. Red Hat Summit 2009 | Michael Stahnke 202 Requirements Formal Specification Document? • Awesome.
203. Red Hat Summit 2009 | Michael Stahnke 203 Requirements Formal Specification Document? • Awesome. • I doubt it
204. Red Hat Summit 2009 | Michael Stahnke 204 Requirements Hundreds of tickets, IM conversations, phone calls, hallway conversations and just plain old complaining?
205. Red Hat Summit 2009 | Michael Stahnke 205 Requirements Hundreds of tickets, IM conversations, phone calls, hallway conversations and just plain old complaining? • Yeah, I thought so.
206. Red Hat Summit 2009 | Michael Stahnke 206 Requirements Let's Get Agile.
207. Red Hat Summit 2009 | Michael Stahnke 207 Requirements Let's Get Agile. Development is agile.
208. Red Hat Summit 2009 | Michael Stahnke 208 Requirements Let's Get Agile. Development Infrastructure is agile.
209. Red Hat Summit 2009 | Michael Stahnke 209 Requirements Stories
210. Red Hat Summit 2009 | Michael Stahnke 210 Requirements Use Cases
211. Red Hat Summit 2009 | Michael Stahnke 211 Requirements Test Cases
212. Red Hat Summit 2009 | Michael Stahnke 212 Requirements Stores, Use Cases, Test Case --> Track them
213. Red Hat Summit 2009 | Michael Stahnke 213 Requirements Use a tracker,
214. Red Hat Summit 2009 | Michael Stahnke 214 Requirements Use a tracker, use a wiki,
215. Red Hat Summit 2009 | Michael Stahnke 215 Requirements Use a tracker, use a wiki, use a notebook,
216. Red Hat Summit 2009 | Michael Stahnke 216 Requirements Use a tracker, use a wiki, use a notebook, a whiteboard
217. Red Hat Summit 2009 | Michael Stahnke 217 Requirements http://www.flickr.com/photos/usonian/106005435/
218. Red Hat Summit 2009 | Michael Stahnke 218 Requirements Use a tracker, http://www.pivotaltracker.com http://pivotaltracker.com
219. Red Hat Summit 2009 | Michael Stahnke 219 Example Case • Application team needs an instance • 2 CPUs • 4 GB RAM • 100G Disk Space • RHEL 5 http://www.flickr.com:80/photos/jimfrazier/1480809458/
220. Red Hat Summit 2009 | Michael Stahnke 220 Requirements • Requirements Story: • An application team should be able to request capacity.
221. Red Hat Summit 2009 | Michael Stahnke 221 Requirements • Requirements Story: • An application team should be able to request capacity. • An application team should be able to see the status of their requests.
222. Red Hat Summit 2009 | Michael Stahnke 222 Requirements • Requirements Story: • An application team should be able to request capacity. • An application team should be able to see the status of their requests. • A capacity planner should approve the request
223. Red Hat Summit 2009 | Michael Stahnke 223 Requirements • Requirements Story: • An application team should be able to request capacity. • An application team should be able to see the status of their requests. • A capacity planner should approve the request • A system should have multipath from RHEL 5.3 installed
224. Red Hat Summit 2009 | Michael Stahnke 224 Requirements • Requirements Story: • An application team should be able to request capacity. • An application team should be able to see the status of their requests. • A capacity planner should approve the request • A system should have multipath from RHEL 5.3 installed • A physical system should have a bonded network connection
225. Red Hat Summit 2009 | Michael Stahnke 225 Design Review Stories
226. Red Hat Summit 2009 | Michael Stahnke 226 Design Look for common abstractions and workflows
227. Red Hat Summit 2009 | Michael Stahnke 227 Design Abstract the process as an interface.
228. Red Hat Summit 2009 | Michael Stahnke 228 Design Review the process
229. Red Hat Summit 2009 | Michael Stahnke 229 Design Track issues.
230. Red Hat Summit 2009 | Michael Stahnke 230 Design Do not automate JUNK
231. Red Hat Summit 2009 | Michael Stahnke 231 Design Break work into small units
232. Red Hat Summit 2009 | Michael Stahnke 232 Design Don't be afraid to say 'good enough for version 1'
233. Red Hat Summit 2009 | Michael Stahnke 233 Design Don't be afraid to say 'good enough for version 1' • Limit Scope.
234. Red Hat Summit 2009 | Michael Stahnke 234 Design Progress over Perfection
235. Red Hat Summit 2009 | Michael Stahnke 235 Design A good enough system today is better than a perfect one that hasn't been invented.
236. Red Hat Summit 2009 | Michael Stahnke 236 Design • You will iterate again. • You will iterate again.
237. Red Hat Summit 2009 | Michael Stahnke 237 Design What points of your process require measurement?
238. Red Hat Summit 2009 | Michael Stahnke 238 Design Can that be automated?
239. Red Hat Summit 2009 | Michael Stahnke 239 Design Can you test for it?
240. Red Hat Summit 2009 | Michael Stahnke 240 Design Your Tests Am I testing directly against my stories? (requirements)
241. Red Hat Summit 2009 | Michael Stahnke 241 Design Your Tests Does it have the right version of Multipath?
242. Red Hat Summit 2009 | Michael Stahnke 242 Design Your Tests Is the network connection bonded?
243. Red Hat Summit 2009 | Michael Stahnke 243 Testing
244. Red Hat Summit 2009 | Michael Stahnke 244 Testing Provides quality measurements
245. Red Hat Summit 2009 | Michael Stahnke 245 Testing Provides metrics for reporting.
246. Red Hat Summit 2009 | Michael Stahnke 246 Testing Your management team will love it
247. Red Hat Summit 2009 | Michael Stahnke 247 Testing You will have facts.
248. Red Hat Summit 2009 | Michael Stahnke 248 Testing You will have facts. Your team will love it.
249. Red Hat Summit 2009 | Michael Stahnke 249 Specifications for Testing Write Simple Tests
250. Red Hat Summit 2009 | Michael Stahnke 250 Specifications for Testing Unit Test your Configuration
251. Red Hat Summit 2009 | Michael Stahnke 251 Specifications for Testing Watch it FAIL.
252. Red Hat Summit 2009 | Michael Stahnke 252 Specifications for Testing Watch it FAIL. Good tests FAIL often.
253. Red Hat Summit 2009 | Michael Stahnke 253 Specifications for Testing http://www.flickr.com/photos/fireflythegreat/2845637227/
254. Red Hat Summit 2009 | Michael Stahnke 254 Specifications for Testing Write a policy/method/procedure/function to fix it
255. Red Hat Summit 2009 | Michael Stahnke 255 Specifications for Testing Watch it PASS
256. Red Hat Summit 2009 | Michael Stahnke 256 Specifications for Testing Store Results.
257. Red Hat Summit 2009 | Michael Stahnke 257 Test Case • Network configuration should be bonded.
258. Red Hat Summit 2009 | Michael Stahnke 258 Test Case • Network configuration should be bonded. • Can I script this?
259. Red Hat Summit 2009 | Michael Stahnke 259 Test Case • Network configuration should be bonded. • Can I script this? • Given enough Time and Money, it's all scriptable.
260. Red Hat Summit 2009 | Michael Stahnke 260 Test Case • Network configuration should be bonded. • Can I script this? • Given enough Time and Money, it's all scriptable. • Can I store the results?
261. Red Hat Summit 2009 | Michael Stahnke 261 Test Case • Network configuration should be bonded. • Can I script this? • Given enough Time and Money, it's all scriptable. • Can I store the results? • Yes
262. Red Hat Summit 2009 | Michael Stahnke 262 Test Case • Network configuration should be bonded. • Can I script this? • Given enough Time and Money, it's all scriptable. • Can I store the results? • Yes • How Often should I test this?
263. Red Hat Summit 2009 | Michael Stahnke 263 Test Case #!/bin/bash . config_vars.sh rc=0 # First, if we a VM exit all good if ( lspci | grep -i vmware &> /dev/null ) ; then rc=0 else if ( ! ifconfig -a | grep -i bond0 &> /dev/null ) ; then echo "FAIL: Network interfaces not bonded." rc=1 fi fi exit $rc
264. Red Hat Summit 2009 | Michael Stahnke 264 Test Case #!/bin/bash . config_vars.sh rc=0 # First, if we a VM exit all good if ( lspci | grep -i vmware &> /dev/null ) ; then rc=0 else if ( ! ifconfig -a | grep -i bond0 &> /dev/null ) ; then echo "FAIL: Network interfaces not bonded." rc=1 fi fi exit $rc Site Specific Data
265. Red Hat Summit 2009 | Michael Stahnke 265 Test Case #!/bin/bash . config_vars.sh rc=0 # First, if we a VM exit all good if ( lspci | grep -i vmware &> /dev/null ) ; then rc=0 else if ( ! ifconfig -a | grep -i bond0 &> /dev/null ) ; then echo "FAIL: Network interfaces not bonded." rc=1 fi fi exit $rc Site Specific Data Decoupled Logic
266. Red Hat Summit 2009 | Michael Stahnke 266 Test Case I have results, where do I put them?
267. Red Hat Summit 2009 | Michael Stahnke 267 Pick One
268. Red Hat Summit 2009 | Michael Stahnke 268 Example Storage Ruby using RHN API http://www.redhat.com/spacewalk
269. Red Hat Summit 2009 | Michael Stahnke 269 Implementation Run through the process a couple times manually
270. Red Hat Summit 2009 | Michael Stahnke 270 Implementation Think about re-use • Inventory Queries • Asking for/updating configuration info • Authentication Modules
271. Red Hat Summit 2009 | Michael Stahnke 271 Implementation Are existing tools close to what you need?
272. Red Hat Summit 2009 | Michael Stahnke 272 Implementation Can you reuse existing code?
273. Red Hat Summit 2009 | Michael Stahnke 273 Test Again After implementation, test again.
274. Red Hat Summit 2009 | Michael Stahnke 274 Test Again After implementation, test again. • It's automated right?
275. Red Hat Summit 2009 | Michael Stahnke 275 Test Again After implementation, test again. • It's automated right? • So it's quick and painless.
276. Red Hat Summit 2009 | Michael Stahnke 276 Close the loop Maintain state of your process
277. Red Hat Summit 2009 | Michael Stahnke 277 Close the loop Maintain state of your components
278. Red Hat Summit 2009 | Michael Stahnke 278 Close the loop If the version of package X changes, and you have a policy for package X, report.
279. Red Hat Summit 2009 | Michael Stahnke 279 Close the loop If the version of package X changes, and you have a policy for package X, report^H^H^H^H^H fix it.
280. Red Hat Summit 2009 | Michael Stahnke 280 Close the loop If don't have a network bond, report it or fix it.
281. Red Hat Summit 2009 | Michael Stahnke 281 Maintain Most fun step
282. Red Hat Summit 2009 | Michael Stahnke 282 Deployment Example • Host is RHEL 5.3 • Host has the correct Version of Multipath • Host should have 4 GB RAM • Host should have 2 CPUs • Host must have a network bond if it's physica
283. Red Hat Summit 2009 | Michael Stahnke 283 Maintain Offers most improvement
284. Red Hat Summit 2009 | Michael Stahnke 284 Maintain Customers offer ideas
285. Red Hat Summit 2009 | Michael Stahnke 285 Maintain Other admins complain (offer ideas)
286. Red Hat Summit 2009 | Michael Stahnke 286 Maintain Iterate
287. Red Hat Summit 2009 | Michael Stahnke 287 Maintain Lather, rinse, repeat.
288. Red Hat Summit 2009 | Michael Stahnke 288 CI You need CI.
289. Red Hat Summit 2009 | Michael Stahnke 289 Continuous Integration CI is used in the best development shops in the world
290. Red Hat Summit 2009 | Michael Stahnke 290 Continuous Integration Run upon commits
291. Red Hat Summit 2009 | Michael Stahnke 291 Continuous Integration You are using a VCS right?
292. Red Hat Summit 2009 | Michael Stahnke 292 Continuous Integration You are using a VCS right? • Please
293. Red Hat Summit 2009 | Michael Stahnke 293 Continuous Integration Track Failures
294. Red Hat Summit 2009 | Michael Stahnke 294 Continuous Integration Any new bugs, require Tests
295. Red Hat Summit 2009 | Michael Stahnke 295 Continuous Integration • Automate Builds • Self-test the Build • Commit early, commit often • Have a test environment • Report Progress for everybody (laconica?) • Automate Deployment – (usage packages)
296. Red Hat Summit 2009 | Michael Stahnke 296 Can you have a 'release' of infrastructure? • Yes • Plan change-sets • Plan your work-life balance • Work with setting expectations for consumers of your infrastructure • Work on hot/rolling releases • Communicate like a politician, except tell the truth
297. Red Hat Summit 2009 | Michael Stahnke 297 Some common areas for improvement A few things to avoid when crafting solutions.
298. Red Hat Summit 2009 | Michael Stahnke 298 Things to Avoid Meatcloud
299. Red Hat Summit 2009 | Michael Stahnke 299 Things to Avoid Automation of Junk
300. Red Hat Summit 2009 | Michael Stahnke 300 Things to Avoid Talking about IT rather than doing IT
301. Red Hat Summit 2009 | Michael Stahnke 301 Things to Avoid Thinking there is one solution
302. Red Hat Summit 2009 | Michael Stahnke 302 Things to Avoid UberTools – Silver Bullets
303. Red Hat Summit 2009 | Michael Stahnke 303 Things to Avoid New Hotness
304. Red Hat Summit 2009 | Michael Stahnke 304 Things to Avoid NIH Syndrome
305. Red Hat Summit 2009 | Michael Stahnke 305 Axiom 1 Reuse before building or purchasing
306. Red Hat Summit 2009 | Michael Stahnke 306 Axiom 2 Don't Leverage the Meatcloud
307. Red Hat Summit 2009 | Michael Stahnke 307 Axiom 3 Decouple your Infrastructure.
308. Red Hat Summit 2009 | Michael Stahnke 308 Go From This:
309. Red Hat Summit 2009 | Michael Stahnke 309 To This:
310. Red Hat Summit 2009 | Michael Stahnke 310
311. Red Hat Summit 2009 | Michael Stahnke 311
312. Red Hat Summit 2009 | Michael Stahnke 312 Sources • http://stochasticresonance.wordpress.com/2009/04/01/ meatcloud-manifesto/ • http://www.boingboing.net/2007/08/22/panflute-flowchar t.html • http://gilesbowkett.blogspot.com/2007_05_01_archive. html • http://www.verber.com/mark/sysadm/how-many-admins .html
313. Red Hat Summit 2009 | Michael Stahnke 313 Suggested Reading • http://meetronome.com • http://gist.github.com/161265 • https://fedoraproject.org/wiki/Infrastructure • https://fedorahosted.org/csi/ • Ship It – Pragmatic Press • Automating Unix and Linux Administration – Apress • Pro Unix Administration – Apress • Pulling Strings with Puppet - Apress • Release IT – Pragmatic Press
314. Red Hat Summit 2009 | Michael Stahnke 314 More Reading • http://groups.google.com/group/agile-system-administra tion • http://www.melconway.com/research/committees.html • http://status.net/?source=laconica • http://reductivelabs.com/trac/puppet/ • https://fedorahosted.org/cobbler/ • http://www.mockobjects.com/book/ •
315. Red Hat Summit 2009 | Michael Stahnke 315 Infrastructure is Development