Practical Methods for Adopting Devops

Presentation I gave to the Spiceworks community on Devops.

Transcript

1. Practical Methods for Adopting DevOps Michael Stahnke, Director of Engineering @stahnma
2. • Introduction • DevOps Overview • When It’s Bad • Making It Better – 5 Methods • Questions Next Week & PuppetConf Overview
3. Next Week & PuppetConfIntroduction • Server architecture • Infrastructure automation • Release management
4. It’s impossible to exceed expectations.
5. Nobody ever got a high-five for a server staying up.
6. Next Week & PuppetConf DevOps Culture • Lots of press, talks, hype • Echo chamber • Does it apply to all of us? • 10 deploys/day • Rockstar ninja pirates • Write your own tools • Awesome app developers • Startups
8. Next Week & PuppetConf Setting the Stage • Large company size with competing priorities • IT teams that don’t sit in the same building • Average time in role: 2-3 years • No dev … “application owners” • Everybody really “nice”
9. Next Week & PuppetConf Setting the Stage • Call the vendor for problems • Customize for us, our needs are unique • Success means everything stays the same • Just looking for approvals • No time to debate
10. Next Week & PuppetConf Setting the Stage • Enterprise-y: Tivoli, BMC, CA, IBM • Apps: Oracle RAC, Oracle EBS, Websphere AS, MQ Series, Weblogic, Legacy stuff • HP-UX, Solaris, AIX, RHEL, VMware Tools
11. Next Week & PuppetConf Infrastructure Team • Small team – 4 of you quit; that’s 50% of the team • Information is tribal – What happens when the tribe leaves? • 16 hour days for some admins • Server team blamed for EVERYTHING This Silo? What Silo?
12. Next Week & PuppetConf Infrastructure Team • No transfers • Windows team doesn’t work with Unix team, who doesn’t talk to Storage team • Exiled to different buildings This Silo? What Silo?
13. Next Week & PuppetConf Is it all bad? • Industry experience • Be the best sysadmin • Open Source tooling • LDAP Motivations
15. Next Week & PuppetConf Databases Rule the World • Variance • 25 db servers • 24 unique configurations • At least 6 cluster pairs
18. Reduce the standard deviation value. Then raise the mean.
19. Next Week & PuppetConf Method 1 Reduce variability.
20. Do you want systems to have planned or unplanned downtime? Because you’re going to get one or the other.
21. Next Week & PuppetConf Results • Failing in known ways • Better uptime • Fewer configurations • Lots of tickets on disk management
22. Collaboration?
23. Next Week & PuppetConf Collaboration • Sudo for disk mgmt • Shared responsibility for outages • Proper classification (production, test, dev) • If I’m awake, you’re awake
24. Next Week & PuppetConf Collaboration • Share the pain • People who carry the pager will make better design decisions when not on-call
25. Next Week & PuppetConf Method 2 Stop. Collaborate. Listen. (Break down silos)
27. Next Week & PuppetConf Root Cause Analysis Meetings • What’s the root cause of the outage? • What’s the exact impact? • What do we do differently next time?
29. Next Week & PuppetConf Method 3 Shout your failures. (Honesty builds credibility)
30. Next Week & PuppetConf Root Cause Analysis Meetings • Shout out your failure • Hand out a written report & timeline of what happened • Remediation plan • Meeting lasts 10 minutes
31. Next Week & PuppetConf Setbacks • Everything isn’t better overnight • Time passes, new issues • Lots of failures • Config Mgmt messes Don’t expect perfection
32. Next Week & PuppetConf Experiments • What can you try while you’re fixing process? • Scrum teams • SDLC • Inventory service • Puppet
33. Next Week & PuppetConf Method 4 Experimentation matters. (Failures are still valid data)
34. Next Week & PuppetConf Periodic Review • Find biggest pain points – optimize • Improving anything that isn’t the biggest bottleneck in a process is a wasted effort! • Drive value up the stack • Find metrics people understand • (Use Puppet)
35. The cost of retiring any technology is much higher than introducing it.
36. Next Week & PuppetConf Learning • Python, Ruby, Puppet, git, REST, APIs, Web Services, Go, Containers • Retrospectives • Daily standups • Lead by example
37. Are we satisfied with how this works, or are we trying to make it better?
38. Next Week & PuppetConf Improvements • Influence other teams • Automation help • Monitoring • Design • Requests to transfer to your team
39. Next Week & PuppetConf Method 5 Solve causes not symptoms. (Continuous improvement)
40. • Reduce variability. • Stop. Collaborate. Listen. • Shout your failures. • Experimentation matters. • Solve causes not symptoms. Next Week & PuppetConf Recap