Mentoring Devs Into DevOps - OpenWest

Lets set aside the buzzword for a moment and have an honest discussion about DevOps. There is the idea of putting more Dev into Ops, but just a crucial (if not more crucial) is getting your Devs to think more like Ops. We'll discuss how to effectively help developers learn more about Ops and help them be a positive contribution to your DevOps efforts.

1. Mentoring Devs Into DevOps Justin Carmony Director of Development Deseret Digital Media
3. Business Track
4. Lets Measure The Audience • Who here is a… • System Administrator? • Developer? • Manager / Management? • “DevOp?”
5. Confession: I’m a Developer
7. Self-Taught Ops Because There Was No One Else To Do It
8. About Me • Director of Development
 for Deseret Digital Media • Utah PHP Usergroup
 President • I Make (and Break) 
 Web Stuff (10 years) • Salt User in Production since 0.8 (I <3 Salt)
9. This Presentation • Lessons learned at DDM & previous jobs • Insight into our process of increasing “DevOps” • We’re still learning, but this what we’ve found. • Slides will be posted online, so don’t worry about copying slide content. • Feel free to ask on-topic questions during, and we’ll have questions at the end.
10. About DDM • Deseret Digital Media runs local website like KSL.com, DeseretNews.com • Running National and International Websites like OK.com, familia.com.br, etc. • ~10 million pageviews a day across sites. • ~150 internal VMs, a few dozen physical machines, some AWS sprinkled around.
11. Lets Start With a Story!
12. You Work for an Awesome Tech Company
13. Team Is Working Hard to Build New Things!
14. You launch your awesome product!
15. A Few More Features…
16. … and next thing you know…
18. Awesome Job Team, We Rock!
19. We Need ! Real-Time XYZ Feature! ASAP! &#$%!
21. “Huh, it works if you ! just turn off caching…”! - Dev @ 80th Hour This Week
22. “I’m sure this ! will work…”
26. “Our servers are melting!”
34. “We Need a Better Solution!”
35. So…
36. Where Do We Start?
38. We Have This Problem
39. Challenges We Faced • Giant mesh-up of technologies • Tightly-coupled & fragile infrastructure • Debugging production only bugs was difficult • Bugs that were part code, part environment were a nightmare to track down.
41. So One Day… We Had A Genius Idea!
42. Lets Hire a DevOp!
43. I’m Not Joking We Actually Said This
44. Two Problems with this “Idea”
45. Problem #1 - We Didn’t Understand What We Really Wanted
46. Step 1: Hire a DevOp! Step 2: ????????????! Step 3: Profit! Everything Works ! Perfectly!
47. Problem #2 - People Who Are Great At Dev & Ops Are Hard To Find
48. Expectation:
49. Reality:
50. Honest Team Discussion: What is it we’re really looking for?
51. We Discovered a Few Things
52. What does DevOps Mean To Us? • DevOps: Dev & Ops, a Culture of Collaboration • Our Goal: “10 deploys a day without issues” • Everyone shares the goal of quick development of features AND a stable system that stays up.
53. Team Structure Devs: 30 Ops: 2 DevOps: 1 Hiring one person won’t just solve all our problems!
54. Team Realizations • Hardest problem already solved: awesome team • No foreseeable rapid expansion, must operate at our current scale • Each Project’s Director of Development was acting as the bridge between Dev and Ops, but would become a bottleneck.
55. Teams Already Had Some Ad-Hoc DevOps Tools - Real-time Logging - Capistrano Deploys - Nagios Alerts - Server Metrics - Puppet for File Mgmt - App Stats w/ Graphite - Graphite Dashboards - Salt for Cfg Management - Homebrewed Metrics Sys. - Homebrewed Alert System
56. Step 1: Hire a DevOp! Step 2: ????????????! Step 3: Profit! Everything Works ! Perfectly!
57. We Formed A Strategy
58. Step #1: Promote Dev to DevOp Role
59. WAIT! Isn’t that the advice you just said was a bad idea?!
60. DevOp Engineer • Well Defined Role: • Ownership over the TOOLS to improve DevOps efforts. • Resource for other teams to help use DevOps Tools. • Easy to work with, aptitude for systems & ops, likes to try new things.
61. Promoting From Within • A seasoned dev for your team already knows: • Your Pain Points • Your System’s Quirks • How the “Chaos Works” • Knows the people & personalities on your team
62. Step #2: Change Team Structure
63. Team Structure Devs: 30 Ops: 2
64. Team Structure Goal: Spread Out Expertise By Increasing Ops Experience & Skills Among Devs Dev Ops
65. Team Structure Dev Ops
66. Increasing Ops Among Devs • Identify Devs who liked “Ops” & wanted to Learn • Pair Dev with Op / Director • Learning Dev works on things, not Op /Director. • Pair program if needed.
67. Step #3: Increase Everyone’s Insight
69. Metrics • Everyone has access to Network, Server, and Application Metrics. • Consolidate & reduce places to look. We try to pipe everything to StatsD / Graphite • Each developer trained to add & track metrics in production. • We’re okay with 98% uptime of stats to avoid complexity.
71. Real-Time Logging
72. Real-Time Logging • Harder & more complicated at scale • Still trying to solve well, we have lots of logs. • Start with small window of data (i.e. 48 hours) and start to expand window. • We’re trying Logstash, ElasticSearch, and Kibana right now. • Generate Statistics off our Logs
73. Tracking Changes • Everything, everything, everything in git 
 (we use GitHub) • Everyone has access to all repos • Everyone does work through Pull Requests • Everyone has their work code reviewed * * - Your can merge w/o a review, but must be willing to defend your choice
74. Deploys
75. Everyone Can Deploy • Automated our deployment process to a single step. • Everyone can deploy, deployments are logged • Easy rollback is a requirement! • Implementing feature flags to turn off single parts of our application.
76. Tests Unit Functional Integration Acceptance etc
77. Automated Tests • If you want to trust your Devs, you need tests • Legacy apps we wrote Integration Tests • New Apps & Refactored Legacy Parts have Unit Tests • Continuous Integration to make sure tests run
78. Step #4: Devs Use The Ops Tools
79. Devs can grok salt
80. Safe Environment For Devs to Learn salt \* cmd.run "rm -rf /tmp /*" Salt is awesome, but it can’t ! recover from that
81. Dev Salt Master Devs Can Look Into Every Server
82. Dev Salt Master • Every server has two minions: • Admin Salt (aka root) • Dev Salt (aka bob) • Each connect to different master server: • All Devs have access to Dev Salt Master • Trusted Devs get access to Admin Salt Master
83. Everything Salty in Git Reminder:
84. Dev Environment • Developers own the Dev Environment • Dev Teams manage the Salt States for their Env • Vagrant + Salt for their Env • Who makes changes? Developers • DevOp helps advise & offer support
85. Team Structure Dev Ops
86. Stage Environment • Stage & Production use same salt repos, different branches • Developers make all the changes for Application Servers • All Changes through Pull Requests • We’ll worry about env changes before code • Small changes we quickly release, large or long running branches are scary & dangerous
87. Production Environment • Merge change to Production Branch • salt \* state.highstate • Reminder: Small quick changes over time, never a large change at once.
88. Environment Caveats • Ops & DevOps Manage VM Hosts, Physical Load Balancers, FireWalls, etc • Ops & DevOps manage servers that deal with data: • MySQL • MongoDB • etc
89. Mentoring Devs
90. Mentoring Devs • Not every Dev will become an amazing DevOp • Thats okay!
91. Level of “DevOps” Skills • Thinks about their impact on Ops: Everyone • Able to debug issues with production: Most • Able to make changes to environments: Many • “Awesome DevOp”: Some
92. So Everything Is Awesome for us, right?
93. Honesty Slide: We Have Skeletons In Our Closets
94. Where We Are At • All Dev Environments using Vagrant + Salt • All New Stage & Prod Environments are Salty • Some Legacy Stage & Production Envs are Salty • Continuously working on getting out stuff salty.
95. Making This Work For Your Team
96. Honest Introspection • Determine for your team what are your… • Strengths • Weaknesses • Problems • Goals
97. Increase Team’s Insight • Make sure devs can see & understand how their code performs • Increase responsibility of team for those metrics. • If they break it, they fix it. 
 Do not always bail them out. • Everyone can see everything.
98. Mentor Those With Desire / Aptitude • Give Developers Safe Environment to Learn • Let them submit code-reviewed changes for Stage & Production • When teaching / mentoring, let the learner drive, kindly offer advice and help. • It takes time, but worth the investment.
99. A Few Final Thoughts
100. Team Culture ! Matters
101. Positive Influence
102. Questions?
103. Thank You Justin Carmony Email: justin@justincarmony.com Twitter: @JustinCarmony IRC: carmony #salt #uphpu Website: justin@justincarmony.com
104. p.s. we’re hiring, email / pm / tweet me