Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mentoring Devs Into DevOps - OpenWest

Mentoring Devs Into DevOps - OpenWest

Lets set aside the buzzword for a moment and have an honest discussion about DevOps. There is the idea of putting more Dev into Ops, but just a crucial (if not more crucial) is getting your Devs to think more like Ops. We'll discuss how to effectively help developers learn more about Ops and help them be a positive contribution to your DevOps efforts.

Justin Carmony

May 09, 2014

More Decks by Justin Carmony

Other Decks in Technology


  1. Lets Measure The Audience • Who here is a… •

    System Administrator? • Developer? • Manager / Management? • “DevOp?”
  2. About Me • Director of Development
 for Deseret Digital Media

    • Utah PHP Usergroup
 President • I Make (and Break) 
 Web Stuff (10 years) • Salt User in Production since 0.8 (I <3 Salt)
  3. This Presentation • Lessons learned at DDM & previous jobs

    • Insight into our process of increasing “DevOps” • We’re still learning, but this what we’ve found. • Slides will be posted online, so don’t worry about copying slide content. • Feel free to ask on-topic questions during, and we’ll have questions at the end.
  4. About DDM • Deseret Digital Media runs local website like

    KSL.com, DeseretNews.com • Running National and International Websites like OK.com, familia.com.br, etc. • ~10 million pageviews a day across sites. • ~150 internal VMs, a few dozen physical machines, some AWS sprinkled around.
  5. Challenges We Faced • Giant mesh-up of technologies • Tightly-coupled

    & fragile infrastructure • Debugging production only bugs was difficult • Bugs that were part code, part environment were a nightmare to track down.
  6. Step 1: Hire a DevOp! Step 2: ????????????! Step 3:

    Profit! Everything Works ! Perfectly!
  7. What does DevOps Mean To Us? • DevOps: Dev &

    Ops, a Culture of Collaboration • Our Goal: “10 deploys a day without issues” • Everyone shares the goal of quick development of features AND a stable system that stays up.
  8. Team Structure Devs: 30 Ops: 2 DevOps: 1 Hiring one

    person won’t just solve all our problems!
  9. Team Realizations • Hardest problem already solved: awesome team •

    No foreseeable rapid expansion, must operate at our current scale • Each Project’s Director of Development was acting as the bridge between Dev and Ops, but would become a bottleneck.
  10. Teams Already Had Some Ad-Hoc DevOps Tools - Real-time Logging

    - Capistrano Deploys - Nagios Alerts - Server Metrics - Puppet for File Mgmt - App Stats w/ Graphite - Graphite Dashboards - Salt for Cfg Management - Homebrewed Metrics Sys. - Homebrewed Alert System
  11. Step 1: Hire a DevOp! Step 2: ????????????! Step 3:

    Profit! Everything Works ! Perfectly!
  12. DevOp Engineer • Well Defined Role: • Ownership over the

    TOOLS to improve DevOps efforts. • Resource for other teams to help use DevOps Tools. • Easy to work with, aptitude for systems & ops, likes to try new things.
  13. Promoting From Within • A seasoned dev for your team

    already knows: • Your Pain Points • Your System’s Quirks • How the “Chaos Works” • Knows the people & personalities on your team
  14. Increasing Ops Among Devs • Identify Devs who liked “Ops”

    & wanted to Learn • Pair Dev with Op / Director • Learning Dev works on things, not Op /Director. • Pair program if needed.
  15. Metrics • Everyone has access to Network, Server, and Application

    Metrics. • Consolidate & reduce places to look. We try to pipe everything to StatsD / Graphite • Each developer trained to add & track metrics in production. • We’re okay with 98% uptime of stats to avoid complexity.
  16. Real-Time Logging • Harder & more complicated at scale •

    Still trying to solve well, we have lots of logs. • Start with small window of data (i.e. 48 hours) and start to expand window. • We’re trying Logstash, ElasticSearch, and Kibana right now. • Generate Statistics off our Logs
  17. Tracking Changes • Everything, everything, everything in git 

    use GitHub) • Everyone has access to all repos • Everyone does work through Pull Requests • Everyone has their work code reviewed * * - Your can merge w/o a review, but must be willing to defend your choice
  18. Everyone Can Deploy • Automated our deployment process to a

    single step. • Everyone can deploy, deployments are logged • Easy rollback is a requirement! • Implementing feature flags to turn off single parts of our application.
  19. Automated Tests • If you want to trust your Devs,

    you need tests • Legacy apps we wrote Integration Tests • New Apps & Refactored Legacy Parts have Unit Tests • Continuous Integration to make sure tests run
  20. Safe Environment For Devs to Learn salt \* cmd.run "rm

    -rf /tmp /*" Salt is awesome, but it can’t ! recover from that
  21. Dev Salt Master • Every server has two minions: •

    Admin Salt (aka root) • Dev Salt (aka bob) • Each connect to different master server: • All Devs have access to Dev Salt Master • Trusted Devs get access to Admin Salt Master
  22. Dev Environment • Developers own the Dev Environment • Dev

    Teams manage the Salt States for their Env • Vagrant + Salt for their Env • Who makes changes? Developers • DevOp helps advise & offer support
  23. Stage Environment • Stage & Production use same salt repos,

    different branches • Developers make all the changes for Application Servers • All Changes through Pull Requests • We’ll worry about env changes before code • Small changes we quickly release, large or long running branches are scary & dangerous
  24. Production Environment • Merge change to Production Branch • salt

    \* state.highstate • Reminder: Small quick changes over time, never a large change at once.
  25. Environment Caveats • Ops & DevOps Manage VM Hosts, Physical

    Load Balancers, FireWalls, etc • Ops & DevOps manage servers that deal with data: • MySQL • MongoDB • etc
  26. Level of “DevOps” Skills • Thinks about their impact on

    Ops: Everyone • Able to debug issues with production: Most • Able to make changes to environments: Many • “Awesome DevOp”: Some
  27. Where We Are At • All Dev Environments using Vagrant

    + Salt • All New Stage & Prod Environments are Salty • Some Legacy Stage & Production Envs are Salty • Continuously working on getting out stuff salty.
  28. Honest Introspection • Determine for your team what are your…

    • Strengths • Weaknesses • Problems • Goals
  29. Increase Team’s Insight • Make sure devs can see &

    understand how their code performs • Increase responsibility of team for those metrics. • If they break it, they fix it. 
 Do not always bail them out. • Everyone can see everything.
  30. Mentor Those With Desire / Aptitude • Give Developers Safe

    Environment to Learn • Let them submit code-reviewed changes for Stage & Production • When teaching / mentoring, let the learner drive, kindly offer advice and help. • It takes time, but worth the investment.