$30 off During Our Annual Pro Sale. View Details »

The Dark Side of DevOps

Chris Short
September 08, 2017

The Dark Side of DevOps

The DevOps journey is a combination of people, processes, and tooling. The culmination of these is a team of allies pushing the envelope and never resting on their laurels. There is no such thing as a completed DevOps transition. Resting on your laurels is not a DevOps mindset. DevOps and its allies should be iterating and improving upon what they have learned daily.

Chris Short

September 08, 2017
Tweet

More Decks by Chris Short

Other Decks in Programming

Transcript

  1. THE DARK SIDE OF DEVOPS DEVOPSDAYS RALEIGH @ChrisShort devopsish.com

  2. INTRODUCTION chrisshort.net @ChrisShort devopsish.com • Hello! HAPPY, LOUD, GOOD MORNING

    • A quick intro about me • I've worked in IT for over twenty years now • I've worked in DevOps capacities since 2011 • I'm currently at working at bankrate.com in Detroit, MI • There's a new project I'm participating in with opensource.com called DevOps Stories • It's about the intersection of open source and DevOps • If you're interested in helping out head over to opensource.com/devops-team
  3. • True Story: Last year I lived in Raleigh, attended

    DevOpsDays here. But, I spoke at DevOpsDays Detroit. • This year, I live in Detroit, will attend DevOpsDays there and am speaking here in Raleigh. • The moral of that story is: be careful where you speak at your first DevOpsDays, you might end up living there. • But, Don't Knock Detroit • Blight is so 2000s • These are just a few pictures I've taken of Detroit since moving there • The city is going through a massive rebirth and it's amazing • If you are ever in town let me know, I'll give you a tour • On to the DevOps part of this talk
  4. • One thing I love about DevOps is how failure

    is embraced • I love failure because it teaches us things we never would know about ourselves • Many people present DevOps as this great thing filled with • Roses • Rainbows • Unicorns • Maybe even a random Leprechaun • Sometimes it's nice to remind ourselves that DevOps has a dark side to it
  5. • The Dark Side of DevOps is Complete Abject Failure

    • Failed DevOps Transformations Are a Real Thing • Organizations try to move their organization towards The Three Ways and fail • It happens; failure is a part of life • Let's Explore a Couple of Examples from My Own DevOps Journey
  6. DISCLAIMER @ChrisShort devopsish.com CSHORT.CO/DARKOPS • Let me drop this quick

    disclaimer • This talk was adapted from an article I wrote in March: http://cshort.co/darkops • These are real world experiences I have had within organizations that I have been employed by • Identifying information is intentionally anonymous • If you think you are in one of these organizations please consider keeping that to yourself for the sake of others • Or don't if you're the rebellious type • Ready? Let’s walk into the cave on Dagobah together.
  7. THE OPS MUTINY @ChrisShort devopsish.com When I say "abject failure",

    this is what I'm referring to
  8. THE OPS MUTINY: BACKSTORY ▸ Dev teams adopted DevOps practices

    ▸ Traditional IT team was not adapting ▸ IT team was actively resisting DevOps ▸ Mutiny!!! Majority of IT team left company en masse ▸ Tribal knowledge left with remaining two team members @ChrisShort devopsish.com • Dev teams adopted DevOps practices for flagship product • The traditional IT team was not adapting • IT team was actively resisting the transition towards a DevOps workflow • There was a mutiny and the majority of the IT team left the company en masse • Tribal knowledge rested with two remaining team members
  9. @ChrisShort devopsish.com Qatar, 2002 • This is where I came

    into this sordid affair (in the twenty-teens) • I joined on as an operations lead to push the team into the future • A younger, less experienced, and slightly mislead Chris Short • LOOK AT SCREEN • "Well maybe not that young" • I should've used young Ewan McGregor but... Ya know... Episode 1
  10. @ChrisShort devopsish.com • Week One was a sign of how

    bad this was going to be • ZERO formal processes and on boarding was terrible • Changed desks halfway through the second day • I had a loaner laptop and never had a work phone number while I worked there • Spent my first week working on compliance training I did not need to do • Did not have access to what I needed to complete the goals laid out for me
  11. • Then the architecture review happened... • Company had been

    acquired for a HUGE sum of cash • The architecture review was not for me, the new lead, it was for the company that had bought them • I learned the office server room with faulty HVAC held some production functions • Learned that Windows NT 4.0 was still present in their infrastructure in 2015 • Changes could not happen due to business concerns • Legal coverage was good enough for the corporate lawyers • No data stored on archaic OS versions; this was pretty much a lie
  12. @ChrisShort devopsish.com • Then I started poking around the network

    • Was immediately told to stop • IT was concerned about the load I was putting on the network (from my laptop) • The "concern" was bogus and it was actually to keep the old guard in a position of power • The mutiny was on-going as the lingering staff continued to consolidate power
  13. • Ambitious plan underway to move the flagship project over

    to a cloud provider and use containers • Majority of that work was being handled by a third party integrator • Plan was for the vendor to integrate and stand up the new infrastructure • Team would then take over operations • Company was relying more and more on the vendor when the opposite was the desired outcome • It was becoming clear the team was going to continue their resistance and I had no allies
  14. @ChrisShort devopsish.com • Shortly before a major US holiday, I

    determined enough was enough • After three weeks, I was putting in my two weeks notice • The anti-DevOps mutiny was successful • The leadership tried very hard to get me to stay on • Leadership offered to terminate the remaining holdouts days before a major holiday • Not okay with morally bankrupting myself in the name of DevOps
  15. THE DEV REBELLION @ChrisShort devopsish.com • I had high hopes

    for my next opportunity • Fresh out of one DevOps failure I unintentionally jumped into another one
  16. @ChrisShort devopsish.com • A full blown DevOps rebellion against the

    monolithic IT department was planned • DevOps was brought to bear within a small team working on an open source project • I had high hopes for this opportunity
  17. @ChrisShort devopsish.com Team was already using cloud native tools because

    existing tooling was inadequate: • Docker • Papertrail • Postgres • Slack for communication
  18. @ChrisShort devopsish.com • CTO said cloud was prohibited; everything had

    to stay in house • Weeks or months to get a VM; DNS change requests took days or weeks • Team was going to utilize Heroku despite the IT organization stating “cloud” was prohibited • Shortly before my arrival they had done all the appropriate paperwork to deploy this new project • Assumption from the IT staff and paper pushers was that this new project was deploying in-house • All the documentation clearly stated Heroku was the workload’s destination • CRAFTY and I loved it! • It was so gangster
  19. • IT department approval authorities signed off on utilizing cloud

    resources despite saying they never would • They just signed off on a full blown shadow IT project and it was right there in their own documents • This proved their ITIL and ITSM death grip process was not meeting their desired objectives • The cloud cat was out of the bag
  20. @ChrisShort devopsish.com • IT attempted to change and satisfy the

    team's needs in-house • The IT process was a strict, by the book adherence to ITIL • It would never be able to shoehorn itself into a speedier process until the need for hand jamming tickets was eliminated • OpenStack instance as a developer playground was going to take a year or more • Tooling to enable API calls for IT resources was outsourced to a vendor with little feedback from our team • IT did not get it but the team was working around the bottlenecks • It was working for the team but my job of trying to get the IT org looking towards the future was next to impossible
  21. • Resistance and educating the IT organization were critical components

    of helping them on their path towards transformation • The alternative was to join in the IT organization's thinking which as stifling innovation • Join them and you can ruin the galaxy together!
  22. • The rebellion was in full force • Microservices were

    being built and deployed • IT organization had no way of keeping up with the multiple releases a day • Change occurred when IT organization realized it was lagging way behind • IT organization realized they couldn't compete on costs • Do to a host of other issues I decided to leave this position for another opportunity • Eventually compliant Azure resources were made available to the teams • Last I heard it was still taking months to get a VM in their datacenter
  23. DEVOPS STRIKES BACK (LESSONS LEARNED) @ChrisShort devopsish.com What did I

    learn from these two journeys? Aside from changing jobs is easy in Raleigh/Durham
  24. "OUR GREATEST GLORY IS NOT IN NEVER FAILING, BUT IN

    RISING UP EVERY TIME WE FAIL." Ralph Waldo Emerson EMBRACE FAILURE, LEARN FROM IT
  25. @ChrisShort devopsish.com • A mutiny and a rebellion might look

    the same from opposing perspectives • The rebellion was to change for the better while the mutiny was to maintain the status quo • This is not to say Ops is bad and Devs are great (far from it) • These jobs along my journey proves that motivated teams can act towards their common goal be it good or bad
  26. @ChrisShort devopsish.com • I have learned that DevOps needs allies

    at all levels • I wish I would have learned this earlier in my career • Not just leadership • Not just individual contributors • People have to want to better their environments
  27. @ChrisShort devopsish.com • The Dev Rebellion had people at all

    levels pointing at successes of the team • The Ops Mutiny had a core group of people trying to cast off change regardless of outcomes • But both were personal failures since I wasn't able to rally enough support in enough time to keep me from going mad
  28. ⚙ ⚙ ⚙ @ChrisShort devopsish.com • In Conclusion... • The

    DevOps journey is a combination of people, processes, and tooling • The culmination of these is a team of allies pushing the envelope • Never resting on their laurels • There is no such thing as a completed DevOps transition • Resting on your laurels is not a DevOps mindset • DevOps and its allies should be iterating and improving upon what they have learned daily
  29. @ChrisShort devopsish.com • Thank you for your time today •

    Feel free to reach out to me on Twitter if you ever need another opinion or a nudge in a direction • Take care of each other out there