THE DARK SIDE OF DEVOPS DEVOPSDAYS RALEIGH @ChrisShort devopsish.com

INTRODUCTION chrisshort.net @ChrisShort devopsish.com • Hello! HAPPY, LOUD, GOOD MORNING • A quick intro about me • I've worked in IT for over twenty years now • I've worked in DevOps capacities since 2011 • I'm currently at working at bankrate.com in Detroit, MI • There's a new project I'm participating in with opensource.com called DevOps Stories • It's about the intersection of open source and DevOps • If you're interested in helping out head over to opensource.com/devops-team

• True Story: Last year I lived in Raleigh, attended DevOpsDays here. But, I spoke at DevOpsDays Detroit. • This year, I live in Detroit, will attend DevOpsDays there and am speaking here in Raleigh. • The moral of that story is: be careful where you speak at your first DevOpsDays, you might end up living there. • But, Don't Knock Detroit • Blight is so 2000s • These are just a few pictures I've taken of Detroit since moving there • The city is going through a massive rebirth and it's amazing • If you are ever in town let me know, I'll give you a tour • On to the DevOps part of this talk

• One thing I love about DevOps is how failure is embraced • I love failure because it teaches us things we never would know about ourselves • Many people present DevOps as this great thing filled with • Roses • Rainbows • Unicorns • Maybe even a random Leprechaun • Sometimes it's nice to remind ourselves that DevOps has a dark side to it

• The Dark Side of DevOps is Complete Abject Failure • Failed DevOps Transformations Are a Real Thing • Organizations try to move their organization towards The Three Ways and fail • It happens; failure is a part of life • Let's Explore a Couple of Examples from My Own DevOps Journey

DISCLAIMER @ChrisShort devopsish.com CSHORT.CO/DARKOPS • Let me drop this quick disclaimer • This talk was adapted from an article I wrote in March: http://cshort.co/darkops • These are real world experiences I have had within organizations that I have been employed by • Identifying information is intentionally anonymous • If you think you are in one of these organizations please consider keeping that to yourself for the sake of others • Or don't if you're the rebellious type • Ready? Let’s walk into the cave on Dagobah together.

THE OPS MUTINY @ChrisShort devopsish.com When I say "abject failure", this is what I'm referring to

THE OPS MUTINY: BACKSTORY ▸ Dev teams adopted DevOps practices ▸ Traditional IT team was not adapting ▸ IT team was actively resisting DevOps ▸ Mutiny!!! Majority of IT team left company en masse ▸ Tribal knowledge left with remaining two team members @ChrisShort devopsish.com • Dev teams adopted DevOps practices for flagship product • The traditional IT team was not adapting • IT team was actively resisting the transition towards a DevOps workflow • There was a mutiny and the majority of the IT team left the company en masse • Tribal knowledge rested with two remaining team members

@ChrisShort devopsish.com Qatar, 2002 • This is where I came into this sordid affair (in the twenty-teens) • I joined on as an operations lead to push the team into the future • A younger, less experienced, and slightly mislead Chris Short • LOOK AT SCREEN • "Well maybe not that young" • I should've used young Ewan McGregor but... Ya know... Episode 1

@ChrisShort devopsish.com • Week One was a sign of how bad this was going to be • ZERO formal processes and on boarding was terrible • Changed desks halfway through the second day • I had a loaner laptop and never had a work phone number while I worked there • Spent my first week working on compliance training I did not need to do • Did not have access to what I needed to complete the goals laid out for me

• Then the architecture review happened... • Company had been acquired for a HUGE sum of cash • The architecture review was not for me, the new lead, it was for the company that had bought them • I learned the office server room with faulty HVAC held some production functions • Learned that Windows NT 4.0 was still present in their infrastructure in 2015 • Changes could not happen due to business concerns • Legal coverage was good enough for the corporate lawyers • No data stored on archaic OS versions; this was pretty much a lie

@ChrisShort devopsish.com • Then I started poking around the network • Was immediately told to stop • IT was concerned about the load I was putting on the network (from my laptop) • The "concern" was bogus and it was actually to keep the old guard in a position of power • The mutiny was on-going as the lingering staff continued to consolidate power

• Ambitious plan underway to move the flagship project over to a cloud provider and use containers • Majority of that work was being handled by a third party integrator • Plan was for the vendor to integrate and stand up the new infrastructure • Team would then take over operations • Company was relying more and more on the vendor when the opposite was the desired outcome • It was becoming clear the team was going to continue their resistance and I had no allies

@ChrisShort devopsish.com • Shortly before a major US holiday, I determined enough was enough • After three weeks, I was putting in my two weeks notice • The anti-DevOps mutiny was successful • The leadership tried very hard to get me to stay on • Leadership offered to terminate the remaining holdouts days before a major holiday • Not okay with morally bankrupting myself in the name of DevOps

THE DEV REBELLION @ChrisShort devopsish.com • I had high hopes for my next opportunity • Fresh out of one DevOps failure I unintentionally jumped into another one

@ChrisShort devopsish.com • A full blown DevOps rebellion against the monolithic IT department was planned • DevOps was brought to bear within a small team working on an open source project • I had high hopes for this opportunity

@ChrisShort devopsish.com Team was already using cloud native tools because existing tooling was inadequate: • Docker • Papertrail • Postgres • Slack for communication

@ChrisShort devopsish.com • CTO said cloud was prohibited; everything had to stay in house • Weeks or months to get a VM; DNS change requests took days or weeks • Team was going to utilize Heroku despite the IT organization stating “cloud” was prohibited • Shortly before my arrival they had done all the appropriate paperwork to deploy this new project • Assumption from the IT staff and paper pushers was that this new project was deploying in-house • All the documentation clearly stated Heroku was the workload’s destination • CRAFTY and I loved it! • It was so gangster

• IT department approval authorities signed off on utilizing cloud resources despite saying they never would • They just signed off on a full blown shadow IT project and it was right there in their own documents • This proved their ITIL and ITSM death grip process was not meeting their desired objectives • The cloud cat was out of the bag

@ChrisShort devopsish.com • IT attempted to change and satisfy the team's needs in-house • The IT process was a strict, by the book adherence to ITIL • It would never be able to shoehorn itself into a speedier process until the need for hand jamming tickets was eliminated • OpenStack instance as a developer playground was going to take a year or more • Tooling to enable API calls for IT resources was outsourced to a vendor with little feedback from our team • IT did not get it but the team was working around the bottlenecks • It was working for the team but my job of trying to get the IT org looking towards the future was next to impossible

• Resistance and educating the IT organization were critical components of helping them on their path towards transformation • The alternative was to join in the IT organization's thinking which as stifling innovation • Join them and you can ruin the galaxy together!

• The rebellion was in full force • Microservices were being built and deployed • IT organization had no way of keeping up with the multiple releases a day • Change occurred when IT organization realized it was lagging way behind • IT organization realized they couldn't compete on costs • Do to a host of other issues I decided to leave this position for another opportunity • Eventually compliant Azure resources were made available to the teams • Last I heard it was still taking months to get a VM in their datacenter

DEVOPS STRIKES BACK (LESSONS LEARNED) @ChrisShort devopsish.com What did I learn from these two journeys? Aside from changing jobs is easy in Raleigh/Durham

"OUR GREATEST GLORY IS NOT IN NEVER FAILING, BUT IN RISING UP EVERY TIME WE FAIL." Ralph Waldo Emerson EMBRACE FAILURE, LEARN FROM IT

@ChrisShort devopsish.com • A mutiny and a rebellion might look the same from opposing perspectives • The rebellion was to change for the better while the mutiny was to maintain the status quo • This is not to say Ops is bad and Devs are great (far from it) • These jobs along my journey proves that motivated teams can act towards their common goal be it good or bad

@ChrisShort devopsish.com • I have learned that DevOps needs allies at all levels • I wish I would have learned this earlier in my career • Not just leadership • Not just individual contributors • People have to want to better their environments

@ChrisShort devopsish.com • The Dev Rebellion had people at all levels pointing at successes of the team • The Ops Mutiny had a core group of people trying to cast off change regardless of outcomes • But both were personal failures since I wasn't able to rally enough support in enough time to keep me from going mad

⚙ ⚙ ⚙ @ChrisShort devopsish.com • In Conclusion... • The DevOps journey is a combination of people, processes, and tooling • The culmination of these is a team of allies pushing the envelope • Never resting on their laurels • There is no such thing as a completed DevOps transition • Resting on your laurels is not a DevOps mindset • DevOps and its allies should be iterating and improving upon what they have learned daily

@ChrisShort devopsish.com • Thank you for your time today • Feel free to reach out to me on Twitter if you ever need another opinion or a nudge in a direction • Take care of each other out there