"Development" and "operations," as independent organizations, have conflicting motivations. DevOps is the buzzword for the understanding that we've missed the forest (providing value) for the trees (features and stability).
I'll share three true stories about three real organizations.
NB. if you work in an organisation with conflicts between development and operations, bring a story to share— and get free advice!
Who has deployed to production?
DevOps is People
4 countries, 8 cities. People everywhere are the same.
processes and tools
Who feels like a "Dev?" / like an "Op?"
Features vs. Uptime
Dev: gets paid for features / Op: paid for uptime — Whose position changed?
I’ll achieve my KPIs by
making you fail to meet
Relevant in services organizations where "Development" and "Operations" are different
groups— lessons apply elsewhere! (Medical machines)
Change is risk
Amazon Web Services, Azure, VMWare, Heroku, GAE, iCloud, Salesforce
Chef, Puppet, Babushka, Juju, Jenkins, Travis, Go
The bar has been raised
•Getting features out faster
•Faster recovery from failure
It's Not a Secret
Get Dev and Ops to
"DevOps" is not a role.
At best: it's a initiative.
Die Technik des Dramas
Gustav Freytag (1863)
Novelist and Playwright
• Rising action
• Falling action
Think of your own story
My ﬁrst DevOps
A "boutique" subsidiary of a major insurance company. Online sales portal gone PCI
compliant using a suspicious third-party vendor ("Can you send us your API?")
I suspected the vendor unreliable. (Test server off network. Same network as production.)
Spoke with the vendor...
That's only the test
service. It's a single
It’s not a problem.
We haven't been told to
monitor the external
It’s not our problem.
We don't need to monitor
We have SLAs for that.
It’s not your problem.
Beer + Credit Card +
Pingdom = ?
I have an equation...
In old times, this have been the end of idle curiosity
Little did our man in operations and our program manager know, but they'd just gone on
By the next Wednesday, there a meeting was held. The vendor, operations, development, and
the business are all in the same room because we had a common goal.
How do we stop losing
•Work when services fail
Internal (nagios) and external (kingdom). Give Dev insight failures.
Process the sale. Ops can manage a queue of e-mails saying the card didn't run.
Give Dev and Ops a club to beat up our common enemy.
By the next Friday, we all had a drink together. Money was being made.
A government regulatory agency. Not a tech shop. Outsourced IT. Ops and Infra. ran by a
Did you raise a ticket?
One ops person on-site to put a human face on ...
You build applications
Working on LOB applications. Had provisioned servers, control over the deployment process,
things were smooth. It can be nice to work with hands-off operations guys.
But, changes were afoot. Infrastructure being virtualized as a cost-saving measure. File
servers, databases, web servers, domain controllers, e-mail-- everything!
Central Point of Failure
One morning, everyone was standing, hands in their pockets, looking and gossiping like an
accident had happened.
CAS had gone down. Every single sign-on service— this included our applications— couldn't
When the server ﬁnally did come up, after a full day, a retrospective was held:
• "What happened?" "We tried migrating
the authorization server to a VM."
• "What went wrong?" "We don't know."
• "Will this happen again?" "We can't
make any promises."
• “What do we do if this happens again?”
Dev helping Ops
Someone else’s problem is your problem
• Stub Authentication Server
•Build levers and knobs for
•Only a button: on and off
Built a stub server
Telecoms have thousands of IT systems and established (entrenched) cultures to manage
Lodge change management request weeks before. This is because you're an agile team.
Ops and stakeholders are notiﬁed. Timelines, resources and billing codes.
Finally, the big day. Everyone says "see you later," because around 9pm everyone is back in
Speaker phone with voices. Laptops, IM, idle Facebook. "App is down." "App is back up...
seems to be stalling. We'll try restarting the web server." More time passes. It's been 30
minutes. "OK, app is up! Let's go!"
Action! Windows and tabs on their machines. Features are validated. “What's the test
username and password again?"
Shaking hands, borrowing cigarettes and catching cabs. You did it. This was the culmination
of three months.
The Next Morning
You're the second one in-- your business analyst is already there because he had a meeting
at 10. As you walk past him, he looks up with a dark look on his face. "We got rolled back this
Operations Has Done
They don’t trust us.
Let’s Earn Some Trust
•Use feature toggles
•Automated acceptance testing
•Painful and honest
conversation with our
•Stop Big Bang releases
Use feature toggles
•Completion of features has
little relation to when
•Gets code out and live
•These are levers and knobs
“Sorry, we’re under
Our ﬁrst toggle
•Only one person from dev
needs to come in for the
•Give conﬁdence things work
•Save people’s time
•Make solid releases
Delight the Business
This won’t get us daily
The next emergency does
Always a next emergency. Out-of-schedule release. You knock it out of the park.
Business thinks, "these guys are solid, and the feature I want out next week is scheduled for
two weeks from now."
Gave them all the ammunition they need; but no one to ﬁght— Dev and Ops have been
working together this whole time.
DevOps is technology
DevOps is people