The Dimly Lit Side of DevOps - Application Support Nightmares

I’d like to propose an Ignite that hits close to home for me. I’ve been in App Support for a number of years and I’ve dealt with a few common nightmares associated with the role. The ignite goes over the top 5 nightmares an app support engineer deals with and I give a bit of insight on how to better come to grips on handling them.

Transcript

1. Dimly Lit Side of DevOps App Support Nightmares Joshua Goldman
 Application Support Engineer @ SecureWorks
2. The Nitty Gritty • Traditional Sysadmin Vs DevOps/SRE • Are the following situations nightmares or rights of passage? • Growing Pains for Traditional Shops • You aren't alone
3. The Nightmares • Poor Service Transition • Broken Call Trees • The Vicious Bug Cycle • Losing Senior Resource(s) • The Dreaded Release Night
4. Poor Service Transitions • Yeah Kevin from Customer Service says the "Pan Galactic Gargle Blaster Ordering Service" Is down • Who, what, where, when, why me? • Lack of documentation and sudden outage are a lethal combination
5. The Birth of an Application • More focus is given to development than support • Shiny New Toy Syndrome - When
 can we deliver
 the next product?
6. Service Transition • Implement a handoff procedure with your Ops teams • Services should not go into production until error budget is satisfied • Try a Devs on call model or a Launch Coordination Engineer role • Make SOMEONE ANYONE document for your support teams
7. Broken Call Trees • We called Ben, Aric, and Jerry although for some reason Jerry's toaster said it's mailbox was full. • How many times did you call them? • Who cares if its 4am in Israel. Wake him up! • Who's next on the call tree? The VP of Engineering?
8. They Didn't Pick Up • Panic is starting to set in • All the ops folks are scrambling • No one knows what the next step is • You've been in outage for several hours
9. Incident Management • Create a incident management procedure that works in your organization • Make sure your applications have clear escalation paths • Consider secondary or tertiary on-call resources
10. Losing Senior Resource(s) • What would ever happen if you lost your "Brent"? • Think about all the knowledge that you would lose. • All the things that won't or can't get done anymore. • How will your newer folks become superstars if your superstar left?
11. "Brent" Used To Do that • Why hasn't project X gotten done? • Why are things starting to break? • Didn't so and so support this?
13. Get Through It • Stay Calm • Allow your junior resources to take on more responsibilities • In fact encourage them to take over where your "Brent" left off • Things will slide but address them instead of trying to create a new rock star
14. The Vicious Bug Cycle • Bugs cause outage • Outage causes developers to be dragged into incident calls • Developers can't work on fixing bugs because they are putting out fires • Bugs pile up causing more outages
15. End The Cycle • Consider a development freeze • Re-evaluate your Service Level Objectives • Focus more resources on preventative care to reduce your firefighting • Move the budget needle from innovation focus towards maintaining what you already have
16. Release Week!!!!! Crash Positions!!!
17. Continuous Deployment or Delivery • Continuous deployment is a term that gets thrown around a lot • What if your organization isn't there yet? • Do you lack a proper test environment?
18. Get Your Act Together • Get a testing environment that mimics production as best you can • Throw some test data in it just in case • Make sure your applications do not exceed their error budgets before being released • Engineers On Call for the release
19. Bring Change & Growth • Not every company is Netflix, Google, Facebook, or Amazon • Many companies still use the monolithic Development and Operations style separation of teams. • Buy your manager a copy of a DevOps Book • We're all in this together.
20. Thank You Remember you aren't alone! Many organizations have these nightmares and many more. Stop by DevOpsDays Chicago in 2017!