DEEP WORK A NEW WORKING MODEL FOR OPS TEAMS IN A DEVOPS ENVIRONMENT Roderick Randolph Arthur Maltson

CURRENT MODEL ✅ /roderickrandolph Before talking about what a new working model might look like, let’s talk about what the current model looks like.

DO YOU?… ✅ /roderickrandolph

TEXT ✅ /roderickrandolph Have Developers lined up at your desk looking for help before you’ve had your morning coffee?

✅ /roderickrandolph Or, do you find yourself multitasking or constantly getting interrupted throughout the day with unplanned tasks?

100 999 ✅ /roderickrandolph Do you have a hard time keeping up with all your Slack messages or do you have an inbox filled with unread email messages that distract you from getting your work done?

DEVOPS INTERRUPTITIS ✅ /roderickrandolph If so, you may have a case of the DevOps Interruptitis. You’re not alone. At Capital One (where Arthur and I work), our Ops team suffers from this problem too. We work in a highly collaborative environment where our Ops team sits near the Devs where the Devs can walk up to our team for help - Jenkins job build failure - Questions about Docker - AWS, or some monitoring tools - Etc This simple tap on the shoulder takes us away from working on developing that new shiny tool or automate a manual process. And by the end of the day you’re exhausted and wondering "what did I do today”? While there's no cure for Interruptitis.

✅ /roderickrandolph We should consider a treatment to help subside the symptoms of being about to focus without interruption. (hopefully without having to take this many pills).

TEXT ✅ /roderickrandolph You could resort to hiding behind your laptop but that’s just not practical in our collaborative environment.

TEXT ✅ /roderickrandolph Or you could try creating a maze of processes between you and your Devs. What this may look like in reality is an intake request process where dev have to create a JIRA issue just to talk to you. By doing this, we lose the ability to collaborate and interact with our devs and this is another way of building a wall between the Ops team and the Dev teams. We don’t want that, it’s not DevOps.

✅ /roderickrandolph Or you can just give up and just always doing interruptive work.

TEXT ✅ /roderickrandolph There has to be a better way! Let’s take a quick detour to explain the new model and the concept of deep work

[1]+ Stopped $ fg % $ bg % ^z

DEEP WORK ✅ @amaltson Before we dive into what deep work is and why we want to experience it, let’s look at a typical office environment looks like.

GLORIA MARK OF THE UNIVERSITY OF CALIFORNIA, IRVINE, FOUND THAT A TYPICAL OFFICE WORKER GETS ONLY 11 MINUTES BETWEEN EACH INTERRUPTION, WHILE IT TAKES AN AVERAGE OF 25 MINUTES TO RETURN TO THE ORIGINAL TASK AFTER AN INTERRUPTION. Brain, Interrupted - Bob Sullivan and Hugh Thompson New York Times - http://nyti.ms/1fdkVUT ✅ @amaltson

VIEWING AN ENGINEER AS AN INTERRUPTIBLE UNIT OF WORK, WHOSE CONTEXT SWITCHES ARE FREE, IS SUBOPTIMAL IF YOU WANT PEOPLE TO BE HAPPY AND PRODUCTIVE. Dave O’Connor Google SRE Book ✅ @amaltson

DEEP WORK SHALLOW WORK ✅ @amaltson In Cal Newport’s book on Deep Work, he defines Deep Work as: “Professional activities performed in a state of distraction-free concentration that push your cognitive capabilities to the limit. These efforts create new value, improve your skills, and are hard to replicate.” On the flip side, shallow work which is what DevOps Interruptitis is, is defined by Cal Newport as: “Noncongitively demanding, logistical-style tasks often performed while distracted. These efforts tend to not create much new value in the world and are easy to replication.”

THE ABILITY TO PERFORM DEEP WORK IS BECOMING INCREASINGLY RARE AT EXACTLY THE SAME TIME IT IS BECOMING INCREASINGLY VALUABLE IN OUR ECONOMY. AS A CONSEQUENCE, THE FEW WHO CULTIVATE THIS SKILL, AND THEN MAKE IT THE CORE OF THEIR WORKING LIFE, WILL THRIVE. Cal Newport Deep Work - Rules for Focused Success in a Distracted World ✅ @amaltson

✅ @amaltson We’ve all experienced occasional bouts of deep work, you may know it by another name in the tech community, usually known as Flow. When you are in flow, code streams from your hands. You solve complex problems that have been plaguing you for a long time.

A DEEP LIFE IS A GOOD LIFE ✅ @amaltson Cal Newport Deep Work - Rules for Focused Success in a Distracted World

NEW MODEL ✅ @amaltson OK, so Deep Work sounds like something you want to experience. But how do we structure our teams to allow for longer stretches of deep work. We really need a new model.

✅ @amaltson We think of the current model analogous to having surgery without anaesthetics and causing massive headaches.

✅ @amaltson We wanted to try a more modern approach. Something that wouldn’t cause as many headaches.

TEXT ✅ @amaltson The first thing we did was to split the team into two separate teams. Each sub-team specializes in a different work mode.

FOREGROUND ✅ @amaltson The first part of the team we call the Foreground team. This team deals with all the interruptions, triages requests, fights fires and deals with outages. This part of the team is also on call, which helps with sustainable on call. People on Foreground know they won’t be making meaningful progress on their project work because the expectation is that they will get interrupted regularly. This is similar to a triage nurse in a hospital who figures out which patients are high priority, which aren’t, answers questions and generally gets interrupted on a regular basis.

fg: fg [job_spec] Place JOB_SPEC in the foreground, and make it the current job. If JOB_SPEC is not present, the shell's notion of the current job is used. ✅ @amaltson And the word “foreground” probably sounds familiar. That’s because we intentionally named it after the fg command in Unix which puts a process on foreground and allows it to accept user input and get interrupted regularly.

BACKGROUND ✅ @amaltson The other part of the team, which we call the Background team, is shielded from these interruptions and works on making meaningful progress on complex problems, investigating and/ or building new tools, etc. This is similar to the doctors working in the OR and are suppose to be undisturbed.

bg: bg [job_spec ...] Place each JOB_SPEC in the background, as if it had been started with `&'. If JOB_SPEC is not present, the shell's notion of the current job is used. ✅ @amaltson We used the Unix metaphor again here, so the background team is similar to the bg command in Unix where it’s not easily interrupted (unless you send a kill command, please don’t do that to team members!).

ROLES AND RESPONSIBILITIES Foreground Background Request intake Work on strategic tools Firefighting Automating away fires On-call Automate manual processes Help with general questions Pay down infrastructure debt Assist with failing builds Improve existing tools Answering Slack channel Deep dive on new tools Answering shared inbox Build new tools ✅ @amaltson Here’s an example of the roles and responsibilities breakdown we have on our team. This will most likely vary for your teams.

✅ @amaltson Being on Foreground all the time would definitely not be sustainable, so make sure to rotate regularly. On our team at Capital One, we rotate weekly and with our size of team, a member is on Foreground for one week and on background for two.

[2]+ Stopped $ fg % $ bg % ^z

GET STARTED ✅ /roderickrandolph This new model sounds fantastic, how do you get started?

As an Ops engineer you’re probably super excited. You now have a new model that can help you make meaningful progress on those new initiatives or projects you’ve always wanted to do.

✅ /roderickrandolph But the Devs probably aren’t as excited because this new model requires them to change their behaviour. It can be difficult to get Devs to change their behaviour, instead of going directly to their favourite Ops engineer they’re going to be forced to communicate with another Ops engineer that on Foreground. So how do you get Devs on board?

✅ /roderickrandolph Start by communicating, a lot. Have a conservation with your stakeholders, developers, business owners. Mention the idea, plant those seeds, etc. Quote the research around how deep work has proven to increase productivity.

✅ /roderickrandolph Sell it as an experiment. Time box it (for the next 1 month, a quarter we’re going to try this new model). Let the Devs and others know that they will still be supported. The idea is to iterate and improve. Ultimately, they want to see new tools and technologies as much as you do.

✅ /roderickrandolph As you iterate and improve the model, keep people in the loop. Provide updates to stakeholders on the team’s progress. Don’t be afraid to share what worked and what didn’t work. Gather feedback and incorporate it into your next iteration.

VISUALIZATION ✅ /roderickrandolph Most importantly, you need to let people know your status. How do Devs know who’s on Foreground and who’s on Background?

✅ /roderickrandolph At Capital One we started out by using sticky notes. Red for Background and Green for Foreground.

✅ /roderickrandolph But we found the stickies didn’t stick, so we tossed them. We found a better solution…

✅ /roderickrandolph When we found the Luxafor lights, and replaced the stickies with the LED lights.

✅ /roderickrandolph The brightness gives a a great status indicator. It also allows people on Foreground indicate they’re occasionally busy, for example when troubleshooting an outage.

✅ /roderickrandolph Stickies only work for walkups, but we also need to let people know online too. At Capital One we use Slack, so we started using the new Slack status indicators. We also get a little creative indicating our status.

✅ /roderickrandolph We also indicate to all members of the DevOps chat room (where people come to ask questions), which members are on Foreground using the Slack channel topic.

I’M SORRY DEV, RODERICK IS BACKGROUND HAL-9000 ✅ /roderickrandolph Our next iteration will be to develop a ChatBot that will intercept Slack messages and kindly redirect a dev to the engineer on Foreground.

[1]+ Stopped $ fg % $ bg % ^z

KEEP IT GOING ✅ @amaltson We’ve convinced people to let us try and we have the necessary visualizations, but how do we keep it going and make it a success.

✅ @amaltson The primary way is constant reminder and reinforcement, like hand washing in hospitals. Visualizations help, like the lights, but sometimes you need to be firm. This is hard in our normally polite culture, but it’s worth it.

✅ @amaltson Expect failures, especially in the first several months. Keep trying, keep iterating, keep reminding. When on Background, try to stay firm. One iteration we recently introduced…

Was pair programming for team members on Background. It’s pretty awkward to interrupt two people in conversation, so it helps keep interruptions at bay. A happy benefit has been the quality of code coming from pairs was much higher than individual, so clearly a huge productivity gain.

To keep Foreground members productive, we created Office Hours (that people know about, of course!). This helps funnel longer discussions that come from interruptions into fixed time slots and less urgent requests go into the Office Hours directly. We’re not trying to make interruptions go away, helping teams is our job. We want to deal with them in batches to improve productivity.

✅ @amaltson Don’t be afraid to work from home. If you’re working on something individually, i.e. not in a pair and you’re having difficulty making progress, work from home. Removing yourself from the work environment helps avoid interruptions. Now you just have to keep yourself focused!

CONCLUSION ✅ @amaltson

#OPSLIFE ✅ @amaltson We live in an interrupt driven world and specifically work in an interrupt driven role.

DEEP WORK ✅ @amaltson But Deep Work is worth it, it’s worth that investment.

KEEP IT GOING ✅ @amaltson So keep going, keep iterating, keep reminding.

DEVOPS INTERRUPTITIS @amaltson ✅ And maybe you’ll find a treatment to DevOps Interruptitis.

✅ @amaltson And have success with a new working model for DevOps teams.

[2]+ Stopped $ fg % $ bg % ^z

WE’RE HIRING ✅ /roderickrandolph If this sounds interesting, and you'd like to work in an environment where this is encouraged, and work with AWS, open source tech, and even contribute and open source your own projects, we're hiring!

RODERICK RANDOLPH Slides: https://speakerdeck.com/roderickrandolph Work: Capital One Canada Loves: DevOps Hates: Poorly written software & software without tests /roderickrandolph PERSON WE’RE HIRING!

ARTHUR MALTSON Slides: https://speakerdeck.com/amaltson Stats: 70% Dev / 30% Ops, 110% DadOps Work: Capital One Canada Loves: Automation, Ruby, Ansible, Terraform Hates: Manual processes @amaltson PERSON WE’RE HIRING! maltson.com

THANK YOU

TEXT Questions?

CREDITS ▸ Slide 1, davebloggs007, Others have passed this way, https://flic.kr/p/ftbu3e ▸ Slide 7, 55, Hamza Butt, Slethoscope, Laptop, Doctor, https://flic.kr/p/U5JSDr ▸ Slide 8, Jamie, Pills, https://flic.kr/p/9KY9Wj ▸ Slide 12, CodyJung, Detour, https://flic.kr/p/n3cTbR ▸ Slide 19, JD Hancock, Calming The Storm, https://flic.kr/p/8oKjx6 ▸ Slide 22, peter layzell, Twitter, https://pbs.twimg.com/media/C683ctQW0AAutRl.jpg ▸ Slide 25, ILO in Asia and the Pacific, hospital reception desk, https://flic.kr/p/dgSLXX ▸ Slide 27, Phalinn Ooi, OT, https://flic.kr/p/8Y8LPX ▸ Slide 30, UGA College of Ag & Environmental Science, Centrifuge_8905, https://flic.kr/p/oMraAq ▸ Slide 33, Can’t talk yet I must try to communicate with them telepathically, http://www.dumpaday.com/wp-content/uploads/2014/05/Cant-talk-yet-I-must-try- to-communicate-with-them-telepathically.jpg ▸ Slide 34, Excited chimp, http://www.memecenter.com/fun/2547937/i-amp-039-m-so-excited ▸ Slide 36, Bridget Coila, Remedy Tea Test tubes, https://flic.kr/p/G5Fih ▸ Slide 48, Katherine Bowman, How to wash your hands, https://flic.kr/p/6eFs7U ▸ Slide 50, , Working From Home, http://theoatmeal.com/comics/working_home ▸ Slide 62, Fredrik Rubensson, thinker, https://flic.kr/p/dC1gvd ▸ Slide 63, Round and round gif, Giphy, https://giphy.com/gifs/round-and-fwdPOapin5BBK/ ▸ Others: DepositPhotos