FROM METRICS TO MASTERY Improving Performance with DORA and the SPACE Framework Nicole Forsgren, PhD

TODAY’S AGENDA 01 DORA 02 SIGNAL & ACTION 03 SPACE 05 CULTURE 06 RECENT RESEARCH

Hello! I’m Nicole! I’ve been a software engineer, sysadmin, and researcher.

DevOps was the original hipster for making work more productive and sustainable.

DevOps was the original hipster for making work more productive and sustainable. DORA and SPACE help us understand, measure, and improve our productivity and our well-being.

DORA 01

DORA – quick review DORA 4 metrics measure speed and stability: ● Lead time for changes ● Number of deploys ● Mean time to recover ● Change fail rate DORA is also a research program that helps teams and organizations improve their performance on those four metrics

What is DORA for? benchmarking

What is DORA for? benchmarking

What is DORA for? DORA is a framework that helps us: ● Improve outcomes ● Measure performance ● Identify capabilities for improvement – and provide evaluative criteria Check out dora.dev for the Quick Check and more!

SIGNAL & ACTION 02

SIGNAL AND ACTION SIGNAL: How am I doing? ACTION: What can – and should – I do to improve?

DORA as Signal & Action SIGNAL: DORA Metrics → How well can my team develop and deliver software? ACTION: What capabilities should I focus on to improve my software delivery? → Can identify using DevEx methods, value stream, or others.

Dora.dev Quick Check

SPACE 03

What is SPACE for? one set of metrics for everything

What is SPACE for? one set of metrics for everything

What is SPACE for? SPACE is a framework that helps us think about how to measure productivity – or any complex, creative work

SPACE and DORA are complementary Once you’ve identified the capabilities you want to improve (using DORA), you can use SPACE to identify how you want to measure it

More on SPACE It IS - Complementary to DORA - a framework used to identify metrics appropriate for a situation or goal - more broad than DORA (e.g., can be used for sales teams or copywriters) It is NOT - An evolution of DORA - a set of prescribed metrics

SPACE FRAMEWORK DIMENSION DEFINITION S: Satisfaction and well-being How fulfilled, happy, and healthy one is P: Performance An outcome of a process A: Activity The count of actions or outputs C: Communication and collaboration How people talk and work together; how systems talk and work together E: Efficiency and flow Doing work with minimal delays or interruptions A holistic way to measure productivity https://queue.acm.org/detail.cfm?id=3454124

APPLYING SPACE TO INCIDENT MANAGEMENT(1 OF 2) DIMENSION EXAMPLE METRICS S: Satisfaction and well-being how satisfied SREs are with the IM process, escalation and routing, and on-call rotations are key metrics to capture, especially since burnout is a significant issue among SREs. P: Performance these measures focus on system reliability; monitoring systems' ability to detect and flag issues faster, before they hit the customer and become an incident. MTTR (mean time to repair) overall, and by severity. A: Activity number of issues caught by the monitoring systems, number of incidents created, number of incidents resolved—and their severity distribution.

APPLYING SPACE TO INCIDENT MANAGEMENT(2 OF 2) DIMENSION EXAMPLE METRICS C: Communication and collaboration people included in resolving the incident, how many teams those people came from, and how they communicate during an incident. Incident resolution documentation outlines the steps involved in resolving incidents; this can be measured by completeness (to check if any resolution data was entered) or quick quality scores (e.g., thumbs up/down). Teams may also include a metric that measures the percentage of incidents resolved that reference these guides and documentation. E: Efficiency and flow incident handoffs, incident assignment/re- assignment, number of hops an incident has to take before it is assigned to the right individual or team.

DORA AS SPACE DORA METRIC SPACE DIMENSION LEAD TIME EFFICIENCY AND FLOW DEPLOY FREQUENCY ACTIVITY MTTR EFFICIENCY AND FLOW CHANGE FAIL RATE PERFORMANCE AVAILABILITY PERFORMANCE Criteria: Have metrics across at least 3 of the 5 dimensions for a given focus area; this covers dimensions for software delivery.

GOOD DAY PROJECT We did a study to help developers get quick and easy signals and patterns to help them have better days, more consistently. (“What makes a good day, and how can I have one more often?”) The deets: ● Based on holistic concept of productivity using the SPACE framework ● Focus on individual: measures for us, not for managers ● To help us measure our energy, not our time ● Goal is quick and easy measures with actionable signal

GOOD DAY PROJECT To capture SPACE quickly, we asked questions like: ● How was your work day? ● I worked with other people ● My work was interrupted ● How many meetings did you have today? ● Today, I felt most productive… (and least productive) https://github.blog/2021-05-25-octoverse-spotlight-good-day-project/

FINDING FLOW IS KEY AND INTERRUPTIONS ARE A DRAG Minimal or no interruptions give developers Chance of having a good day https://github.blog/2021-05-25-octoverse-spotlight-good-day-project/ Interruptions throughout the day Chance of having a good day Minimal or no interruptions give developers Chance of having a good day 82% 7%

Metrics can give us insight and direction to improve productivity and well- being, but they aren’t enough to create change

Culture 04

Technology and culture influence each other

Tech influences culture

NUMMI When we change the way we work, we influence our culture

Culture influences tech

No content

—GREG LINDEN "I think building this culture is the key to innovation. Creativity must flow from everywhere. Whether you are a summer intern or the CTO, any good idea must be able to seek an objective test, preferably a test that exposes the idea to real customers. Everyone must be able to experiment, learn, and iterate. Position, obedience, and tradition should hold no power. For innovation to flourish, measurement must rule.” http://glinden.blogspot.com/2006/04/early-amazon-shopping-cart.html

RECENT RESEARCH 04

A COUPLE HIGHLIGHTS Patterns in behavior change when using AI- assisted programming. For example, we see (and maybe expect) that users will spend more time reviewing than writing, but we now also see that users spend much more time not only prompt crafting (and waiting for a suggestion) but then returning to verify suggestions. This upends traditional models and suggests new user interfaces when AI- assisted models are present. https://arxiv.org/pdf/2210.14306.pdf This work study investigated hybrid work among 3400+ people, 2,300 of them developers+, across 28 companies. It found that the highest productivity is found among those who work entirely from home, even when controlling for several factors. This may be because hybrid work is difficult to optimize. Key challenges include missing social interactions, too many meetings, inefficient work practices, lack of clear business objectives, and difficulty collaborating. Several additional findings and implications are discussed. aka.ms/hybridworkpaper Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming. 2022 Mozannar et al. The Best of Both Worlds: Unlocking the Potential of Hybrid Work for Software Engineers. 2023 Houck et al.

THANK YOU! I’M EXCITED TO JOIN YOU FOR Q&A