開発生産性Conference Keynote Dr.Nicole Forsgren : From Metrics to Mastery: Improving Performance with DORA and the SPACE Framework

開発生産性ConferenceのKeynoteにて登壇いただいたDr.Nicole Forsgrenの資料になります。
ご本人に許可をいただき公開させていただきます。

◆タイトル◆
From Metrics to Mastery: Improving Performance with DORA and the SPACE Framework

◆アブストラクト◆
Join Dr. Nicole Forsgren for an enlightening talk on utilizing DORA metrics, the SPACE productivity framework, and the latest productivity research in software development. Discover how to use DORA metrics to measure DevOps performance and drive actionable insights. Learn how DORA and the SPACE productivity framework are similar – and different. Explore the impact of culture on software delivery and productivity. And learn about some of the latest productivity research, covering hybrid work models and tools like GitHub Copilot. In this talk, you’ll gain practical insights to enhance software delivery, improve productivity, and navigate the evolving landscape of development. Don't miss this opportunity to learn from Nicole Forsgren and optimize you and your team's work.

（キーノートでは、DORAメトリクス、SPACEの開発生産性フレームワーク、そしてソフトウェア開発における最新の開発生産性の研究結果についてお話しします。
まずは、DevOpsのパフォーマンス計測のためにDORAメトリクスをどのように活用し、どのように実践的なインサイトを得るのかについて探ります。そして、DORAメトリクスとSPACEフレームワークの共通点、相違点についてもお伝えします。
最後に、リモート・出社のハイブリットワークやGitHub Copilotのような生産性を高めるツールを含めて、最新の開発生産性の研究結果についてお話しします。
セッションを通して、ソフトウェアデリバリーを強化しながら開発生産性を上げ、進化し続ける開発環境を同ナビゲートしていくのかについての実践的なインサイトを得ることができるでしょう。
ぜひ本セッションから学びを得て、自分自身とチームの仕事でより生産性を高めてください。）

Transcript

1. FROM METRICS TO MASTERY Improving Performance with DORA and the SPACE Framework Nicole Forsgren, PhD
2. TODAY’S AGENDA 01 DORA 02 SIGNAL & ACTION 03 SPACE 05 CULTURE 06 RECENT RESEARCH
3. Hello! I’m Nicole! I’ve been a software engineer, sysadmin, and researcher.
4. DevOps was the original hipster for making work more productive and sustainable.
5. DevOps was the original hipster for making work more productive and sustainable. DORA and SPACE help us understand, measure, and improve our productivity and our well-being.
6. DORA 01
7. DORA – quick review DORA 4 metrics measure speed and stability: • Lead time for changes • Number of deploys • Mean time to recover • Change fail rate DORA is also a research program that helps teams and organizations improve their performance on those four metrics
8. What is DORA for? benchmarking
9. What is DORA for? benchmarking
10. What is DORA for? DORA is a framework that helps us: • Improve outcomes • Measure performance • Identify capabilities for improvement – and provide evaluative criteria Check out dora.dev for the Quick Check and more!
11. SIGNAL & ACTION 02
12. SIGNAL AND ACTION SIGNAL: How am I doing? ACTION: What can – and should – I do to improve?
13. DORA as Signal & Action SIGNAL: DORA Metrics → How well can my team develop and deliver software? ACTION: What capabilities should I focus on to improve my software delivery? → Can identify using DevEx methods, value stream, or others.
14. Dora.dev Quick Check
15. SPACE 03
16. What is SPACE for? one set of metrics for everything
17. What is SPACE for? one set of metrics for everything
18. What is SPACE for? SPACE is a framework that helps us think about how to measure productivity – or any complex, creative work
19. SPACE and DORA are complementary Once you’ve identified the capabilities you want to improve (using DORA), you can use SPACE to identify how you want to measure it
20. More on SPACE It IS - Complementary to DORA - a framework used to identify metrics appropriate for a situation or goal - more broad than DORA (e.g., can be used for sales teams or copywriters) It is NOT - An evolution of DORA - a set of prescribed metrics
21. SPACE FRAMEWORK DIMENSION DEFINITION S: Satisfaction and well-being How fulfilled, happy, and healthy one is P: Performance An outcome of a process A: Activity The count of actions or outputs C: Communication and collaboration How people talk and work together; how systems talk and work together E: Efficiency and flow Doing work with minimal delays or interruptions A holistic way to measure productivity https://queue.acm.org/detail.cfm?id=3454124
22. APPLYING SPACE TO INCIDENT MANAGEMENT(1 OF 2) DIMENSION EXAMPLE METRICS S: Satisfaction and well-being how satisfied SREs are with the IM process, escalation and routing, and on-call rotations are key metrics to capture, especially since burnout is a significant issue among SREs. P: Performance these measures focus on system reliability; monitoring systems' ability to detect and flag issues faster, before they hit the customer and become an incident. MTTR (mean time to repair) overall, and by severity. A: Activity number of issues caught by the monitoring systems, number of incidents created, number of incidents resolved—and their severity distribution.
23. APPLYING SPACE TO INCIDENT MANAGEMENT(2 OF 2) DIMENSION EXAMPLE METRICS C: Communication and collaboration people included in resolving the incident, how many teams those people came from, and how they communicate during an incident. Incident resolution documentation outlines the steps involved in resolving incidents; this can be measured by completeness (to check if any resolution data was entered) or quick quality scores (e.g., thumbs up/down). Teams may also include a metric that measures the percentage of incidents resolved that reference these guides and documentation. E: Efficiency and flow incident handoffs, incident assignment/re- assignment, number of hops an incident has to take before it is assigned to the right individual or team.
24. DORA AS SPACE DORA METRIC SPACE DIMENSION LEAD TIME EFFICIENCY AND FLOW DEPLOY FREQUENCY ACTIVITY MTTR EFFICIENCY AND FLOW CHANGE FAIL RATE PERFORMANCE AVAILABILITY PERFORMANCE Criteria: Have metrics across at least 3 of the 5 dimensions for a given focus area; this covers dimensions for software delivery.
25. GOOD DAY PROJECT We did a study to help developers get quick and easy signals and patterns to help them have better days, more consistently. (“What makes a good day, and how can I have one more often?”) The deets: • Based on holistic concept of productivity using the SPACE framework • Focus on individual: measures for us, not for managers • To help us measure our energy, not our time • Goal is quick and easy measures with actionable signal
26. GOOD DAY PROJECT To capture SPACE quickly, we asked questions like: • How was your work day? • I worked with other people • My work was interrupted • How many meetings did you have today? • Today, I felt most productive… (and least productive) https://github.blog/2021-05-25-octoverse-spotlight-good-day-project/
27. FINDING FLOW IS KEY AND INTERRUPTIONS ARE A DRAG Minimal or no interruptions give developers Chance of having a good day https://github.blog/2021-05-25-octoverse-spotlight-good-day-project/ Interruptions throughout the day Chance of having a good day Minimal or no interruptions give developers Chance of having a good day 82% 7%
28. Metrics can give us insight and direction to improve productivity and well- being, but they aren’t enough to create change
29. Culture 04
30. Technology and culture influence each other
31. Tech influences culture
32. NUMMI When we change the way we work, we influence our culture
33. Culture influences tech
35. —GREG LINDEN "I think building this culture is the key to innovation. Creativity must flow from everywhere. Whether you are a summer intern or the CTO, any good idea must be able to seek an objective test, preferably a test that exposes the idea to real customers. Everyone must be able to experiment, learn, and iterate. Position, obedience, and tradition should hold no power. For innovation to flourish, measurement must rule.” http://glinden.blogspot.com/2006/04/early-amazon-shopping-cart.html
36. RECENT RESEARCH 04
37. A COUPLE HIGHLIGHTS Patterns in behavior change when using AI- assisted programming. For example, we see (and maybe expect) that users will spend more time reviewing than writing, but we now also see that users spend much more time not only prompt crafting (and waiting for a suggestion) but then returning to verify suggestions. This upends traditional models and suggests new user interfaces when AI- assisted models are present. https://arxiv.org/pdf/2210.14306.pdf This work study investigated hybrid work among 3400+ people, 2,300 of them developers+, across 28 companies. It found that the highest productivity is found among those who work entirely from home, even when controlling for several factors. This may be because hybrid work is difficult to optimize. Key challenges include missing social interactions, too many meetings, inefficient work practices, lack of clear business objectives, and difficulty collaborating. Several additional findings and implications are discussed. aka.ms/hybridworkpaper Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming. 2022 Mozannar et al. The Best of Both Worlds: Unlocking the Potential of Hybrid Work for Software Engineers. 2023 Houck et al.
38. THANK YOU! I’M EXCITED TO JOIN YOU FOR Q&A