Measure, Don't Guess: Observability as the key to performance tuning Software Delivery

4:20 PM > 4:40 PM (PST)

The golden rule of application performance tuning: measure, don’t guess. Yet when it comes to developer productivity, too many teams still guess. Builds are slow, tests are flaky, CI feels overloaded—and the default response is to throw hardware at the problem or hope it goes away.

In this talk, we’ll apply the performance engineering mindset to developer experience, showing how observability data from Develocity can profile builds and tests just like applications. By measuring and optimizing build and test performance, teams directly improve the DORA metrics that matter: shorter lead time for changes, lower change failure rates, faster recovery, and higher deployment frequency.

Developer productivity is a performance problem. If you want faster delivery and happier developers, the path is the same as for applications in production: measure first, then optimize.

Transcript

1. Trisha Gee
 Head of Developer Advocacy, Gradle Measure, Don’t Guess Observability as the key to performance tuning Software Delivery
2. Have you ever done performance tuning on an application without:
3. Have you ever done performance tuning on an application without: • Benchmarking • Pro fi ling
4. Have you ever done performance tuning on an application without: • Benchmarking • Pro fi ling
5. More computing sins are committed in the name of efficiency (without necessarily achieving it) than for any other single reason - including blind stupidity.” A Case Against the GOTO William Wulf (1972)
6. Image by Holly Cummins, used with permission. All rights reserved. “Measure, don’t guess”
7. What if we applied the same performance engineering mindset from optimising application performance to optimising developer experience?
8. The Pain
9. Waiting for Builds
10. Troubleshooting failures
11. Flaky Tests
13. Oh and security
14. CI cost Developer frustration Security risk Flaky tests Slow builds 🔒 🤯 💸 🔁 🐌
15. Quick Fix
16. Throw hardware at it!
17. …when does it make sense to throw hardware at a programming problem? As a general rule, I’d say almost always.” http://blog.codinghorror.com/hardware-is-cheap-programmers-are-expensive Jeff Atwood (2008)
18. Did it fix the problem?
19. Was it cost effective?
20. How can you tell?
21. 90% But still have queuing issues of CPUs in CI are unused
22. We are way behind where we ought to be as an industry. We are shipping code we don’t understand, to systems we have never understood.” https://charity.wtf/2020/03/03/observability-is-a-many-splendored-thing Charity Majors (2020)
24. Image by Dave Farley, used with permission. All rights reserved.
26. Image by Dave Farley, used with permission. All rights reserved.
27. We’re talking about measuring the “path to production”
28. And we’re talking about observability, not monitoring
29. Monitoring is about known-unknowns and actionable alerts, observability is about unknown-unknowns and empowering you to ask arbitrary new questions and explore where the cookie crumbs take you.” Observability: A Manifesto Charity Majors
30. The Solution
31. If you can not measure it, you can not improve it.” Lord Kelvin (1847)
32. I often say that when you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of a meagre and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely, in your thoughts, advanced to the stage of science, whatever the matter may be.” Lord Kelvin (1847)
33. If you can not measure it, you can not improve it.” Lord Kelvin (1847)
34. How do you get your gold star if you can’t prove it worked?
35. Let’s use science!
36. The ability to experiment is a critical part of innovation and creating business value.” Accelerate: The Science of Lean Software and DevOps Nicole Forsgren, Jez Humble, Gene Kim
38. How do you measure?
43. MCP
44. Your best and wisest refuge from all troubles is in your science.” Ada Lovelace
45. What do you see?
46. “It just takes a long time” • Assumption: “It’s just big.” • Measure: Task-level breakdown shows dependency resolution dominating.
47. 30%-40% of all CI build time at large organizations is spent downloading dependencies
48. Add caching
49. “Flaky tests are randomly occurring” • Assumption: “It’s random.” • Measure: • A small number of tests are responsible for most fl akiness • Tests are “random” but failures align with increased load • More fl akiness on certain infrastructure
50. Flaky tests are hard to fix because you have to find the problem
51. “…our analysis revealed that re-running the failing build and attempting to repair the flaky test were the most common actions.” Surveying the Developer Experience of Flaky Tests https://mcminn.info/publications/c72.pdf
52. “…our analysis revealed that re-running the failing build and attempting to repair the flaky test were the most common actions. Our findings also suggested that developers who experience flaky tests more often are more likely to take no action in response to them.” Surveying the Developer Experience of Flaky Tests https://mcminn.info/publications/c72.pdf
53. • Assumption: “Our pipeline is secure.” • Measure: An unveri fi ed dependency download identi fi ed in a build
54. Without obsesrvability you can’t know your pipeline is secure
55. DORA • Shorter builds/tests → faster lead time. • Reliable pipelines → higher deployment frequency. • Catching fl akes/failures → lower change failure rate. • Faster troubleshooting → lower MTTR. • And: complete visibility into builds/tests/dependencies → better security and audit readiness (Pervasive Security)
56. Observability is the foundation for performance, productivity, and compliance.
57. Start Measuring Today
58. Without observability you can’t:
59. See the state of where you started
60. Identify potential bottlenecks
61. Show that you’ve made positive changes
62. Thank you