Reldex: measuring the effectiveness of your app release process

First presented at Droidcon San Francisco 2024.

Transcript

1. PRATUL KALIA • TRAMLINE.APP Reldex Measuring the e ff ectiveness of your app release process
2. We measure what matters.
3. For e.g. Application performance • Cold start time • API throughput • Memory pressure
4. For e.g. Product metrics • Daily active users • Session length • Rate of feature adoption
5. For e.g. DevOps metrics • Lead time • Deployment frequency • Change failure rate • Mean time to restore
6. DORA metrics
7. Unfortunately, DORA metrics do not 
 fit app releases very well • No continuous delivery • Deployment frequency is too low • MTTR for... apps?
8. So, we’ve come up with something new.
9. Reldex RELease process inDEX
10. Apdex? Reldex uses the same grading maths
11. Apdex? We love the compact name
12. That’s it. No other similarity!
13. Reldex vs. Apdex A way to grade end user experience 
 by converting measurements from multiple metrics
 into a single score
14. Reldex vs. Apdex A way to grade release process performance 
 by converting measurements from multiple metrics
 into a single score Reldex does not care about end user experience at all.
15. So how does it work?
16. 1. Metrics 2. Acceptable ranges 3. Weights 4. Individual scores 5. Reldex value & grade
17. 1. Overall time to release 2. Time to stabilize RC 3. Time to complete staged rollout 4. Number of hot fi xes during staged rollout 5. Number of days since last release 6. Number of days between last release and latest hot fi x The 6 Metrics
18. Sort of like DORA? 1. Overall time to release 2. Time to stabilize RC 3. Time to complete staged rollout 4. Number of hot fi xes during staged rollout 5. Number of days since last release 6. Number of days between last release and latest hot fi x Lead time
19. 1. Overall time to release 2. Time to stabilize RC 3. Time to complete staged rollout 4. Number of hot fi xes during staged rollout 5. Number of days since last release 6. Number of days between last release and latest hot fi x Deployment Frequency Sort of like DORA?
20. 1. Overall time to release 2. Time to stabilize RC 3. Time to complete staged rollout 4. Number of hot fi xes during staged rollout 5. Number of days since last release 6. Number of days between last release and latest hot fi x Change Failure Rate Sort of like DORA?
21. 1. Overall time to release 2. Time to stabilize RC 3. Time to complete staged rollout 4. Number of hot fi xes during staged rollout 5. Number of days since last release 6. Number of days between last release and latest hot fi x Mean Time To Restore Sort of like DORA?
22. 1. Metrics 2. Acceptable ranges 3. Weights 4. Individual scores 5. Reldex value & grade
23. Scoring • Every metric has two parts. • Acceptable Range • Weight
24. Scoring • For e.g. Overall time to release: • Acceptable range: 10-12 days • Weight: 20%
25. Scoring Using the acceptable range, we arrive at the score: • If metric is better than the range, the score is 1 • If metric is within the range, the score is 0.5 • If metric is worse than the range, the score is 0
26. reldex = Σ(metric_score × metric_weight)
27. Reldex • Excellent > 0.75 • Acceptable 0.5 - 0.75 • Mediocre < 0.5
28. ACCEPTABLE RANGE WEIGHT Overall time to release 10-12 days 20% Time to stabilize RC 2-3 days 20% Time to complete staged rollout 7 days 10% Number of hotfixes during staged rollout 0-2 10% Number of days since last release 5-7 days 20% Number of days between last release & hotfix 1 20%
29. Reldex is relative, not absolute • Your numbers ≠ another team’s numbers • Relative to your team and process • You set the range and weights
30. Not a north star metric for all teams equally!
31. Not same with respect to the metrics either!
32. 🚨 Check Engine Light
33. 🚨 Check Engine Light Great for what? Not for why?
34. Useful over time! Are you working in the right direction?
35. Automatic in Tramline!
36. Open Source Apache 2.0 Calculate it by hand! Build it in your own systems! Send us your ideas and suggestions!