Code Health Score: How Slack Handles Tech Debt at Scale

All codebases have technical debt. Sometimes we reluctantly pass by leaving a TODO to clean it up later and sometimes we act on the urge to go on refactoring binge to clean it up. Addressing technical debt can be rewarding and useful, but how can we make sure that it is not simply left to the whim of a good samaritan? And can such work be consistently seen and encouraged by engineering leadership?

Following the trope of "you can't improve what you don't measure", at Slack, we have employed code health score: an empirical measure that provides visibility into cumulative and team-based technical debt to address these questions. But measurement alone isn't enough; it's equally important to integrate such measurement into the day-to-day engineering culture of the team.

This talk provides the formula for building a code health score for any codebase, touches on the nuances of its implementation on Android/iOS, and shares lessons learned about integrating this tool into the engineering process.

Transcript

1. CODE HEALTH SCORE HOW HANDLES TECH DEBT AT SCALE @valera_zakharov
2. TECH DEBT? NEVER HEARD OF IT… ~ No Engineer. EVER.
3. WE ARE HERE TO SHIP FEATURES TO USERS
4. A HEALTHY CODEBASE ALLOWS US TO SHIP FASTER AND SAFER
6. AT SCALE, AD-HOC CLEAN UP IS NOT ENOUGH
11. 3RD-PARTY SOLUTIONS
12. SIMPLE HEALTH SCORE RECIPE ▸ Count problems ▸ Multiply each by how bad they are ▸ The score = 0 – (the sum of weighted problems)
13. A NAIVE IMPLEMENTATION ▸ Examine each fi le in the codebase ▸ For each check, return weighted score ▸ Aggregate by fi le & upload to a backend ▸ Display in a pretty dashboard
14. NAIVE IMPLEMENTATION HEALTH_SCORE_CHECK INTERFACE
15. INTRODUCING A NEW CHECK = DEPLOYING NEW CODE
16. INTRODUCING A NEW CHECK REQUIRES DEPLOYING NEW CODE DECLARATIVE CONFIGURATION + USAGE_CHECK
17. CAN WE DO BETTER THAN STRING REGEX MATCHING?
18. CAN WE DO BETTER THAN STRING REGEX MATCHING? BUILD ON TOP OF STATIC ANALYSIS TOOLS ▸ Can work with Abstract Syntax Tree for more sophisticated checks ▸ Example: Android Lint
19. CAN WE DO BETTER THAN STRING REGEX MATCHING? ANDROID LINT .sarif fi le
20. CAN WE DO BETTER THAN STRING REGEX MATCHING? V1 ANDROID HEALTH SCORE: BUILT ENTIRELY ON LINT “SickFile.kt”: { “DeprecatedCall”: -100, }
21. CAN WE DO BETTER THAN STRING REGEX MATCHING? V1 ANDROID HEALTH SCORE ▸ Advantages ▸ Easy to wire up dependencies (lint must run before health score) ▸ Ecosystem around lint ▸ e.g. violations show up in IDE ▸ Issues ▸ New check = new lint rule. Hard. ▸ Platform/build-system speci fi c
22. CAN WE DO BETTER THAN STRING REGEX MATCHING? V2: INTEGRATE LINT INTO EXISTING MODEL
23. MY CODEBASE IS LARGE. EXAMINING EACH FILE IS COSTLY.
24. MY CODEBASE IS LARGE HELP CRAWLER DO LESS WORK WITH FILTERS TAKES ABOUT 44SEC TO CALCULATE HEALTH SCORE FOR OUR ANDROID REPO OF ~16K KOTLIN FILES WITH 1.3M LOC
25. BUT IT WASN’T ME WHO MADE THIS MESS!
26. TECH DEBT BY FUNCTIONAL AREA CODE OWNERSHIP ▸ Every fi le in codebase is mapped (ensured by breaking PR check)
28. CAN WE PREVENT THE MESS IN THE FIRST PLACE?
29. PREVENTING NEW TECH DEBT FROM ENTERING THE SYSTEM HEALTH SCORE IMPACT ON PULL REQUESTS: V1 ▸ On dev branch build ▸ Get a list of [modi fi ed, added, deleted] ▸ Calculate local score for only these fi les ▸ Check out main branch and do the same ▸ Diff main score vs dev score ▸ Report to GitHub via PR comment
30. PREVENTING NEW TECH DEBT FROM ENTERING THE SYSTEM PROBLEMS WITH V1 ▸ Doesn’t work for lint checks (sarif output fi les are not in source control) ▸ Doesn’t show diffs if health score itself changes
31. PREVENTING NEW TECH DEBT FROM ENTERING THE SYSTEM HEALTH SCORE IMPACT ON PULL REQUESTS: V2 ▸ Store health score output from main branch build in a json payload ▸ On dev build ▸ Download latest main branch health score ▸ Run health score locally ▸ Diff main vs dev ▸ Post to GitHub
32. HOW DO WE ALIGN HEALTH SCORE WITH CURRENT PRIORITIES?
33. ALIGNING HEALTH SCORE WITH CURRENT PRIORITIES LIST OF CHECKS ▸ Best if decided collectively by codebase contributors ▸ May take time to align ▸ Better to start the conversation and optimize ▸ Some ideas ▸ Disabled tests/lints ▸ Deprecated API usage ▸ A11y violations ▸ Number of fi les in app module (build time bottleneck) ▸ Lines of objc code (when migrating to swift)
34. ALIGNING HEALTH SCORE WITH CURRENT PRIORITIES THOUGHTS ON WEIGHING ▸ How bad is this thing? ▸ How hard is it to fi x? ▸ How much do you want people to fi x it?
35. ALIGNING HEALTH SCORE WITH CURRENT PRIORITIES LITMUS TEST
36. WE ADDRESS TECH DEBT, BUT THE TARGET KEEPS MOVING!
37. PREVENT MOVING TARGET FREEZE HEALTH SCORE RECIPE FOR A PERIOD (E.G. QUARTER) ▸ Maintain 2 (or more) versions at the same time ▸ [Current]: used to track progress for this quarter ▸ [Next]: used to experiment with new recipe for next quarter
38. HEALTH SCORE IS JUST A TOOL 16 oz. Jacketed Fiberglass Claw Hammer available for $10.45 $15 at Home Depot
39. USING HEALTH SCORE WEEKLY REPORTS
40. USING HEALTH SCORE ENGINEERING LEADERSHIP ENGAGEMENT
41. USING HEALTH SCORE LEAN INTO EXISTING PROGRAMS BETTER ENGINEERING
42. USING HEALTH SCORE LEAN INTO EXISTING PROGRAMS BETTER ENGINEERING
43. USING HEALTH SCORE USE HEALTH SCORE TO TRACK LARGE MIGRATIONS
44. USING HEALTH SCORE IT’S A SCORE. THERE ARE POINTS. MAKE IT A GAME?
45. DEVELOPERS CARE ABOUT CODE HEALTH