Why is this so HARD! Conveying the Business Value of Open Source

KubeCon EU 2024

Are you a maintainer struggling to justify your continued time working on an OSS project? Or are you an end-user trying to determine if you should use or support a project? Or how about a business trying to map your employee's contributions to internal initiatives? These struggles are prevalent throughout the ecosystem and often boil down to ineffectively conveying the "value" and "risks" of a project. In this talk, we'll cover: - Why communicating a project's value & risks is essential for project growth and long-term sustainability. - Strategies projects can adopt to better surface their value & risks. - Several unconventional methods for end-users and vendors to evaluate their contributions. - Highlight new non-code opportunities for people to get involved.

Transcript

1. Why is this so Bob Killen @mrbobbytables bob.killen@linux.com Conveying the Business Value of Open Source HARD!
2. • Program Manager @ Google OSPO • Previously Academia (16 years) • Kubernetes Steering Committee Member • Former Chair of Kubernetes SIG Contributor Experience Bob Killen @mrbobbytables 👋 Hi.
3. 👋 Hi. • Program Manager @ Google OSPO • Previously Academia (16 years) • Kubernetes Steering Committee Member • Former Chair of Kubernetes SIG Contributor Experience Corporate Contributor Maintainer Hobby Contributor End User Bob Killen @mrbobbytables
4. Why is this so Conveying the Business of Open Source HARD! Value Bob Killen @mrbobbytables bob.killen@linux.com
5. CNCF Contributions CNCF Devstats - Companies Table (All Projects - 1yr) 95.8% Affiliated 4.2% Independent Other Organizations (not VMware)
6. CNCF Contributions CNCF Devstats - Companies Table (All Projects - 1yr) 95.8% Affiliated 4.2% Independent Contributions Maintainers 94.7% Affiliated 5.3% Unaffiliated CNCF project-maintainers.csv Other Organizations (not VMware)
7. 52% Contributed in both personal and professional time. 30% Contributed in professional time only. What brings you to open source? - August 2023 Personal or Professional? 18% Contributed in personal time only.
8. TL;DR MOST contributors are contributing on behalf of an organization. MOST contributors LIKE and WANT to contribute more to OSS.
9. TL;DR MOST contributors are contributing on behalf of an organization. MOST contributors LIKE and WANT to contribute more to OSS. And more importantly… To get PAID for it!
10. Explaining the Business Value Trying to make a bunch of random connections about influence, reducing risk of use, and trying to attract/retain talent… Can Feel Like…
11. Explaining the Business Value Trying to make a bunch of random connections about influence, reducing risk of use, and trying to attract/retain talent… Can Feel Like… ...but that frequently doesn’t hold up when pressed for why you should contribute vs. focusing on other initiatives.
12. Data-Enablement Communication Ensuring the right data is available to support OSS investments. Providing context in a human-friendly manner & telling the story of the data.
13. Data Enablement aka label all the things
14. Insufficient Data The Problem with
15. Insufficient Data 50% Labeled Issues Consistently 17% Labeled PRs Consistently 58% Used Milestones Source: • 12 random CNCF projects • 4 from maturity tier
16. The Importance of Labels Labels can • Provide context • Prioritization • Scope • State • … Answer Questions & Track Trends
17. Examples Question: How many bugs were in the 1.30 Kubernetes release? Query: is:pr label:kind/bug milestone:v1.30 Answer: 180 Question: How are we progressing on tech-debt in SIG-Node? Query: label:kind/cleanup label:sig/node milestone:v1.30 Answer: 71
18. Example: Bug Trends is:pr label:kind/bug milestone:v1.XX Question: Has Kubernetes Stability improved over the past 6 releases? (2 years)
19. Example: Bug Trends is:pr label:kind/bug label:kind/regression milestone:v1.XX Question: Have there been fewer regressions in newer Kubernetes releases?
20. Example: Bug Trends is:pr label:kind/bug label:kind/api-change milestone:v1.XX Question: Have there been fewer regressions in newer Kubernetes releases?
21. Example: Bug Trends is:pr label:kind/bug label:area/kube-proxy milestone:v1.XX Question: Have there been fewer regressions in newer Kubernetes releases?
22. Which is easier?
23. Milestones Milestones offer an incredibly convenient way to associate issues/PRs into a logical grouping (e.g. releases). milestone:v1.30 created:2024-03-01..2024-03-05 VS Items opted into milestone Glob, queries everything between date range
24. Why Projects Don’t Triage Why are so many projects inconsistent with labels and milestones? • Unaware of GitHub Triage role and believe write access is still required to apply labels & milestones. • Don’t think they actually matter and are just another piece of “overhead” • Too small a project to matter or warrant use.
25. Helpful Triage Actions Label Actions • Copy Issue Label - Sync issue label to PR following “fixed”, “solved” etc. • Pull Request Labeler - Applies labels to PRs based on glob patterns. • Label Syncer - Declarative way to ensure labels are consistent across repos/orgs Project Actions • Add To GitHub Projects - Automatically adds Issues / PRs to project boards based on labels used.
26. Triage With Prow Prow Prow fundamentally encourages label use, and can auto-apply labels via OWNERS files. Prow-github-actions Utilities action that mimics Prow functionality. Example Functions • /milestone <milestone> • /kind <bug|feature|etc> • /area <component> • /label <label name> • /approve & /lgtm
27. Data Conclusions Consistent data from labels & milestones is crucial for surfacing the state of a project and enabling both contributors and businesses to evaluate the impact of investment. Data provides the evidence to back up your messaging to leadership. Recruiting people to help with triage and using tools to automate the process can ensure things are properly labeled and milestoned.
28. Communication Back to first principles
29. OSS Enthusiast • Knows how to navigate OSS Ecosystem & projects well • Does not mind diving deep into technical details • May be participating as a hobby or as part of their job duties
30. OSS Enthusiast (Maintainer) • Knows how to navigate OSS Ecosystem well projects well • Does not mind diving deep into technical details • May be participating as a hobby or as part of their job duties • Burned Out, does not understand why more people don’t contribute • Does “Hero” work to keep project(s) going • …does not like to do project management / non-code type work …generally
31. Leadership (VPs, SVPs, C*Os) Concerned with resources, health and growth of company. Focused on Health of business, opportunities & risks. Prioritizes Return of Investment. Least technical. Product Managers Prioritizes making the best product. Skilled at conveying technical details in user friendly fashions. Desires customer/user feedback. End User Prioritizing how tools, features, etc can benefit them / their organization. Diverse range of technical skill. Bombarded by options. Managers / Leads Concerned about employees and meeting org objectives. Better understanding of tech and knowledge of what team is working on. Intermediary between team and leadership.
32. Limiting growth If it takes unnecessary additional cognitive overhead to: - Understand in basic terms what that feature does - Find your roadmap or the features & initiatives you’re working on - Know what parts of the project/feature/initiative are at risk You are losing potential users, future contributors, and making it more difficult to justify further investment of resources (e.g. headcount)
33. Example 1 Before: “This KEP proposes an automatic topology aware hinting mechanism that would provide a way for EndpointSlice producers to indicate where consumers should use specific endpoints.”
34. Example 1 Before: “This KEP proposes an automatic topology aware hinting mechanism that would provide a way for EndpointSlice producers to indicate where consumers should use specific endpoints.”
35. Example 1 After: “Topology Aware Routing adjusts routing behavior to prefer keeping traffic in the zone it originated from. In some cases this can help reduce costs or improve network performance.” Before: “This KEP proposes an automatic topology aware hinting mechanism that would provide a way for EndpointSlice producers to indicate where consumers should use specific endpoints.”
36. Example 2 Before: “This KEP proposes to allow mutating spec.completions for Indexed job if spec.completions equals to spec.parallelism before and after the update, meaning that spec.completions is mutable only in tandem with spec.parallelism.”
37. Example 2 After: “Enables autoscaling for Indexed Jobs” Before: “This KEP proposes to allow mutating spec.completions for Indexed job if spec.completions equals to spec.parallelism before and after the update, meaning that spec.completions is mutable only in tandem with spec.parallelism.” I can get behind that!
38. Surfacing Information Information regarding roadmaps, features, and their state should be easily accessible and written for a wide audience. • Regular Project Reports / Blog posts • Roadmap on website • GitHub Project Board
39. Project Boards • Project Boards are highly automatable via graphQL API • Support for multiple views • Exportable as tsv Kubernetes Release Team Board Automation Scripts
40. Status & Roadmap Issue Description Status Notes Target Tracking Issue - “Friendly” short description. - Parsed from tracking issue body - Alpha/Beta/Stable - Red/Yellow/Green - Optional notes to describe status - Parsed from tracking issue body Target Release • Discoverable and easy to link to • Provides just enough information to convey feature / initiative state & target release • Status field can be used to signal things that are at risk • Easily mappable to internal tracking information
41. Talking to Leadership • How critical is the project? Can you associate $$ or level of risk with it. • How many SWE hours are being spent? • What is the business getting in return for those SWE hours? What happens if you stop? ◦ List features, bugs - both driven by you, and ones that your org uses that you influenced, and others that were completely developed by others. ◦ Try to associate with anything internal and $$ Leadership (VPs, SVPs, C*Os) Concerned with resources, health and growth of company. Focused on Health of business, opportunities & risks. Prioritizes Return of Investment. Least technical.
42. Managers & Users • What is the roadmap? • What is the status of items on the roadmap? And the things directly impacting us. • What features / things are at risk? ◦ If at risk, what is needed? • What are the blockers / bugs? • Managers: what is my team involved in?
43. Communication Summary How you communicate value is just as important as the metrics you use to back it up. They are complimentary, and should be used together. Understanding your audience and communicating with them in terms & language they are accustomed to can help you maximize the impact of what you are presenting. GitHub project boards are an excellent tool to use to for triage and sharing information about the status of enhancements, or used as a roadmap.
44. Based on True Events
45. Based on True Events The Troubled End User
46. The Troubled End User • CNCF End User Company with historically very pro Open Source stance. • Company’s Software Engineers & SysAdmins actively contribute multiple Open Source Projects. • Portion of staff are maintainers of several projects that are used in the Company’s stack. • Recently undergoing significant pressure from leadership to drop / cut back OSS to focus internal initiatives to deliver more “impact” to organization.
47. Understanding the problem Managers/Tech Leads generated a report for leadership about their open source presence. • Overview of what was used internally in their stack. • Included stats about influence and “work” done such as: % contributions, # of maintainers, commits. • Included information about what features being developed upstream were being driven by company.
48. Understanding the problem Managers/Tech Leads generated a report for leadership about their open source presence. • Overview of what was used internally in their stack. • Included stats about influence and “work” done such as: % contributions, # of maintainers, commits. • Included information about what features being developed upstream were being driven by company. Not giving leadership what they want
49. Understanding the problem - What is the criticality of the projects they are maintaining? - What are we getting out of contributing to the project? - How much SWE time is ACTUALLY being spent on maintaining the projects?
50. Bug Fix Stats (prev. year) Bug Statistics (prev. year) Total bugs 55 Submitted by Company members 11 Company bugs fixed by Company members 6 Mean time to fix Company member bugs 3~ days GitHub Query: is:issue closed:2023-01-01..2023-12-31 label:kind/bug • Filter for company contributors: author:<member1>...author:<member6> • Used GitHub API to query created_at and closed_at for each issue created by compantry contributors. • Scriptable to generate regular report using GitHub GraphQL API.
51. Company Activity (prev. year) Company SWEs: 6 Average SWE time allocation: 10% Total SWE allocation: .6 Company Activity vs Others Project Committers: 54 Total Project Contributions: 4663 Company Contributions: 633 % Company Contributions: 14%
52. Outcomes & Lessons Learned Presenting information to leadership in ways they understand and align with the kind of decisions they were making, worked very well. Tying projects to their stack, and what services it enabled. Let leadership understand the criticality of those projects, and something to “weigh” the SWE investment.
53. Based on True Events The Struggling Maintainer
54. Struggling Maintainer • Struggling maintainer was the only active maintainer left, others all drifted away from project. • Could not adequately stay on top of issues/PRs. • Pressure from $dayjob to spend time elsewhere. Project
55. Understanding the Problem • No easily discernible way to see state of project. • No easy way to follow up if someone could commit serious resources.
56. Understanding the Problem Not communicating the problem! • No easily discernible way to see state of project. • No easy way to follow up if someone could commit serious resources.
57. Actions Taken • Updated Issue/PR template, and repo README communicating project is down to 1 maintainer and is at risk of going unmaintained. • Reached out to known users/companies that have a dependency on project and highlighted risks & opportunities. • Offered mentoring to those that could commit resources to become future maintainers.
58. Focus: Outreach Reached out to known users/companies that have a dependency on project and highlighted risks & opportunities. • # of open bugs + rate of being addressed • Looked at open feature requests • Drafted “job description” for maintainer role • Estimated weekly SWE hours for “maintenance” once onboarded • Proposed mentorship to onboard if they can commit resources.
59. Focus: Mentoring Offered mentoring to those that could commit resources to become future maintainers. • Convinced maintainer’s employer to let them spend more time on the project for a short period of time to on-ramp others to de-risk their continued usage of the project and let them spend more time on other things long-term. • 3 month mentor cohort of 8~ people that met “job description” from various vendors and users with the goal of getting 3~ people who could take on maintainer duties.
60. Outcomes & Lessons Learned Framing the problem around what’s important to the company, and conveying it in a way those not “close” to the project understand was essential to getting buy-in. Providing some base metrics (bugs, issues etc) as a starting point, ensured everyone had a way to follow progress and track success.
61. Putting it all Together There is significant unrealized value in Open Source Software. Projects can begin to surface that value by following good triage practices and surfacing information about their state in more accessible ways to support the maintainers and the businesses behind them to justify their investments.
62. Thanks for listening to me ramble for 30min <3 Want to follow up? bob.killen@linux.com @mrbobbytables https://mrbobbytabl.es Plz may I have some feedback 🙏