Chasing the White Whale of Open Source - ROI

Have you ever found yourself trying to justify a commitment to an open source project and struggled to communicate the value beyond the usual suspects, such as increased influence or attracting talent? You are not alone. In today’s economic climate, organizations are looking for more concrete returns; they want to know how their investment impacts their business goals. In this talk, Bob will go over some proven successful strategies and tools to help you convey the impact of your organization’s open source investment and provide a lightweight framework that can be used to continue to tell this story over-time and in a sustainable way.

LFMS NA 2024

Transcript

1. Chasing the White Whale of Open Source - ROI
2. • Currently TPgM @ CNCF • Previously PgM @ Google’s OSPO • Previously Academia (16 years) • Emeritus Kubernetes Steering Committee Member & former SIG Contributor Experience Chair Bob Killen 👋 Hi. Contact info: bkillen@linuxfoundation.org GitHub: @mrbobbytables bsky: @mrbobbytabl.es Site: mrbobbytabl.es
3. Bob Killen 👋 Hi. Corporate Contributor Maintainer Hobby Contributor End User • Currently TPgM @ CNCF • Previously PgM @ Google’s OSPO • Previously Academia (16 years) • Emeritus Kubernetes Steering Committee Member & former SIG Contributor Experience Chair Contact info: bkillen@linuxfoundation.org GitHub: @mrbobbytables bsky: @mrbobbytabl.es Site: mrbobbytabl.es
4. Assertion: There is significant unrealized value in Open Source Software.
5. Assertion: There is significant unrealized value in Open Source Software.
6. Background • CNCF End User Company with historically very pro Open Source stance. • Company’s Software Engineers & SysAdmins actively contribute multiple Open Source Projects. • Portion of staff are maintainers of several projects that are used in the Company’s stack. • Recently undergoing significant pressure from leadership to drop / cut back OSS to focus internal initiatives to deliver more “impact” to organization.
7. Understanding the problem Managers/Tech Leads generated a report for leadership about their open source presence. • Overview of what was used internally in their stack. • Included stats about influence and “work” done such as: % contributions, # of maintainers, commits. • Included information about what features being developed upstream were being driven by company.
8. Understanding the problem Managers/Tech Leads generated a report for leadership about their open source presence. • Overview of what was used internally in their stack. • Included stats about influence and “work” done such as: % contributions, # of maintainers, commits. • Included information about what features being developed upstream were being driven by company. Not giving leadership what they want
9. Understanding the problem - What is the criticality of the projects they are maintaining? - What are we getting out of contributing to the project? - How much SWE time is ACTUALLY being spent on maintaining the projects?
10. Bug Fix Stats (prev. year) Bug Statistics (prev. year) Total bugs 55 Submitted by Company members 11 Company bugs fixed by Company members 6 Mean time to fix Company member bugs 3~ days GitHub Query: is:issue closed:2023-01-01..2023-12-31 label:kind/bug • Filter for company contributors: author:<member1>...author:<member6> • Used GitHub API to query created_at and closed_at for each issue created by compantry contributors. • Scriptable to generate regular report using GitHub GraphQL API.
11. Company Activity (prev. year) Company SWEs: 6 Average SWE time allocation: 10% Total SWE allocation: .6 Company Activity vs Others Project Committers: 54 Total Project Contributions: 4663 Company Contributions: 633 % Company Contributions: 14%
12. Outcomes & Lessons Learned Presenting information to leadership in ways they understand and align with the kind of decisions they were making, worked very well. Tying projects to their stack, and what services it enabled. Let leadership understand the criticality of those projects, and something to “weigh” the SWE investment.
13. Making it Count A framework for quantifying the ROI of contributing to open source
14. The Open Source Pitfall Many orgs have NO overall open source strategy (or it’s limited to licensing & compliance) Employees are frequently encouraged to contribute to OSS directly or indirectly without proper guidance to tie it back to value, and this creates a negative feedback loop: • Employees encouraged to contribute • Value & Impact is not understood; leadership asks: “Why are we spending time on something that doesn’t help us? ” • Employees told to spend less time on open source, but they understands the impact and feel unrecognized and undervalued; becomes burned out. • Both the project AND the organization begin to suffer.
15. Making it Count • What are your goals? • What matters to your organization? • How healthy are the projects that you use? • What resources do you have? • Tracking & framing impact
16. What are your goals? What do you want to get out of contributing to open source?
17. TL;DR… Prioritizing investments
18. TL;DR… Priority = Criticality + Health Prioritizing investments (Business Risk & Opportunities)
19. What matters to your organization What are you using? • A full software inventory is more essential with growth in supply chain attacks (who doesn’t just love~ doing a full software inventory) How critical is it to your stack or product? • How difficult would it be to switch to something else? Or fork and maintain it? • Are there features that are important to you being developed? If it went in another direction, how much would it impact you? Example Criticality Levels High (3) • Critical to core business function • Extremely difficult to swap or maintain internally • Difficult to backfill expertise • Roadmap has features that would be very beneficial to org Moderate (2) • Software that supports core functionality • Could be swapped out with reasonable effort • Easy to backfill expertise Low (1) • Non-essential tools/apps with minimal business impact • Could be swapped out with minimal effort (API compatible)
20. Description Questions Organization Dependency How critical is the project to your organization? • Critical: Core component to business function or initiative. • Operational: Supportive to critical; does not disrupt core business function. • Administrative: Provides a useful function such as automation, but has no direct impact on business function. • What are downstream impacts? projects, products, services, or teams • How difficult would it be to switch to something else? Or fork and maintain internally? And what would that cost be? • What would the impact of an unplanned security event be? Development Opportunities What opportunities are there to drive or impact the roadmap of the project to better suit business needs? • Does the project currently have features or initiatives that would be beneficial? If yes, how so? • Could development create a competitive advantage? Supportability How difficult is the project to support? • How easy is it to support? Is it well documented? • How difficult is it skill-up employees or backfill on expertise? Cost Management How does the project help with managing costs? • Does the project help you reduce or manage costs? (e.g. right-sizing pods) If so, by what factor? • Would using the project and committing resources to it be a better option than a vendored option? • Does using the project enable you to pick from multiple solutions? Brand Affiliation & Marketing How important is it to be associated with the project? • Do we want to be seen as leaders in this space? Or strongly associated with it? • Are there specific outlets or demographics we want to reach? How can we measure reach? Ecosystem Potential Does the project create or support an ecosystem that is important to the business? • Do we have the resources to capitalize on creating an ecosystem? • What kind of benefit would it bring our projects, products, or services?
21. How healthy are the projects that you use? Quality codebase Responsive to issues / PRs Healthy Contributor Diversity Has goals and a roadmap Quality documentation for both users & contributors Good security and releasing controls What is a healthy project?
22. How healthy are the projects that you use? Quality codebase Sound architectural design, static code analysis, sufficient testing Responsive to issues / PRs Time to first response / review Healthy Contributor Diversity No single vendor drives the whole project Has goals and a roadmap They are planning, and have a design review process Quality documentation for both users & contributors Docs are critical for both adoption and contributor growth Good security and releasing controls Do they have a history of triaging and resolving security issues, are they using/investigating supply chain security best practices What is a healthy project?
23. What matters to your organization Project Org Dependency Project Opportunities Supportability Health Foo (32) • Score: 10 • Critical to core business function • Migrating to an alternative would be extremely difficult • Score: 7 • Roadmap has very useful features • High barrier of entry to contribute and will require more time to engage • Score: 8 • Extremely difficult to backfill expertise • Not well documented internally • Score: 7 • Project has a large contributor base, but few senior maintainers • Contributor ladder needs work Bar (24) • Score: 6 • Not critical to business function • Score: 9 • Project is a better option than <Baz> and easy to drive features • Popular, strong marketing opportunity to be associated with it • Score: 6 • Easy to backfill expertise • Could be swapped out with reasonable effort • Score: 3 • Project has a good contributor base and strong pipeline 25-30 Critical 20-25 High 15-20 Moderate 10-15 Low 0-10 Negligible
24. What resources do you have? Organizations have much more than just Software Engineers that can help projects. They also need people with these skills: • Triage & Program/Product Management • Tech Writing • Communications & Marketing • Event Management If you can’t commit your own people. You can help with donating funds or hiring contractors to help in these areas.
25. Priority = Criticality + Health Putting it together How healthy is the project? Does it need support? How much do we depend on it? What opportunities are there for us in the project? How does it fit into an overall strategy? Number to help you prioritize where you have the largest potential business impact. Resources you have available
26. Tracking & Framing Impact
27. Tracking & Framing Impact Is this worth the resource investment vs. maintaining internally or accepting the risk. Most common pitfall: Organizations bias towards easy to measure metrics such as total contributions, but these do NOT hold up under scrutiny. Metrics should support your goals and be able to be tied back to “value” and the types of resources you are investing in the project. • Are the features we’re interested in progressing? • Are the issues we’re concerned about being addressed? • Is stability being improved? Bugs fixed? • Is the project itself healthy?
28. Features & Initiatives How to determine and prioritize what features & initiatives to track Issues & PRs Tracking impact of code contributions to business priorities Project Health Focus areas Tracking overall project health and how it can benefit your organization
29. Classifying features & initiatives Invest Direct benefit to organization that warrants allocating a high level of staff time to drive the initiative. Frequently these are important features to organization implementation or initiatives deemed important to overall health of the project. Support Initiative driven by another entity, but is beneficial to organization. Worth allocating time to review and be involved in a supportive method. Watch Initiative driven by another entity that is not currently relevant, but has potential to either be disruptive or beneficial depending on implementation. Discourage Potential feature, initiative, or request that could be harmful or introduce undue amounts of complexity that could reduce stability or make it much more difficult to support. Ignore No potential benefit or impact to how the project is used by your organization.
30. Issues & PR KPIs Type Metric Source Bugs • Bugs reported vs resolved • Time to resolution • What % are resolved by your org vs others Bugs closed this year (GitHub Query): is:issue closed:2024-01-01..2024-11-01 label:kind/bug Security • Security issues reported vs resolved • Time to resolution Time to resolve security issues (GitHub cli tool): gh issue list –search “is:closed label:security” –json “id,openedAt, closedAt” Issue/PR • Time to resolution • What % are resolved by your org vs others Queries can be tailored with author: keyword look at company created items to track own items vs. project as a whole. Issues/PR queries can be tailored to track initiatives
31. Project Health KPIs Project health is crucial to de-risk usage and can drive direction if a project is unhealthy. • Organization Diversity • Contributor Growth & Retention • PR & issue velocity • Adoption • Contributor & adopter sentiment • Documentation & frequency of support requests
32. Triage & Program Management Review incoming issues, assign labels and priorities and aid in roadmap planning. Org Benefits • Prioritized triage of issues & bugs • Decreased time to resolve • Early awareness of potential breaking changes & security issues • De-risk usage of project by improving overall project health • Introduces better data to answer questions & track trends. KPIs • Decreased time to first response on issues • Decreased time to assignment • Decreased issue/PR open time Long Term Health KPIs • Increased contributor engagement: unique #, frequency, retention • Positive sentiment on issues/PRs and other communication channels
33. Triage & Program Management Review incoming issues, assign labels and priorities and aid in roadmap planning. Org Benefits • Prioritized triage of issues & bugs • Decreased time to resolve • Early awareness of potential breaking changes & security issues • De-risk usage of project by improving overall project health • Introduces better data to answer questions & track trends. KPIs • Decreased time to first response on issues • Decreased time to assignment • Decreased issue/PR open time Long Term Health KPIs • Increased contributor engagement: unique #, frequency, retention • Positive sentiment on issues/PRs and other communication channels Gives you much more data to track impact • Labels provide context: prioritization, scope, state etc. • You cannot answer questions like “has there been a decrease in bugs since we engaged” unless the issues and PRs are actually labeled.
34. Documentation Well-documented projects are easier for users to understand & utilize - aiding growth and retention. Org Benefits • Better developer experience • Increases ability to self-service / decreased engineering time • De-risk usage of project by improving overall project health “Having good docs leads to better docs” KPIs • Decreased support questions opened • Increased site traffic & accompanying site metrics Long Term Health KPIs • Increased new contributor engagement & retention • Positive sentiment on issues/PRs
35. Comms & Marketing Projects often lack both the knowledge and skills regarding communication & marketing best practices. Example: A critical CNCF project was in desperate need of maintainers; but they only raised the issue within private circles and on twitter - netting zero growth. Org Benefits • Improved brand awareness & association • De-risk usage of project by driving more adoption & conversion to contributors • Early awareness of potential breaking changes KPIs • Increased traffic to website & accompanying site metrics • Increased social engagement* • Increased share of voice Long Term Health KPIs • Growth in adoption - downloads, GitHub stars, share of voice etc. • Positive sentiment on communication channels
36. Events Events serve many purposes, from new contributor workshops to help with onboarding, to summits with many high-bandwidth conversations that can unblock development. Org Benefits • Workshops can serve as onboarding or additional skill-up opportunities • Being “in-the-room” can ensure org priorities are addressed • De-risk usage of project by improving overall project health KPIs • Tracking contributions from attendees • Conversions from attendees / contributors to maintainers • Unblocked issues/features Long Term Health KPIs • Increased contributor engagement - unique #, frequency, retention • Increase in conversion from contributors to maintainers
37. Project Total Bugs Total bugs resolved Bugs submitted by org Org bugs fixed % bugs fixed by others Avg. time to fix bug Security issues reported Security issues fixed Avg. time to fix sec. Foo (32) 41 32 11 11 36% 3 days~ 3 3 2 days Bar (24) 21 19 7 7 57% 4.5 days~ 1 1 1 day Foo (32) - 5 SWEs @ 20% - 1 SWE/quarter • Opportunities ◦ Feature <baz> that will improve our developer productivity is set to release next month. Should be deployed internally 2 weeks post release. • Supportability ◦ Hired maintainer John Doe, investing time in mentoring dev team in <Foo> • Health ◦ Health is improving. Implemented triage best practices and time to first response has been cut down to 24hrs Bar (24) - 2 SWEs @ 20%, 1 PgM @ 30% - .7 FTEs/quarter • Opportunities ◦ Feature <qux> has gained support and has begun development, will ensure our product supports new security standard requested by customers - $Xm/year ◦ BarCon took place last month; between our speakers and presence at the event our share of voice has outpaced all other vendors investing in <Bar>. • Supportability ◦ Project is overhauling docs with support from a TW hired by <quux> • Health ◦ Project remains healthy. <BarCon> has brought more interest in contributing.
38. A lot more to cover Hiring & retention… Knowledge transfer… Managing changing priorities… Divesting from a project without impacting it…
39. Just a fraction of potential OSS investment can be tracked and tied back to business value effectively. It requires the right framing and ensuring your resources are allocated to where they may make the most impact.