Building and Supporting Open Source Communities through Metrics

Each community is different and therefore requires different metrics for data-driven decisions about building and supporting it. The CHAOSS Project was established to help empower open source communities through metrics. We have had many conversations with contributors and realized that everyone pays attention to different aspects of their communities.

This presentation shares what we have learned in the CHAOSS Project about having metrics for open source communities. Once the right metrics have been decided on, both technical and organizational challenges need to be overcome, which we discuss how to do. Finally, this presentation will describe real-world examples of how metrics have been used to help build and support open source communities.

Transcript

1. Building and Supporting Open Source Communities Through Metrics Open Source Summit Europe ‘22 Georg Link and Emilio Galeano
2. At the core of open source we care about using, sharing and collaborating
3. 58% of software in Enterprises is open source 63% of companies want to increase their use Source: TODO Group Survey 2021
4. • The heartbleed incident • The Struts/Equifax debacle • The Log4j security vulnerability
5. The Mozilla Foundation released a report in 2019 on the different types of open source projects How can we solve these problems?
6. using metrics when building and supporting open source projects
7. • More than 15 years of experience • Maintains the open source GrimoireLab metrics tools • Official Metrics Partner of OpenInfra and NumFOCUS foundations • Co-founded the CHAOSS project in 2017
8. Metrics in the Framework
9. Metrics in the CHAOSS framework Common Metrics Diversity, Equity, & Inclusion Metrics Value Metrics Evolution Metrics Risk Metrics https://chaoss.community/metrics/
10. Metrics in the CHAOSS framework Common Metrics Diversity, Equity, & Inclusion Metrics Value Metrics Evolution Metrics Risk Metrics https://chaoss.community/metrics/ • Understand what contributions from organizations and people are being made. • With focus areas such as Contributions, Time, People, Place • Type of Contributions: we can measure the types of contributions are being made
11. • Identify the degree to which a project is valuable to researchers and academic institutions. • Academic, communal, individual and organizational value. • Project Velocity: with we can find What is the development speed for an organization? Metrics in the CHAOSS framework https://chaoss.community/metrics/ Common Metrics Diversity, Equity, & Inclusion Metrics Value Metrics Evolution Metrics Risk Metrics
12. • How the source code changes over time, mechanisms that the project has to perform and control changes. • Code Development Activity, Efficiency, Process Quality, Issue Resolution and Community Growth • New Contributors: How many contributors are making their first contribution to a given project and who are they? Metrics in the CHAOSS framework https://chaoss.community/metrics/ Common Metrics Diversity, Equity, & Inclusion Metrics Value Metrics Evolution Metrics Risk Metrics
13. Metrics in the CHAOSS framework https://chaoss.community/metrics/ • Identify the diversity, equity, and inclusion aspects at events. • Event diversity, Governance, leadership, and project and community. • Time Inclusion for Virtual Events: organizers be mindful of attendees and speakers in other time zones Common Metrics Diversity, Equity, & Inclusion Metrics Value Metrics Evolution Metrics Risk Metrics
14. Metrics in the CHAOSS framework https://chaoss.community/metrics/ • Understand how active a community exists around/to support a given software package. • Business risk, code quality, Dependency Risk Assessment, Licensing and Security. • Elephant Factor: with we can measure What is the distribution of work in the community? Common Metrics Diversity, Equity, & Inclusion Metrics Value Metrics Evolution Metrics Risk Metrics
15. Metrics: how to interpret them
16. New Contributors & Contributions An increase or decline in new contributors and contributions can be an early indicator of community health.
17. New Contributors https://www.mautic.org/blog/community/q1-2020-mautic-community-round
18. New Contributors https://www.mautic.org/blog/community/q1-2020-mautic-community-round
19. Growth in overall contributions https://www.mautic.org/blog/community/q1-2020-mautic-community-round
20. Foundation for community growth https://www.mautic.org/blog/community/q1-2020-mautic-community-round Report Conclusion: Q1 2020 has been a busy one in the Mautic Community. A lot of positive steps have been taken to establish a solid foundation for growth. Teams are becoming more proactive and processes are being established which helps the community to function more effectively. We have seen, as a result, a significant increase in engagement and have welcomed new contributors to the project as a result.
21. Organizational Diversity Organizational diversity expresses how many different organizations are involved in a project.
22. Organizational Diversity https://dri.es/who-sponsors-drupal-development-2020
23. Contributions by Organizations Who is making contributions? https://katacontainers.biterg.io/
24. Change Requests Change requests are intended to be reviewed by other developers, who may suggest improvements. • Pull Request (GitHub) • Merge Request (GitLab) • Changeset (Gerrit)
25. StarlingX Change Request Duration https://starlingx.biterg.io/
26. StarlingX Change Request Duration https://starlingx.biterg.io/
27. StarlingX Change Request Duration https://starlingx.biterg.io/
28. Organizational and Technical Challenges
29. image goes here Organizational Challenges ➔ Deciding on the right metrics ➔ Knowing what to do about metrics ➔ Knowing what is good and bad ➔ PII* concerns *Personally identifiable information (PII) uses data to confirm an individual's identity.
30. image goes here Technical Challenges Where is the data? ➢ Git ➢ GitHub ➢ GitLab ➢ BitBucket ➢ Jira ➢ Gerrit ➢ Confluence ➢ …
31. image goes here Technical Challenges Where is the community? ➢ Wiki ➢ Discourse ➢ Mailing List ➢ IRC ➢ Slack ➢ Meetup.com ➢ StackOverflow ➢ …
32. image goes here How to get the data Raw data: • Get data from data sources Enriched data: • Unify data • Manage identities • Calculate metrics Useful data: • Visualize, report
33. image goes here Raw data ➔ Easiest step ➔ Query logs ➔ Use APIs Challenge: Changing APIs and data formats
34. image goes here Enriched data ➔ Unify data ➔ Manage identities ➔ Calculate metrics • Date formats • Level of detail • Metadata about different contributions • Convert everything into the desired database structure
35. image goes here ➔ Unify data Enriched data ➔ Manage identities ➔ Calculate metrics • Who is who in the community • Who do contributors work for (now and before) • Different usernames and email • Assigning contributions to the correct person
36. image goes here ➔ Calculate metrics Enriched data ➔ Unify data ➔ Manage identities • Primary metrics - summarizing original data • Secondary metrics ◦ Calculation from different data fields ◦ Combining data from different data sources ◦ Value judgements on data - e.g., quality models
37. image goes here Useful data ➔ Who is the data user? ➔ How should the data be presented? ➔ What visualizations are most meaningful? ➔ What story does the data tell?
38. image goes here Technical Solutions OSS Solutions • CHAOSS GrimoireLab ◦ Cauldron.io • CHAOSS Augur • Apache Kibble • CNCF Dev Analytics GrimoireLab cauldron.io
39. Summary: Lessons learned
40. Lessons learned Summary ➔ Use metrics ➔ Track metrics early ➔ Start with easy to get metrics ➔ Present metrics in context ➔ Be transparent with the community
41. Georg Link Questions? Let’s Connect! Emilio Galeano glink@bitergia.com Georg Link @georglink egaleano@bitergia.com Emilio Galeano BOOTH LOCATION B24
42. Thank You!