Building and Supporting Open Source Communities through Metrics

Each community is different and therefore requires different metrics for data-driven decisions about building and supporting it. The CHAOSS Project was established to help empower open source communities through metrics. We have had many conversations with contributors and realized that everyone pays attention to different aspects of their communities.

This presentation shares what we have learned in the CHAOSS Project about having metrics for open source communities. Once the right metrics have been decided on, both technical and organizational challenges need to be overcome, which we discuss how to do. Finally, this presentation will describe real-world examples of how metrics have been used to help build and support open source communities.

Transcript

1. Building and Supporting
   Open Source Communities
   Through Metrics
   Open Source Summit
   Europe ‘22
   Georg Link and Emilio Galeano
   

   View Slide

2. At the core of open source
   we care about
   using, sharing and collaborating
   

   View Slide

3. 58%
   of software in Enterprises
   is open source
   63%
   of companies want to
   increase their use
   Source: TODO Group Survey 2021
   

   View Slide

4. ● The heartbleed incident
   ● The Struts/Equifax
   debacle
   ● The Log4j security
   vulnerability
   

   View Slide

5. The Mozilla Foundation released a
   report in 2019 on the different types of
   open source projects
   How can we solve these
   problems?
   

   View Slide

6. using metrics when
   building and supporting
   open source projects
   

   View Slide

7. ● More than 15 years of experience
   ● Maintains the open source GrimoireLab metrics tools
   ● Official Metrics Partner of OpenInfra and NumFOCUS foundations
   ● Co-founded the CHAOSS project in 2017
   

   View Slide

8. Metrics in the
   Framework
   

   View Slide

9. Metrics in the CHAOSS framework
   Common Metrics
   Diversity, Equity, &
   Inclusion Metrics
   Value Metrics Evolution Metrics
   Risk Metrics
   https://chaoss.community/metrics/
   

   View Slide

10. Metrics in the CHAOSS framework
    Common Metrics
    Diversity, Equity, &
    Inclusion Metrics
    Value Metrics Evolution Metrics
    Risk Metrics
    https://chaoss.community/metrics/
    ● Understand what contributions from organizations and people are being made.
    ● With focus areas such as Contributions, Time, People, Place
    ● Type of Contributions: we can measure the types of contributions are being made
    

    View Slide

11. ● Identify the degree to which a project is valuable to researchers and academic institutions.
    ● Academic, communal, individual and organizational value.
    ● Project Velocity: with we can find What is the development speed for an organization?
    Metrics in the CHAOSS framework
    https://chaoss.community/metrics/
    Common Metrics
    Diversity, Equity, &
    Inclusion Metrics
    Value Metrics Evolution Metrics
    Risk Metrics
    

    View Slide

12. ● How the source code changes over time, mechanisms that the project has to perform and control changes.
    ● Code Development Activity, Efficiency, Process Quality, Issue Resolution and Community Growth
    ● New Contributors: How many contributors are making their first contribution to a given project and who are they?
    Metrics in the CHAOSS framework
    https://chaoss.community/metrics/
    Common Metrics
    Diversity, Equity, &
    Inclusion Metrics
    Value Metrics Evolution Metrics
    Risk Metrics
    

    View Slide

13. Metrics in the CHAOSS framework
    https://chaoss.community/metrics/
    ● Identify the diversity, equity, and inclusion aspects at events.
    ● Event diversity, Governance, leadership, and project and community.
    ● Time Inclusion for Virtual Events: organizers be mindful of attendees and speakers in other time zones
    Common Metrics
    Diversity, Equity, &
    Inclusion Metrics
    Value Metrics Evolution Metrics
    Risk Metrics
    

    View Slide

14. Metrics in the CHAOSS framework
    https://chaoss.community/metrics/
    ● Understand how active a community exists around/to support a given software package.
    ● Business risk, code quality, Dependency Risk Assessment, Licensing and Security.
    ● Elephant Factor: with we can measure What is the distribution of work in the community?
    Common Metrics
    Diversity, Equity, &
    Inclusion Metrics
    Value Metrics Evolution Metrics
    Risk Metrics
    

    View Slide

15. Metrics:
    how to interpret them
    

    View Slide

16. New Contributors & Contributions
    An increase or decline in new contributors
    and contributions can be an early indicator of
    community health.
    

    View Slide

17. New Contributors
    https://www.mautic.org/blog/community/q1-2020-mautic-community-round
    

    View Slide

18. New Contributors
    https://www.mautic.org/blog/community/q1-2020-mautic-community-round
    

    View Slide

19. Growth in overall contributions
    https://www.mautic.org/blog/community/q1-2020-mautic-community-round
    

    View Slide

20. Foundation for community growth
    https://www.mautic.org/blog/community/q1-2020-mautic-community-round
    Report Conclusion:
    Q1 2020 has been a busy one in the Mautic Community. A lot of positive
    steps have been taken to establish a solid foundation for growth.
    Teams are becoming more proactive and processes are being established
    which helps the community to function more effectively.
    We have seen, as a result, a significant increase in engagement and have
    welcomed new contributors to the project as a result.
    

    View Slide

21. Organizational Diversity
    Organizational diversity expresses
    how many different organizations are
    involved in a project.
    

    View Slide

22. Organizational Diversity
    https://dri.es/who-sponsors-drupal-development-2020
    

    View Slide

23. Contributions by Organizations
    Who is making
    contributions?
    https://katacontainers.biterg.io/
    

    View Slide

24. Change Requests
    Change requests are intended to be
    reviewed by other developers, who
    may suggest improvements.
    ● Pull Request (GitHub)
    ● Merge Request (GitLab)
    ● Changeset (Gerrit)
    

    View Slide

25. StarlingX
    Change Request Duration
    https://starlingx.biterg.io/
    

    View Slide

26. StarlingX
    Change Request Duration
    https://starlingx.biterg.io/
    

    View Slide

27. StarlingX
    Change Request Duration
    https://starlingx.biterg.io/
    

    View Slide

28. Organizational
    and Technical
    Challenges
    

    View Slide

29. image goes here
    Organizational Challenges
    ➔ Deciding on the right metrics
    ➔ Knowing what to do about metrics
    ➔ Knowing what is good and bad
    ➔ PII* concerns
    *Personally identifiable information (PII) uses data to confirm an individual's identity.
    

    View Slide

30. image goes here
    Technical Challenges
    Where is the data?
    ➢ Git
    ➢ GitHub
    ➢ GitLab
    ➢ BitBucket
    ➢ Jira
    ➢ Gerrit
    ➢ Confluence
    ➢ …
    

    View Slide

31. image goes here
    Technical Challenges
    Where is the community?
    ➢ Wiki
    ➢ Discourse
    ➢ Mailing List
    ➢ IRC
    ➢ Slack
    ➢ Meetup.com
    ➢ StackOverflow
    ➢ …
    

    View Slide

32. image goes here
    How to get the data
    Raw data:
    ● Get data from data sources
    Enriched data:
    ● Unify data
    ● Manage identities
    ● Calculate metrics
    Useful data:
    ● Visualize, report
    

    View Slide

33. image goes here
    Raw data
    ➔ Easiest step
    ➔ Query logs
    ➔ Use APIs
    Challenge: Changing APIs and data formats
    

    View Slide

34. image goes here
    Enriched data
    ➔ Unify data
    ➔ Manage identities
    ➔ Calculate metrics
    ● Date formats
    ● Level of detail
    ● Metadata about different contributions
    ● Convert everything into the desired database structure
    

    View Slide

35. image goes here
    ➔ Unify data
    Enriched data
    ➔ Manage identities
    ➔ Calculate metrics
    ● Who is who in the community
    ● Who do contributors work for (now and before)
    ● Different usernames and email
    ● Assigning contributions to the correct person
    

    View Slide

36. image goes here
    ➔ Calculate metrics
    Enriched data
    ➔ Unify data
    ➔ Manage identities
    ● Primary metrics - summarizing original data
    ● Secondary metrics
    ○ Calculation from different data fields
    ○ Combining data from different data sources
    ○ Value judgements on data - e.g., quality models
    

    View Slide

37. image goes here
    Useful data
    ➔ Who is the data user?
    ➔ How should the data be presented?
    ➔ What visualizations are most meaningful?
    ➔ What story does the data tell?
    

    View Slide

38. image goes here
    Technical Solutions
    OSS Solutions
    ● CHAOSS GrimoireLab
    ○ Cauldron.io
    ● CHAOSS Augur
    ● Apache Kibble
    ● CNCF Dev Analytics
    GrimoireLab
    cauldron.io
    

    View Slide

39. Summary:
    Lessons learned
    

    View Slide

40. Lessons learned
    Summary
    ➔ Use metrics
    ➔ Track metrics early
    ➔ Start with easy to get metrics
    ➔ Present metrics in context
    ➔ Be transparent with the community
    

    View Slide

41. Georg Link
    Questions? Let’s Connect!
    Emilio Galeano
    glink@bitergia.com
    Georg Link
    @georglink
    egaleano@bitergia.com
    Emilio Galeano
    BOOTH LOCATION B24
    

    View Slide

42. Thank You!
    

    View Slide