Metrics: When One Size Doesn't Fit All

There are different ways to assess an open source project’s success, but measurable metrics (like pull requests and contributions) typically play some role. One of the challenges Uber has faced involves maintaining many open source projects of different sizes and maturity levels. It’s not possible to create one universal metrics baseline for all projects and gauge individual project metrics against it. In this presentation, Carsten Jacobsen, Open Source Developer Advocate at Uber, and Manrique Lopez, CEO at Bitergia, talk about how different metrics are captured and evaluated for different open source projects at Uber, and how domain knowledge and analytics expertise, when working together, play a key role in providing insights in a compelling way.

Transcript

1. Metrics: When one size doesn’t fit all August 21, 2019 Jose Manrique López de la Fuente CEO, Bitergia Carsten Jacobsen Open Source Developer Advocate, Uber
2. Repositories on GitHub 350+ Contributors worldwide 2,000+ Community projects: Jaeger, Horovod, Pyro, Apache Hudi, Kepler.gl 5 2 of the top open source projects on InfoWorld’s awards list 2 Stars on GitHub. Average 365 per repository 117K+ High level metrics from Uber’s Open Source program
3. Repositories on GitHub 350+ Contributors worldwide 2,000+ Community projects: Jaeger, Horovod, Pyro, Apache Hudi, Kepler.gl 5 2 of the top open source projects on InfoWorld’s awards list 2 Stars on GitHub. Average 365 per repository 117K+ High level metrics from Uber’s Open Source program Open Sourced one project every week since 2012!
4. Should we care about project metrics? Introduction
5. Absolutely! There are at least a few good reasons: • Open sourcing a project is just as much a business decision as an engineering decision. We must be able to prove the value. • Decisions should always be made based on facts - and facts are a product of metrics and analytics • The insights we can gain from metrics can make us better project maintainers. Introduction
6. Data, data, data
7. Without data, you are just another person with an opinion W. Edwards Deming
8. If we cast our net wide, we will probably get a lot of useful data... Data, data, data
9. We can capture all kinds of data Data, data, data Chats Slack Mattermo st Telegram Tickets/Issues GitHub GitLab Bugzilla Jira Mailing lists Mailman Groups.io G Groups Events Meetup Eventbrite Q&A Forums Askbot StackOverflow Discourse Documentation Read the docs Confluence MediaWiki Code Review Gerrit GitHub GitLab Coding Git Mercurial Bazaar SVNi
10. With Bitergia Analytics, we can even look at all the data Data, data, data
11. But what do we do with all that data? Data, data, data
12. Strategy without tactics is the slowest route to victory. Tactics without strategy is the noise before defeat. Sun Tzu Data, data, data
13. Examples of relevant questions to seek answers to: • How does the project do? • What is the project’s the value? • How big is the project community? • Should we continue developing it? • How can we improve the project? Data, data, data
14. Initial Learnings
15. Working with Uber’s Open Source project metrics has shown a clear trend. All projects cannot be treated the same way. We have to ask different questions for projects that are in different stages of their lives. Initial learnings
16. Projects have different maturity levels, and if we use the same metrics to assess their success, we will typically not get the right picture of less mature projects. Initial learnings
17. Initial learnings New Project Mature Project Issues Mailing lists Mailman Groups.io G Groups Stars Engage- ment Clones Contri- butors Pull Request
18. Initial Learnings New Project Issues Mailing lists Mailman Groups.io G Groups Stars Clones While popularity metrics like stars may not have much value for mature projects, it’s a good indicator of interest in newer projects. Another good indicator for newer projects is how many times developers have tested the project - measured in clones, installs etc.
19. Initial Learnings Mature Project Mailing lists Mailman Groups.io G Groups Engage- ment Contri- butors Pull Request Projects in later stages of their lives are expected to have more contributions, and a lot of the metrics we are interested are centered around contributions. We look not only the quantity of contributions, but also the demographics and level of the contributor engagement.
20. In the early stages of an open source project’s life, we want to look for intend to adopt, engage and contribute. In later stages we want to see action. Initial Learnings
21. The path to reporting metrics
22. The path to reporting metrics • Strategy • Analytics • Customization • Reporting The path to reporting metrics Reporting Customization Analysis Strategy OSPO Analytics Path
23. Strategy The goals defined by OSPO. Examples could be: • Collaboration • Attract talent • Gain influence • Give back • Foster OSS participation The path to reporting metrics Reporting Customi- zation Analysis Strategy
24. Analysis Define the questions you want answered. • Where are contributors coming from? • How much engagement do my projects get? • How many core, regular, and casual contributors do my projects have? • How quickly are we handling external contributions? • What is the company’s OSS footprint? • What is contributors path in our projects? The path to reporting metrics Reporting Customi- zation Analysis Strategy
25. Customization Identifying the sources of insights • +30 data sources supported • Predefined and customizable dashboards • Contributors identity information management • Rest API for data consumption • 100% free, open source software The path to reporting metrics Reporting Customi- zation Analysis Strategy GrimoireLab chaoss.github.io/grimoirelab
26. Reporting Where contributors are coming from? The path to reporting metrics Reporting Customi- zation Analysis Strategy
27. Reporting How much engagement do my projects get? The path to reporting metrics Reporting Customi- zation Analysis Strategy
28. Reporting How much engagement do my projects get? The path to reporting metrics Reporting Customi- zation Analysis Strategy
29. Reporting How many core, regular and casual contributors do I have? The path to reporting metrics Reporting Customi- zation Analysis Strategy
30. Reporting How fast I am dealing with external contributions? The path to reporting metrics Reporting Customi- zation Analysis Strategy
31. Reporting What’s the company’s OSS footprint? (WIP) The path to reporting metrics Reporting Customi- zation Analysis Strategy
32. Reporting What is contributors path in our projects? (WIP) The path to reporting metrics Reporting Customi- zation Analysis Strategy Users External people asking questions Contributors External people answering questions External people submitting patches Mantainers External people committing code
33. Recap
34. Recap • Data is almost unlimited; the art is to be selective about what to measure • Metrics should always answer questions relevant for both the project and the business • Use metrics to gain insights; metrics without context are not useful • Open source projects go through different life stages and should be evaluated against goals appropriate for the current stage
35. Thank you Jose Manrique López de la Fuente CEO, Bitergia jsmanrique@bitergia.com @jsmanrique Carsten Jacobsen Open Source Developer Advocate, Uber carstenj@uber.com @CarstenJacobsen
36. Proprietary © 2019 Uber Technologies, Inc. All rights reserved. No part of this document may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval systems, without permission in writing from Uber. This document is intended only for the use of the individual or entity to whom it is addressed and contains information that is privileged, confidential or otherwise exempt from disclosure under applicable law. All recipients of this document are notified that the information contained herein includes proprietary and confidential information of Uber, and recipient may not make use of, disseminate, or in any way disclose this document or any of the enclosed information to any person other than employees of addressee to the extent necessary for consultations with authorized personnel of Uber.