Are you sure you’re measuring what you want to measure?

Presentation by Daniel Izquierdo at the InnerSource Commons Fall Summit

Cheating on metrics is easy. And it happens that from time to time we realize that the metrics that we thought were the right ones, are in reality the wrong ones. This talk will analyze one of the first steps that everyone is measuring in the InnerSource initiative, and these are the contributions. There are of course several types of them, and not only source code: reviews, edits in a wiki, or conversations in asynchrounous channels. However, how can we sure that we all mean the same when we measure those contributions? Does a commit have the same meaning for everyone? Have we ever thought about automated actions by bots, merge commits, or people adding tabs to the source code? And even more, is better to measure commits than source code lines? And what about files?

Transcript

1. Are you sure you’re measuring what you want to measure? Daniel Izquierdo Cortázar @dizquierdo dizquierdo at bitergia.com https://speakerdeck.com/[bitergia|dicortazar] InnerSource Commons Fall Summit CA Technologies. Oct, 2018.
2. /context Measuring inner source Let’s assume that we already have: Methodology Infrastructure Strategy
3. /data_sources
4. /collaboration Everything is based on certain granularity We can measure collaboration at the level of: Repositories, commits, files? Pages in wikis, communication in threads? Pull requests comments, reviews or submissions? Issues, comments, participation IRC/Slack/Mattermost channels, threads
5. /viz
6. /viz
7. /motivation When we say {commit}, do we all mean the same?
8. /git_collaboration What is a commit? a commit adds … changes to … the repository, making these changes part of the head revision of the repository. … commits in version control systems are kept in the repository indefinitely. https://en.wikipedia.org/wiki/Commit_(version_control)
9. /git_collaboration What is a commit? a commit adds … changes to … the repository, making these changes part of the head revision of the repository. … commits in version control systems are kept in the repository indefinitely. https://en.wikipedia.org/wiki/Commit_(version_control)
10. /git_collaboration What is a commit? a commit adds … changes to … the repository, making these changes part of the head revision of the repository. … commits in version control systems are kept in the repository indefinitely. https://en.wikipedia.org/wiki/Commit_(version_control)
11. /git_collaboration What is a commit? a commit adds … changes to … the repository, https://en.wikipedia.org/wiki/Commit_(version_control)
12. /git_collaboration What is a commit? ¯\_(ツ)_/¯
13. /git_log What is a commit? commit 80d45e055f12c98b28d47cbff6a9d9b599b90181 2abd2ad00d6fee314762c32200f796ef0c36ad6a Author: alpgarcia <alpgarcia@bitergia.com> AuthorDate: Wed Dec 20 18:35:11 2017 +0100 Commit: Daniel Izquierdo <dizquierdo@bitergia.com> CommitDate: Wed Dec 20 19:44:09 2017 +0100 Fix error when enriching just the last Git commit [...] :100644 100644 7a84317... 7134d40... M enrich/enrich.py :100644 100644 400c5fd... 17de5ca... M tests/test_enrich.py 60 48 enrich/enrich.py 39 7 tests/test_enrich.py
14. /git_log What is a commit? commit 80d45e055f12c98b28d47cbff6a9d9b599b90181 2abd2ad00d6fee314762c32200f796ef0c36ad6a Author: alpgarcia <alpgarcia@bitergia.com> AuthorDate: Wed Dec 20 18:35:11 2017 +0100 Commit: Daniel Izquierdo <dizquierdo@bitergia.com> CommitDate: Wed Dec 20 19:44:09 2017 +0100 Fix error when enriching just the last Git commit [...] :100644 100644 7a84317... 7134d40... M enrich/enrich.py :100644 100644 400c5fd... 17de5ca... M tests/test_enrich.py 60 48 enrich/enrich.py 39 7 tests/test_enrich.py
15. /git_log What is a commit? commit 80d45e055f12c98b28d47cbff6a9d9b599b90181 2abd2ad00d6fee314762c32200f796ef0c36ad6a Author: alpgarcia <alpgarcia@bitergia.com> AuthorDate: Wed Dec 20 18:35:11 2017 +0100 Commit: Daniel Izquierdo <dizquierdo@bitergia.com> CommitDate: Wed Dec 20 19:44:09 2017 +0100 Fix error when enriching just the last Git commit [...] :100644 100644 7a84317... 7134d40... M enrich/enrich.py :100644 100644 400c5fd... 17de5ca... M tests/test_enrich.py 60 48 enrich/enrich.py 39 7 tests/test_enrich.py
16. /git_log What is a commit? commit 80d45e055f12c98b28d47cbff6a9d9b599b90181 2abd2ad00d6fee314762c32200f796ef0c36ad6a Author: alpgarcia <alpgarcia@bitergia.com> AuthorDate: Wed Dec 20 18:35:11 2017 +0100 Commit: Daniel Izquierdo <dizquierdo@bitergia.com> CommitDate: Wed Dec 20 19:44:09 2017 +0100 Fix error when enriching just the last Git commit [...] :100644 100644 7a84317... 7134d40... M enrich/enrich.py :100644 100644 400c5fd... 17de5ca... M tests/test_enrich.py 60 48 enrich/enrich.py 39 7 tests/test_enrich.py
17. /git_log What is a commit? commit 80d45e055f12c98b28d47cbff6a9d9b599b90181 2abd2ad00d6fee314762c32200f796ef0c36ad6a Author: alpgarcia <alpgarcia@bitergia.com> AuthorDate: Wed Dec 20 18:35:11 2017 +0100 Commit: Daniel Izquierdo <dizquierdo@bitergia.com> CommitDate: Wed Dec 20 19:44:09 2017 +0100 Fix error when enriching just the last Git commit [...] :100644 100644 7a84317... 7134d40... M enrich/enrich.py :100644 100644 400c5fd... 17de5ca... M tests/test_enrich.py 60 48 enrich/enrich.py 39 7 tests/test_enrich.py
18. /git_log What is a commit? commit 80d45e055f12c98b28d47cbff6a9d9b599b90181 2abd2ad00d6fee314762c32200f796ef0c36ad6a Author: alpgarcia <alpgarcia@bitergia.com> AuthorDate: Wed Dec 20 18:35:11 2017 +0100 Commit: Daniel Izquierdo <dizquierdo@bitergia.com> CommitDate: Wed Dec 20 19:44:09 2017 +0100 Fix error when enriching just the last Git commit [...] :100644 100644 7a84317... 7134d40... M enrich/enrich.py :100644 100644 400c5fd... 17de5ca... M tests/test_enrich.py 60 48 enrich/enrich.py 39 7 tests/test_enrich.py
19. /git_log What is a commit? commit 80d45e055f12c98b28d47cbff6a9d9b599b90181 2abd2ad00d6fee314762c32200f796ef0c36ad6a Author: alpgarcia <alpgarcia@bitergia.com> AuthorDate: Wed Dec 20 18:35:11 2017 +0100 Commit: Daniel Izquierdo <dizquierdo@bitergia.com> CommitDate: Wed Dec 20 19:44:09 2017 +0100 Fix error when enriching just the last Git commit [...] :100644 100644 7a84317... 7134d40... M enrich/enrich.py :100644 100644 400c5fd... 17de5ca... M tests/test_enrich.py 60 48 enrich/enrich.py 39 7 tests/test_enrich.py
20. /git_bots
21. /git_merge A---B---C branch / \ D---E---F---G---H master
22. /git_merge A---B---C branch / \ D---E---F---G---H master OpenStack vs InnerSource Patterns (1:1) vs (1:n)
23. /infra CVS vs SVN vs Git
24. /git_branches Branches vs Master A---B---C---I---J---Z branch / \ D---E---F---G---H master
25. /git_author Authors vs Committers Author: alpgarcia <alpgarcia@bitergia.com> AuthorDate: Wed Dec 20 18:35:11 2017 +0100 Commit: Daniel Izquierdo <dizquierdo@bitergia.com> CommitDate: Wed Dec 20 19:44:09 2017 +0100
26. /openstack_nova All commits
27. /openstack_nova All commits, but merges
28. /openstack_nova All commits, but merges and bots
29. /wrong_assumption Examples of wrong assumptions Automated commits but existing under some developer account. Example at OpenShift in 2013/2014. More than 7K commits.
30. /git_collaboration Examples of wrong assumptions Automated commits:
31. /wrong_assumption Brian Chang as the main developer by far in the project, even more than the bots https://github.com/liferay/liferay-portal/graphs/contributors
32. /summary
33. /keep_calm Define in a proper way each metric you’re using Involve developers and other roles into this Be able to reproduce the analysis Understand your infrastructure Dig into the data before assuming things Double check your final dataset
34. /viz
35. Software Development Analytics for your peace of mind dizquierdo@bitergia.com Chief Data Officer at Bitergia