Contributor Leaderboards to Incentivize Good Community Citizenship

It takes a village to build open source software, or really a community of people with many different skills. Leaderboards can help us show the top-performing contributors and celebrate them, but they are double-edge swords in a community's toolbox. Tracking contributions can introduce an element of gamification, especially when contributors gain privileges in a community based on their level of contributions. As a positive outcome, leaderboards can increase participation and contributions which hopefully advances the community.

1. OpenJS World June 6-10, 2022 Incentivizing Good Community Citizenship with Contributor Leaderboards Georg Link, Director of Sales at Bitergia
2. Agenda • Leaderboards • Metrics • Best Practices @GeorgLink
3. Leaderboards @GeorgLink
4. What are Leaderboards? : a large board for displaying the ranking of the leaders in a competitive event (such as a golf tournament) Definition of leaderboard @GeorgLink
5. Other Examples of Leaderboards Countries @GeorgLink
6. Other Examples of Leaderboards Countries Corporations @GeorgLink
7. Other Examples of Leaderboards Countries Corporations Games @GeorgLink
8. Other Examples of Leaderboards Countries Corporations Games People @GeorgLink
9. Other Examples of Leaderboards Countries Corporations Games People Open Source @GeorgLink
10. Leaderboards for GitHub contributors GitHub Insights shows contributors with most commits @GeorgLink
11. Leaderboards in Drupal Marketplace @GeorgLink
12. Leaderboard Theory @GeorgLink
13. image goes here Why do we use Leaderboards? • Give instant feedback for contributors • Create sense of community • Recognize contributors • Reward activity • Create competition • Incentivize more activity @GeorgLink
14. How do Leaderboards work? • The Self-Determination Theory ◦ The need for competence ◦ The need for autonomy ◦ The need for social relatedness • Goal-Setting Theory of Motivation ◦ Goal specific leaderboards work better • Social Comparison Theory ◦ Understanding of your own abilities and reactions ◦ Facilitating social interaction @GeorgLink
15. Motivational issues with Leaderboards If competitors were not of equal ability: Less skilled competitors became less motivated as the task seemed unreachable Highly skilled competitors found the activity too easy, eventually making them less motivated too Landers (2015) @GeorgLink
16. How a Leaderboard Ranks People? • Score ◦ Progress towards a success measure ◦ Embedded criteria for what counts ◦ Bias towards measurable things @GeorgLink
17. Metrics @GeorgLink
18. Metrics A method of measuring something, or the results obtained from this: project data metric method @GeorgLink
19. Metrics • Created accidentally • Contributions and their metadata • Contains personal identifiable information (PII) like names and emails Project Data as Trace Data @GeorgLink
20. Where is the data? Where is the community? Metrics Git Wiki GitHub Discourse GitLab Mailing List BitBucket IRC Jira Slack Gerrit Meetup.com Confluence StackOverflow @GeorgLink
21. Raw data: • Get data from data sources Enriched data: • Unify data • Manage identities • Calculate metrics Useful data: • Visualize, report How to get the data @GeorgLink
22. Easiest step Query logs Use APIs Challenge: Changing APIs and data formats Raw data @GeorgLink
23. Unify data • Date formats • Level of detail • Metadata about different contributions • Convert everything into the desired database structure Manage identities Calculate metrics Enriched data @GeorgLink
24. Unify data Manage identities • Who is who in the community • Who do contributors work for (now and before) • Different usernames and email • Assigning contributions to the correct person Calculate metrics Enriched data @GeorgLink
25. Unify data Manage identities Calculate metrics • Primary metrics - summarizing original data • Secondary metrics ◦ Calculation from different data fields ◦ Combining data from different data sources ◦ Value judgements on data - e.g., quality models Enriched data @GeorgLink
26. Useful data @GeorgLink
27. Useful data Who is the data user? @GeorgLink
28. Useful data Who is the data user? How should the data be presented? @GeorgLink
29. Useful data Who is the data user? How should the data be presented? What visualizations are most meaningful? @GeorgLink
30. Useful data Who is the data user? How should the data be presented? What visualizations are most meaningful? What story does the data tell? @GeorgLink
31. • Cauldron.io • CHAOSS GrimoireLab • CHAOSS Augur • Apache Kibble • CNCF Dev Analytics OSS Solutions @GeorgLink
32. Best Practices @GeorgLink
33. image goes here About CHAOSS Short for: Community Health Analytics Open Source Software Linux Foundation project, cofounded by Bitergia Started in 2017 Focused on “creating analytics and metrics to help define community health” -- https://chaoss.community @GeorgLink
34. image goes here Analytics Best Practices Follow Goal-Question-Metric approach Use metrics to tell a story Evaluate the usefulness of metric strategy Minimize gaming of metrics - careful how leaderboards score Start small and “get off zero” @GeorgLink
35. image goes here Leaderboards Best Practices • Be transparent about scoring algorithm • Revise scoring algorithm with community feedback • Continuously update leaderboard (automate) • Many leaderboards for different categories • Recognize diverse types of contributions • Contextualize by showing immediate neighbors • Reset leaderboard or ignore older contributions @GeorgLink
36. Recap • Leaderboards • Metrics • Best Practices @GeorgLink
37. Email georglink@bitergia.com LinkedIn/GitHub/Twitter @GeorgLink Thank You! @GeorgLink
38. References • https://medium.com/practical-motivation-science/how-to-motivate-with-leaderboards-da5a 461fb2f6 • https://www.digits.co.uk/lms/the-psychology-of-leaderboards-why-they-do-in-fact-work/ • https://chaoss.community/ @GeorgLink