Contributor Leaderboards to Incentivize Good Community Citizenship

It takes a village to build open source software, or really a community of people with many different skills. Leaderboards can help us show the top-performing contributors and celebrate them, but they are double-edge swords in a community's toolbox. Tracking contributions can introduce an element of gamification, especially when contributors gain privileges in a community based on their level of contributions. As a positive outcome, leaderboards can increase participation and contributions which hopefully advances the community.

Transcript

1. OpenJS World
   June 6-10, 2022
   Incentivizing Good
   Community Citizenship
   with Contributor
   Leaderboards
   Georg Link, Director of Sales at Bitergia
   

   View Slide

2. Agenda
   ● Leaderboards
   ● Metrics
   ● Best Practices
   @GeorgLink
   

   View Slide

3. Leaderboards
   @GeorgLink
   

   View Slide

4. What are Leaderboards?
   : a large board for displaying the ranking of the leaders in a competitive
   event (such as a golf tournament)
   Definition of
   leaderboard
   @GeorgLink
   

   View Slide

5. Other Examples of Leaderboards
   Countries
   @GeorgLink
   

   View Slide

6. Other Examples of Leaderboards
   Countries
   Corporations
   @GeorgLink
   

   View Slide

7. Other Examples of Leaderboards
   Countries
   Corporations
   Games
   @GeorgLink
   

   View Slide

8. Other Examples of Leaderboards
   Countries
   Corporations
   Games
   People
   @GeorgLink
   

   View Slide

9. Other Examples of Leaderboards
   Countries
   Corporations
   Games
   People
   Open Source
   @GeorgLink
   

   View Slide

10. Leaderboards for GitHub contributors
    GitHub Insights shows
    contributors with most
    commits
    @GeorgLink
    

    View Slide

11. Leaderboards in Drupal Marketplace
    @GeorgLink
    

    View Slide

12. Leaderboard Theory
    @GeorgLink
    

    View Slide

13. image goes here
    Why do we use Leaderboards?
    ● Give instant feedback for contributors
    ● Create sense of community
    ● Recognize contributors
    ● Reward activity
    ● Create competition
    ● Incentivize more activity
    @GeorgLink
    

    View Slide

14. How do Leaderboards work?
    ● The Self-Determination Theory
    ○ The need for competence
    ○ The need for autonomy
    ○ The need for social relatedness
    ● Goal-Setting Theory of Motivation
    ○ Goal specific leaderboards work better
    ● Social Comparison Theory
    ○ Understanding of your own abilities and reactions
    ○ Facilitating social interaction
    @GeorgLink
    

    View Slide

15. Motivational issues with Leaderboards
    If competitors were not of equal ability:
    Less skilled competitors became less motivated as the task seemed
    unreachable
    Highly skilled competitors found the activity too easy, eventually making
    them less motivated too
    Landers (2015)
    @GeorgLink
    

    View Slide

16. How a Leaderboard Ranks People?
    ● Score
    ○ Progress towards a success measure
    ○ Embedded criteria for what counts
    ○ Bias towards measurable things
    @GeorgLink
    

    View Slide

17. Metrics
    @GeorgLink
    

    View Slide

18. Metrics
    A method of measuring something, or the results obtained from this:
    project data metric
    method
    @GeorgLink
    

    View Slide

19. Metrics
    ● Created accidentally
    ● Contributions and their metadata
    ● Contains personal identifiable information (PII)
    like names and emails
    Project Data
    as Trace Data
    @GeorgLink
    

    View Slide

20. Where is the data? Where is the community?
    Metrics
    Git Wiki
    GitHub Discourse
    GitLab Mailing List
    BitBucket IRC
    Jira Slack
    Gerrit Meetup.com
    Confluence StackOverflow
    @GeorgLink
    

    View Slide

21. Raw data:
    ● Get data from data sources
    Enriched data:
    ● Unify data
    ● Manage identities
    ● Calculate metrics
    Useful data:
    ● Visualize, report
    How to get the data
    @GeorgLink
    

    View Slide

22. Easiest step
    Query logs
    Use APIs
    Challenge: Changing APIs and data
    formats
    Raw data
    @GeorgLink
    

    View Slide

23. Unify data
    ● Date formats
    ● Level of detail
    ● Metadata about different contributions
    ● Convert everything into the desired database
    structure
    Manage identities
    Calculate metrics
    Enriched data
    @GeorgLink
    

    View Slide

24. Unify data
    Manage identities
    ● Who is who in the community
    ● Who do contributors work for (now and
    before)
    ● Different usernames and email
    ● Assigning contributions to the correct person
    Calculate metrics
    Enriched data
    @GeorgLink
    

    View Slide

25. Unify data
    Manage identities
    Calculate metrics
    ● Primary metrics - summarizing original data
    ● Secondary metrics
    ○ Calculation from different data fields
    ○ Combining data from different data sources
    ○ Value judgements on data - e.g., quality models
    Enriched data
    @GeorgLink
    

    View Slide

26. Useful data
    @GeorgLink
    

    View Slide

27. Useful data
    Who is the data user?
    @GeorgLink
    

    View Slide

28. Useful data
    Who is the data user?
    How should the data be presented?
    @GeorgLink
    

    View Slide

29. Useful data
    Who is the data user?
    How should the data be presented?
    What visualizations are most meaningful?
    @GeorgLink
    

    View Slide

30. Useful data
    Who is the data user?
    How should the data be presented?
    What visualizations are most meaningful?
    What story does the data tell?
    @GeorgLink
    

    View Slide

31. ● Cauldron.io
    ● CHAOSS GrimoireLab
    ● CHAOSS Augur
    ● Apache Kibble
    ● CNCF Dev Analytics
    OSS Solutions
    @GeorgLink
    

    View Slide

32. Best Practices
    @GeorgLink
    

    View Slide

33. image goes here
    About CHAOSS
    Short for: Community Health Analytics Open Source Software
    Linux Foundation project, cofounded by Bitergia
    Started in 2017
    Focused on “creating analytics and metrics to help define
    community health” -- https://chaoss.community
    @GeorgLink
    

    View Slide

34. image goes here
    Analytics Best Practices
    Follow Goal-Question-Metric approach
    Use metrics to tell a story
    Evaluate the usefulness of metric strategy
    Minimize gaming of metrics - careful how leaderboards score
    Start small and “get off zero”
    @GeorgLink
    

    View Slide

35. image goes here
    Leaderboards Best Practices
    ● Be transparent about scoring algorithm
    ● Revise scoring algorithm with community feedback
    ● Continuously update leaderboard (automate)
    ● Many leaderboards for different categories
    ● Recognize diverse types of contributions
    ● Contextualize by showing immediate neighbors
    ● Reset leaderboard or ignore older contributions
    @GeorgLink
    

    View Slide

36. Recap
    ● Leaderboards
    ● Metrics
    ● Best Practices
    @GeorgLink
    

    View Slide

37. Email georglink@bitergia.com
    LinkedIn/GitHub/Twitter @GeorgLink
    Thank You!
    @GeorgLink
    

    View Slide

38. References
    ● https://medium.com/practical-motivation-science/how-to-motivate-with-leaderboards-da5a
    461fb2f6
    ● https://www.digits.co.uk/lms/the-psychology-of-leaderboards-why-they-do-in-fact-work/
    ● https://chaoss.community/
    @GeorgLink
    

    View Slide

39. View Slide