Work Practices and Challenges in Pull-Based Development: the Integrator’s Perspective

Work Practices and Challenges in Pull-Based Development: the Integrator’s Perspective

Slides from my talk at ICSE 2015

43df3993acc9af4e9f619e59cd849aee?s=128

Georgios Gousios

May 21, 2015
Tweet

Transcript

  1. Work Practices and Challenges in Pull-Based Development: the Integrator’s Perspective

    Georgios Gousios, Andy Zaidman, Margaret-Anne Storey, Arie van Deursen
  2. How does software grow? http://travelspirit333.com

  3. How does OSS grow? http://travelspirit333.com

  4. The onion model Core developers Co-developers Community Users

  5. The pull-based development model Here are my changes Please fix

    those issues Here are my updates Looks great, thanks! contributor integrator changes integrated changes examined changes re- examined
  6. GitHub: made pull requests popular

  7. 45% of collaborative projects Projects with > 1 committers Gousios

    et al. ICSE14
  8. 45% of collaborative projects Projects with > 1 committers 55%

    use shared repository 45% use pull requests Gousios et al. ICSE14
  9. Widely popular and increasing

  10. Widely popular and increasing 90k repositories 400k pull requests

  11. Widely popular and increasing Per month 90k repositories 400k pull

    requests
  12. Large scale collaboration 2710 committers 309 code reviewers 9120 commenters

    657 collaborators 125 code reviewers 3904 commenters 875 committers 92 code reviewers 5200 commenters
  13. Too successful?

  14. Too successful? “Lack of knowledge of git from contributors; most

    don’t know how to resolve a merge conflict.” “Sifting through the GitHub information flood to find what, if any, I should address.” “Dealing with loud and trigger-happy developers.”
  15. Integrators: guardians of quality

  16. WUGRWNNTGSWGUVU! FGEKFGVJGHCVGQHCEQPVTKDWVKQP! GXCNWCVGVJGSWCNKV[QHEQPVTKDWVKQPU! RTKQTKVKUGVJGCRRNKECVKQPQHEQPVTKDWVKQPU! YJCVMG[EJCNNGPIGUFQVJG[HCEG! 4GUGCTEJ3WGUVKQPU *QYFQKPVGITCVQTU

  17. Pilot survey GHTorrent 250 integrators (25 responses) Literature Survey Final

    Survey Analysis
  18. How we reached out to integrators Lifelines for 10% slowest

    pull reqs community participation in commits % comments (red) and commenters (blue) from the community http://ghtorrent.org/pullreq-perf/
  19. GHTorrent Final Survey Performance Reports

  20. GHTorrent Final Survey Performance Reports 3,200 integrators (749 responses)

  21. GHTorrent Final Survey Analysis / filtering (15 closed questions) Performance

    Reports 4 researchers Card sorting (7 open questions) 3,200 integrators (749 responses)
  22. GHTorrent Final Survey Analysis / filtering (15 closed questions) Performance

    Reports 4 researchers Card sorting (7 open questions) 3,200 integrators (749 responses) Results
  23. GHTorrent Final Survey Analysis / filtering (15 closed questions) Performance

    Reports 4 researchers Card sorting (7 open questions) 3,200 integrators (749 responses) Results 645 integrators
  24. Overlay *QYFQKPVGITCVQTUWUGRWNNTGSWGUVU! 

  25. How integrators use pull requests? Code reviews Issue fixes Soliciting

    contributions Discussing new features Work distribution Other 0 20 40 60 80 % of responses
  26. How integrators use pull requests? Code reviews Issue fixes Soliciting

    contributions Discussing new features Work distribution Other 0 20 40 60 80 % of responses
  27. How integrators use pull requests? Code reviews Issue fixes Soliciting

    contributions Discussing new features Work distribution Other 0 20 40 60 80 % of responses
  28. How projects do code reviews?

  29. How projects do code reviews? 65%: Inline code comments

  30. How projects do code reviews? 65%: Inline code comments 60%:

    Integration pipeline
  31. How projects do code reviews? 65%: Inline code comments 60%:

    Integration pipeline 55%: The community participates
  32. Overlay *QYFQKPVGITCVQTUFGEKFGYJKEJ EQPVTKDWVKQPUVQCEEGRV! 

  33. Factors for determining acceptance Code quality Code style Project fit

    Technical fit Testing Documentation Feature importance Code review result 0 5,5 11 16,5 22 % of responses
  34. Factors for determining acceptance Code quality Code style Project fit

    Technical fit Testing Documentation Feature importance Code review result 0 5,5 11 16,5 22 Quality Project Fit % of responses
  35. PR characteristics leading to merge PR hotness PR has tests

    PR track record PR churn PR # comments 0 50 100 150 200 Not or Mildly Important Quite or Very Imporant 100 0 100 % of responses
  36. Overlay *QYFQKPVGITCVQTUGXCNWCVGVJG SWCNKV[QHEQPVTKDWVKQPU! 

  37. - Following project's code style - Comments in code -

    Quality of the commit message - Readability of the code - Test coverage and their readability - Included documentation updates “Proper use of C++ language constructs. Knowledge and use of our dependent libraries […]” Views on quality
  38. Contribution quality perception Style conformance Test coverage of PR Code

    quality Code review Test/CI result Documentation Experience Author reputation Project conventions 0 3,25 6,5 9,75 13 % of responses
  39. Contribution quality perception Style conformance Test coverage of PR Code

    quality Code review Test/CI result Documentation Experience Author reputation Project conventions 0 3,25 6,5 9,75 13 % of responses Conformance
  40. Contribution quality perception Style conformance Test coverage of PR Code

    quality Code review Test/CI result Documentation Experience Author reputation Project conventions 0 3,25 6,5 9,75 13 % of responses Conformance Technical Excellence
  41. Contribution quality perception Style conformance Test coverage of PR Code

    quality Code review Test/CI result Documentation Experience Author reputation Project conventions 0 3,25 6,5 9,75 13 % of responses Conformance Technical Excellence
  42. Overlay *QYFQKPVGITCVQTURTKQTKVKUGVJG CRRNKECVKQPQHEQPVTKDWVKQPU! 

  43. Prioritisation factors

  44. Prioritisation factors “How important the issue is to the project.

    How serious the bug is. […]” Criticality
  45. Prioritisation factors Urgency Criticality “Whether the functionality is critical (bug

    fixes or very important feature blocking many users)[…]”
  46. Prioritisation factors Urgency Criticality Size/Complexity “If the merge is small,

    or affects a large issue, it'll be merged fast. If it's a feature request or large change it will take longer.”
  47. Prioritisation factors Urgency Mostly FIFO, but critical bug fixes or

    hot fixes get pushed to the top Age Criticality Size/Complexity
  48. Overlay 9JCVCTGVJGMG[EJCNNGPIGUHCEGF D[KPVGITCVQTU! 

  49. Technical challenges

  50. Technical challenges code base quality “Ensuring the codebase remains consistent”

  51. Technical challenges impact assessment “Huge, unwieldy, complected bundles of ‘hey

    I added a LOT of features and fixes ALL AT ONCE!’ that are hell to review and that I’d like to *partially* reject if only the parts were in any way separable…” code base quality
  52. Technical challenges impact assessment contributor experience code base quality “Lack

    of knowledge of git from contributors; most don’t know how to resolve a merge conflict.”
  53. Technical challenges impact assessment “Sifting through the GitHub information flood

    to find what, if any, I should address” contributor experience code base quality volume
  54. Social challenges

  55. Social challenges “Time is the biggest challenge -- I would

    love to have a more hierarchical structure to spread out the work; however, the community's development capacity is somewhat limited.” workload
  56. Social challenges workload responsiveness “People become non-responsive, never write tests,

    and never update style. They make a change once when they need their problem solved and disappear basically.”
  57. Social challenges explaining rejection workload responsiveness “Disappointing people when they

    put in a lot of work but the quality is low.”
  58. Social challenges explaining rejection workload responsiveness motivating contributors “On-boarding new

    contributors: helping them set up the development environment and learn enough Git/ Github to submit a pull request”
  59. How does OSS grow? The pull- based development perspective http://travelspirit333.com

  60. Integrators use pull requests for code review https://www.digitalassurance.com/sites/default/files/iStock_000017496218_Medium.jpg

  61. Integrators use testing as a safety net http://www.tradeandexportme.com/wp-content/uploads/2013/02/shutterstock_91331678.jpg

  62. Integrators face technical challenges

  63. Integrators face technical challenges Impact assessment Quality evaluation Work prioritisation

  64. Integrators do not use track records (much)

  65. Not a better online forum experience

  66. Recommendations for integrators

  67. Recommendations for integrators automation contribution guidelines testing be proactive/reactive

  68. New research directions

  69. New research directions prioritisation quality analysis impact analysis code review

    automation
  70. New research directions prioritisation quality analysis impact analysis code review

    automation @gousiosg