Scalability, Practicability (and Promotion) in SE Research

GitHub is the world’s largest collection of open source software, with around 27 million users and 80 million repositories. These numbers make GitHub an invaluable source of data for large-scale empirical software engineering research. In this talk, we describe recent research conducted in our group, based on GitHub data. For example, we are using GitHub to predict the popularity of open source projects, to understand the motivations behind refactoring, to characterize the evolution of APIs, and to reveal key characteristics of open source development teams. In the talk, we also plan to discuss the strategies we are using to make our research results known by practitioners.

Transcript

1. Scalability, Practicability (and Promotion)
   in SE Research
   Marco Tulio Valente
   ASERG, DCC, UFMG, BR
   @mtov
   1
   SBCARS, September 2018
   

   View Slide

2. My take on SE research
   2
   

   View Slide

3. SE research is still relevant, after 50 years
   3
   

   View Slide

4. but we should focus on
   scalable solutions to practical problems
   4
   

   View Slide

5. Counterexample #1:
   solution to a very simple context or system
   5
   

   View Slide

6. Counterexample #2:
   solution to a problem developers will have in 10
   years
   6
   

   View Slide

7. We need scalability & practicability
   7
   

   View Slide

8. Scalability & Practicability ➜ Data
   8
   

   View Slide

9. The world’s most valuable resource is no longer
   oil, but data
   9
   The Economist, May 2017
   

   View Slide

10. For the first time in 50 yrs, we have lots of data
    10
    

    View Slide

11. This talk's story is about using as much as possible
    data to shed light on modern software engineering
    problems
    11
    

    View Slide

12. Of course, we didn't discover GitHub alone;
    almost everyone is doing the same
    12
    

    View Slide

13. Data ➜ Quantitative & Qualitative
    13
    

    View Slide

14. Part I
    Large scale surveys
    ( Surveys = Qualitative Studies )
    14
    

    View Slide

15. "Why" surveys
    ● Why do we refactor?
    ● Why do we break APIs?
    ● Why do open source projects fail?
    ● Why do we star GitHub projects?
    15
    

    View Slide

16. Why do we refactor?
    FSE 2016, with Danilo and Tsantalis
    16
    

    View Slide

17. Why do we really refactor, in practice?
    17
    

    View Slide

18. Why do we really refactor?
    ● Danilo tracked (using a tool) refactorings in
    ○ 748 Java projects, 61 days
    ○ 1,411 refactorings, 185 projects, by 465 devs
    ○ 195 answers (42%), right after refactoring
    18
    

    View Slide

19. FSE 2016
    44 real reasons for refactoring
    19
    

    View Slide

20. Key Finding
    Refactoring is driven by the need to add new
    features and fix bugs and much less by code smell
    resolution
    20
    

    View Slide

21. Why do we break APIs?
    SANER 2018, with Aline, Laerte and Andre
    21
    

    View Slide

22. Why do we break APIs?
    ● Aline tracked (using a tool) breaking changes (BCs) in
    ○ 400 Java libraries & frameworks
    ○ 116 days
    ○ 282 possible BCs, by 102 developers
    ○ 56 answers (55%), right after the BCs
    22
    

    View Slide

23. Key Finding
    We break APIs to implement new features (32%), to
    simplify the APIs (29%) and to improve
    maintainability (24%)
    23
    

    View Slide

24. [ firehouse interviews =
    surveys/interviews right after the event of interest ]
    24
    

    View Slide

25. Why do open source projects fail?
    FSE 2017, with Jailton
    25
    

    View Slide

26. fail ➜ become deprecated
    26
    

    View Slide

27. Why do open source projects fail?
    ● Jailton asked this question to the maintainers of
    ○ 408 projects without commits for one year
    ○ 118 answers (29%)
    27
    

    View Slide

28. Reason Projects
    Usurped by competitor 27
    Obsolete 20
    Lack of time 18
    Lack of interest 18
    Outdated technologies 14
    Low maintainability 7
    Conflicts among developers 3
    Legal problems 2
    Acquisition 1
    Why do open source
    projects fail?
    28
    

    View Slide

29. Why do we star GitHub projects?
    JSS, to appear, with Hudson
    29
    

    View Slide

30. View Slide

31. Interesting: developers care about this metric!
    31
    

    View Slide

32. View Slide

33. Why do we star GitHub projects?
    ● Hudson asked this question to 4,370 GitHub users
    ○ right after they starred a popular repository
    ○ 791 answers (19%)
    33
    

    View Slide

34. Key Finding #1
    We star repositories to show appreciation (~50%)
    and to bookmark projects (~50%)
    34
    

    View Slide

35. Key Finding #2
    3 out of 4 devs consider stars before contributing
    or using GitHub projects
    35
    

    View Slide

36. Why do we need "why-surveys"?
    Where's the practicability?
    36
    

    View Slide

37. (1) Surveys require building tools
    (2) Surveys are used to evaluate tools
    (3) Surveys motivate building tools
    (4) Surveys contribute to public datasets
    37
    

    View Slide

38. 38
    (1) Surveys require building tools
    

    View Slide

39. Refactoring Detection Tools
    ● RefactoringMiner (1.0) (Tsantalis, CASCON 2013)
    ● RefactoringMiner (1.1) (Tsantalis, Danilo, MT, FSE 2016)
    ● RefDiff (new tool) (1.0) (Danilo, MT, MSR 2017)
    ● RefactoringMiner (2.0) (Tsantalis et al., ICSE 2018)
    ● RefDiff (2.0) (??)
    39
    

    View Slide

40. Refactoring Detection Tools
    ● RefactoringMiner (1.0) (Tsantalis, CASCON 2013)
    ● RefactoringMiner (1.1) (Tsantalis, Danilo, MT, FSE 2016)
    ● RefDiff (new tool) (1.0) (Danilo, MT, MSR 2017)
    ● RefactoringMiner (2.0) (Tsantalis et al., ICSE 2018)
    ● RefDiff (2.0) (??)
    ⇒ refactoring-aware tools (code reviews, MSR etc)
    [ see Andre, Romain & MT, ICSE18 ]
    40
    

    View Slide

41. (2) Surveys are used to evaluate tools
    41
    

    View Slide

42. Truck (or Bus) Factor
    42
    

    View Slide

43. Truck (or Bus) Factor
    43
    The minimum number of developers that if hit by a truck (or
    bus) will put a project in a serious risk
    

    View Slide

44. TF reveals the concentration of knowledge in
    software projects
    44
    

    View Slide

45. Interesting:
    (1) developers care about this metric
    45
    

    View Slide

46. Interesting:
    (2) TF has also value on closed projects
    46
    

    View Slide

47. 47
    ICPC 2016, with Guilherme, Leonardo, and Andre
    

    View Slide

48. Evaluation with GitHub Projects
    133
    projects (TF ≤ 2)
    65%
    48
    

    View Slide

49. Validation with Developers
    ● 114 projects
    ● 62 answers (54%)
    ● 84% agree/partially agree
    49
    

    View Slide

50. (3) Surveys motivate building tools
    50
    

    View Slide

51. Example: gittrends.io
    51
    

    View Slide

52. gittrends.io
    52
    

    View Slide

53. gittrends.io
    53
    

    View Slide

54. Growth Patterns
    54
    

    View Slide

55. (4) Surveys contribute to public datasets
    55
    

    View Slide

56. Example: Refactoring instances
    56
    https://github.com/aserg-ufmg/why-we-refactor
    

    View Slide

57. Ethical Considerations
    57
    

    View Slide

58. 58
    Slide used by the authors in their ESEM presentation
    

    View Slide

59. "I get emails like this every week ... This problem [is]
    worse than spam, since Google at least filters out
    spam for me".
    59
    

    View Slide

60. Recommendations for
    Large Scale Surveys
    60
    

    View Slide

61. Based on our experience/lessons learned
    1. Questions should focus on practical and prevailing problems
    2. Questions should focus on recent events
    3. Questions should be sent by e-mail
    4. Mails should have 2-3 short and clear questions
    5. Avoid sending thousands of mails
    6. Never send two mails to the same person (even in distinct studies)
    7. Never identify the participants (names, e-mails, projects, etc)
    61
    

    View Slide

62. In our experience,
    Strong correlation: (practical value) vs (response rate)
    practical value ➜ at least 20% response rates
    62
    

    View Slide

63. Part II
    Large Scale Quantitative Studies
    (no mails to developers)
    63
    

    View Slide

64. Example 1:
    Quantitative Analysis of Breaking Changes
    64
    

    View Slide

65. 65
    501,645
    LIBRARY
    CHANGES
    28%
    BREAKING
    CHANGES
    SANER 2017, with Laerte, Aline, Andre
    

    View Slide

66. Example 2:
    Quantitative Analysis of Deprecation Messages
    66
    

    View Slide

67. 67
    API Deprecation Messages
    

    View Slide

68. 68
    % API elements deprecated with replacement msgs
    SANER 2016 & JSS 2018, with Gleison and Andre
    

    View Slide

69. 69
    % API elements deprecated with replacement msgs
    SANER 2016 & JSS 2018, with Gleison and Andre
    ⇒ automatic documentation
    

    View Slide

70. Part III
    Research Promotion
    (no mails to developers)
    70
    

    View Slide

71. Key to make our results known by developers
    (i.e. to transfer knowledge)
    71
    

    View Slide

72. Promotion Channels
    ● http://aserg.labsoft.dcc.ufmg.br
    ● https://github.com/aserg-ufmg
    ● @mtov
    ● https://medium.com/@aserg.ufmg
    ● https://arxiv.org
    ● https://speakerdeck.com/aserg_ufmg
    72
    

    View Slide

73. [ there are other forms to transfer knowledge
    e.g. open tools = important, but very hard ]
    73
    

    View Slide

74. Lots of (mostly positive) feedback!
    74
    

    View Slide

75. 75
    "An astonishing paper that may explain why it's so
    difficult to patch"
    

    View Slide

76. 76
    Hacker News Effect
    

    View Slide

77. PeerJ Preprint (2015)
    77
    https://peerj.com/preprints/1233
    

    View Slide

78. 78
    "Oh, nice to hear from you! I heard a lot about (and
    read) your group's truck factor paper. Cool work!"
    (answer received in another survey, not related with TFs)
    

    View Slide

79. Best citations ever ...
    79
    

    View Slide

80. 80
    Best citations ever: devs conferences ...
    Heather Miller's talk, Lambda Days 2018
    

    View Slide

81. 81
    Best citations ever … or tech reports
    Nadia Eghbal, Ford Foundation.
    

    View Slide

82. Scalability & Practicability
    (& Promotion)
    Thanks!
    82
    

    View Slide