Scalability, Practicability (and Promotion) in SE Research

13beaa3b7239eca3319d54c6a9f3a85a?s=47 ASERG, DCC, UFMG
September 17, 2018

Scalability, Practicability (and Promotion) in SE Research

GitHub is the world’s largest collection of open source software, with around 27 million users and 80 million repositories. These numbers make GitHub an invaluable source of data for large-scale empirical software engineering research. In this talk, we describe recent research conducted in our group, based on GitHub data. For example, we are using GitHub to predict the popularity of open source projects, to understand the motivations behind refactoring, to characterize the evolution of APIs, and to reveal key characteristics of open source development teams. In the talk, we also plan to discuss the strategies we are using to make our research results known by practitioners.

13beaa3b7239eca3319d54c6a9f3a85a?s=128

ASERG, DCC, UFMG

September 17, 2018
Tweet

Transcript

  1. Scalability, Practicability (and Promotion) in SE Research Marco Tulio Valente

    ASERG, DCC, UFMG, BR @mtov 1 SBCARS, September 2018
  2. My take on SE research 2

  3. SE research is still relevant, after 50 years 3

  4. but we should focus on scalable solutions to practical problems

    4
  5. Counterexample #1: solution to a very simple context or system

    5
  6. Counterexample #2: solution to a problem developers will have in

    10 years 6
  7. We need scalability & practicability 7

  8. Scalability & Practicability ➜ Data 8

  9. The world’s most valuable resource is no longer oil, but

    data 9 The Economist, May 2017
  10. For the first time in 50 yrs, we have lots

    of data 10
  11. This talk's story is about using as much as possible

    data to shed light on modern software engineering problems 11
  12. Of course, we didn't discover GitHub alone; almost everyone is

    doing the same 12
  13. Data ➜ Quantitative & Qualitative 13

  14. Part I Large scale surveys ( Surveys = Qualitative Studies

    ) 14
  15. "Why" surveys • Why do we refactor? • Why do

    we break APIs? • Why do open source projects fail? • Why do we star GitHub projects? 15
  16. Why do we refactor? FSE 2016, with Danilo and Tsantalis

    16
  17. Why do we really refactor, in practice? 17

  18. Why do we really refactor? • Danilo tracked (using a

    tool) refactorings in ◦ 748 Java projects, 61 days ◦ 1,411 refactorings, 185 projects, by 465 devs ◦ 195 answers (42%), right after refactoring 18
  19. FSE 2016 44 real reasons for refactoring 19

  20. Key Finding Refactoring is driven by the need to add

    new features and fix bugs and much less by code smell resolution 20
  21. Why do we break APIs? SANER 2018, with Aline, Laerte

    and Andre 21
  22. Why do we break APIs? • Aline tracked (using a

    tool) breaking changes (BCs) in ◦ 400 Java libraries & frameworks ◦ 116 days ◦ 282 possible BCs, by 102 developers ◦ 56 answers (55%), right after the BCs 22
  23. Key Finding We break APIs to implement new features (32%),

    to simplify the APIs (29%) and to improve maintainability (24%) 23
  24. [ firehouse interviews = surveys/interviews right after the event of

    interest ] 24
  25. Why do open source projects fail? FSE 2017, with Jailton

    25
  26. fail ➜ become deprecated 26

  27. Why do open source projects fail? • Jailton asked this

    question to the maintainers of ◦ 408 projects without commits for one year ◦ 118 answers (29%) 27
  28. Reason Projects Usurped by competitor 27 Obsolete 20 Lack of

    time 18 Lack of interest 18 Outdated technologies 14 Low maintainability 7 Conflicts among developers 3 Legal problems 2 Acquisition 1 Why do open source projects fail? 28
  29. Why do we star GitHub projects? JSS, to appear, with

    Hudson 29
  30. 30

  31. Interesting: developers care about this metric! 31

  32. 32

  33. Why do we star GitHub projects? • Hudson asked this

    question to 4,370 GitHub users ◦ right after they starred a popular repository ◦ 791 answers (19%) 33
  34. Key Finding #1 We star repositories to show appreciation (~50%)

    and to bookmark projects (~50%) 34
  35. Key Finding #2 3 out of 4 devs consider stars

    before contributing or using GitHub projects 35
  36. Why do we need "why-surveys"? Where's the practicability? 36

  37. (1) Surveys require building tools (2) Surveys are used to

    evaluate tools (3) Surveys motivate building tools (4) Surveys contribute to public datasets 37
  38. 38 (1) Surveys require building tools

  39. Refactoring Detection Tools • RefactoringMiner (1.0) (Tsantalis, CASCON 2013) •

    RefactoringMiner (1.1) (Tsantalis, Danilo, MT, FSE 2016) • RefDiff (new tool) (1.0) (Danilo, MT, MSR 2017) • RefactoringMiner (2.0) (Tsantalis et al., ICSE 2018) • RefDiff (2.0) (??) 39
  40. Refactoring Detection Tools • RefactoringMiner (1.0) (Tsantalis, CASCON 2013) •

    RefactoringMiner (1.1) (Tsantalis, Danilo, MT, FSE 2016) • RefDiff (new tool) (1.0) (Danilo, MT, MSR 2017) • RefactoringMiner (2.0) (Tsantalis et al., ICSE 2018) • RefDiff (2.0) (??) ⇒ refactoring-aware tools (code reviews, MSR etc) [ see Andre, Romain & MT, ICSE18 ] 40
  41. (2) Surveys are used to evaluate tools 41

  42. Truck (or Bus) Factor 42

  43. Truck (or Bus) Factor 43 The minimum number of developers

    that if hit by a truck (or bus) will put a project in a serious risk
  44. TF reveals the concentration of knowledge in software projects 44

  45. Interesting: (1) developers care about this metric 45

  46. Interesting: (2) TF has also value on closed projects 46

  47. 47 ICPC 2016, with Guilherme, Leonardo, and Andre

  48. Evaluation with GitHub Projects 133 projects (TF ≤ 2) 65%

    48
  49. Validation with Developers • 114 projects • 62 answers (54%)

    • 84% agree/partially agree 49
  50. (3) Surveys motivate building tools 50

  51. Example: gittrends.io 51

  52. gittrends.io 52

  53. gittrends.io 53

  54. Growth Patterns 54

  55. (4) Surveys contribute to public datasets 55

  56. Example: Refactoring instances 56 https://github.com/aserg-ufmg/why-we-refactor

  57. Ethical Considerations 57

  58. 58 Slide used by the authors in their ESEM presentation

  59. "I get emails like this every week ... This problem

    [is] worse than spam, since Google at least filters out spam for me". 59
  60. Recommendations for Large Scale Surveys 60

  61. Based on our experience/lessons learned 1. Questions should focus on

    practical and prevailing problems 2. Questions should focus on recent events 3. Questions should be sent by e-mail 4. Mails should have 2-3 short and clear questions 5. Avoid sending thousands of mails 6. Never send two mails to the same person (even in distinct studies) 7. Never identify the participants (names, e-mails, projects, etc) 61
  62. In our experience, Strong correlation: (practical value) vs (response rate)

    practical value ➜ at least 20% response rates 62
  63. Part II Large Scale Quantitative Studies (no mails to developers)

    63
  64. Example 1: Quantitative Analysis of Breaking Changes 64

  65. 65 501,645 LIBRARY CHANGES 28% BREAKING CHANGES SANER 2017, with

    Laerte, Aline, Andre
  66. Example 2: Quantitative Analysis of Deprecation Messages 66

  67. 67 API Deprecation Messages

  68. 68 % API elements deprecated with replacement msgs SANER 2016

    & JSS 2018, with Gleison and Andre
  69. 69 % API elements deprecated with replacement msgs SANER 2016

    & JSS 2018, with Gleison and Andre ⇒ automatic documentation
  70. Part III Research Promotion (no mails to developers) 70

  71. Key to make our results known by developers (i.e. to

    transfer knowledge) 71
  72. Promotion Channels • http://aserg.labsoft.dcc.ufmg.br • https://github.com/aserg-ufmg • @mtov • https://medium.com/@aserg.ufmg

    • https://arxiv.org • https://speakerdeck.com/aserg_ufmg 72
  73. [ there are other forms to transfer knowledge e.g. open

    tools = important, but very hard ] 73
  74. Lots of (mostly positive) feedback! 74

  75. 75 "An astonishing paper that may explain why it's so

    difficult to patch"
  76. 76 Hacker News Effect

  77. PeerJ Preprint (2015) 77 https://peerj.com/preprints/1233

  78. 78 "Oh, nice to hear from you! I heard a

    lot about (and read) your group's truck factor paper. Cool work!" (answer received in another survey, not related with TFs)
  79. Best citations ever ... 79

  80. 80 Best citations ever: devs conferences ... Heather Miller's talk,

    Lambda Days 2018
  81. 81 Best citations ever … or tech reports Nadia Eghbal,

    Ford Foundation.
  82. Scalability & Practicability (& Promotion) Thanks! 82