Assessing the Threat of Untracked Changes in So!ware Evolution (ICSE 2018)

While refactoring is extensively performed by practitioners, many Mining Software Repositories (MSR) approaches do not detect nor keep track of refactorings when performing source code evolution analysis. In the best case, keeping track of refactorings could be unnecessary work; in the worst case, these untracked changes could significantly affect the performance of MSR approaches. Since the extent of the threat is unknown, the goal of this paper is to assess whether it is significant. Based on an extensive empirical study, we answer positively: we found that between 10 and 21% of changes at the method level in 15 large Java systems are untracked. This results in a large proportion (25%) of entities that may have their histories split by these changes, and a measurable effect on at least two MSR approaches. We conclude that handling untracked changes should be systematically considered by MSR studies.

Transcript

1. Assessing the Threat of Untracked Changes in Software Evolution André Hora, Danilo Silva, Marco Tulio Valente, Romain Robbes ICSE 2018
2. Outline 1. Context 2. Problem 3. Background 4. Study Design 5. Results 6. Final Remarks 2
3. Mining Software Repositories (MSR) 3
4. MSR Examples • Library migration • Change prediction • Bug fixing • Warnings prioritization • Code expert computation • … 4
5. Level of analysis Changes in classes and methods over time 5
6. Example 1 java.util.Vector —> java.util.List 6
7. Example 2 7 FileInputStream() —> Okio.source()
8. Example 2 8 FileInputStream() —> Okio.source()
9. Example 2 9 FileInputStream() —> Okio.source() Rule would not be detected due to the method renaming
10. Several other examples… 10
11. Refactoring is common practice in software development 11
12. Outline 1. Context 2. Problem 3. Background 4. Study Design 5. Results 6. Final Remarks 12
13. MSR studies may be affected by refactoring 13
14. MSR researchers are aware about this “threat”, but they often do not assess it “Our tool is unable to verify if an entity in revision n has been renamed in revision n+1” [48] “The development history of a file can be lost in case of renaming operations, copy or file split” [3] “It is possible to miss bug-introducing changes when a file changes its name since the approach does not track such name changes” [38] “We detect renamed or moved units as units that are removed first and added later” [50] 14
15. MSR researchers are aware about this “threat”, but they often do not assess it “Our tool is unable to verify if an entity in revision n has been renamed in revision n+1” [48] “The development history of a file can be lost in case of renaming operations, copy or file split” [3] “It is possible to miss bug-introducing changes when a file changes its name since the approach does not track such name changes” [38] “We detect renamed or moved units as units that are removed first and added later” [50] 15 [2, 5, 6, 7, 12, 22, 26, 27, 28, 29, 34, 36, 42, 45, 53, 54, 59, 61, 62, 66, 67, 68…]
16. What is the impact of refactoring on MSR studies? 16
17. Outline 1. Context 2. Problem 3. Background 4. Study Design 5. Results 6. Final Remarks 17
18. Tracked and Untracked Changes version 1 version 2 public void foo() { obj.print() } public void foo() { obj.println() } version 3 public void bar() { obj.println() } tracked change: preserves the entity name and modifies its source code untracked change: modifies the entity name, and may also modify its source code 18
19. Change Graph class Foo { mA() {…} } class Bar { mB() {…} mC() {…} } class Foo { mA() {…} } class Bar { mX() {…} mC() {…} } class Foo { mA() {…} } class Bar { mX() {…} } class Foo { mA() {…} } class Baz { mY() {…} } class Qux { mC() {…} } class Qux { mC() {…} mE() {…} } version 1 version 2 version 3 version 4 19 tracked change untracked change Legend
20. Outline 1. Context 2. Problem 3. Background 4. Study Design 5. Results 6. Final Remarks 20
21. Research Questions • RQ1.What is the frequency of untracked changes? • RQ2. What is the extension of untracked changes? • RQ3. What is the impact of untracked changes in existing MSR-based approaches? 21
22. Case Studies 22
23. Tracked and Untracked Changes Computation Refactoring resolution • RefDiff [Silva et al., MSR 2017] • Precision: 85.6% - 100% • Recall: 89.8% - 93.9% 1. Rename Class 2. Move Class 3. Extract Superclass 4. Move and Rename Class 5. Extract Interface 6. Rename Method 7. Move Method 8. Extract Method 9. Inline Method 10. Pull Up Method 11. Push Down Method 23
24. Outline 1. Context 2. Problem 3. Background 4. Study Design 5. Results 6. Final Remarks 24
25. RQ1 What is the frequency of untracked changes? 25
26. RQ1. What is the frequency of untracked changes? (example) class Foo { mA() {…} } class Bar { mB() {…} mC() {…} } class Foo { mA() {…} } class Bar { mX() {…} mC() {…} } class Foo { mA() {…} } class Bar { mX() {…} } class Foo { mA() {…} } class Baz { mY() {…} } class Qux { mC() {…} } class Qux { mC() {…} mE() {…} } version 1 version 2 version 3 version 4 26 17 changes 12 tracked changes 5 untracked changes
27. RQ1. What is the frequency of untracked changes? (example) class Foo { mA() {…} } class Bar { mB() {…} mC() {…} } class Foo { mA() {…} } class Bar { mX() {…} mC() {…} } class Foo { mA() {…} } class Bar { mX() {…} } class Foo { mA() {…} } class Baz { mY() {…} } class Qux { mC() {…} } class Qux { mC() {…} mE() {…} } version 1 version 2 version 3 version 4 27 17 changes 12 tracked changes 5 untracked changes
28. RQ1. What is the frequency of untracked changes? (example) class Foo { mA() {…} } class Bar { mB() {…} mC() {…} } class Foo { mA() {…} } class Bar { mX() {…} mC() {…} } class Foo { mA() {…} } class Bar { mX() {…} } class Foo { mA() {…} } class Baz { mY() {…} } class Qux { mC() {…} } class Qux { mC() {…} mE() {…} } version 1 version 2 version 3 version 4 28 17 changes 12 tracked changes 5 untracked changes
29. RQ1. What is the frequency of untracked changes? (example) class Foo { mA() {…} } class Bar { mB() {…} mC() {…} } class Foo { mA() {…} } class Bar { mX() {…} mC() {…} } class Foo { mA() {…} } class Bar { mX() {…} } class Foo { mA() {…} } class Baz { mY() {…} } class Qux { mC() {…} } class Qux { mC() {…} mE() {…} } version 1 version 2 version 3 version 4 29 Not desirable: relevant data may be missed !!! 17 changes 12 tracked changes 5 untracked changes
30. RQ1. What is the frequency of untracked changes? Untracked changes Classes 2% to 15% Methods 10% to 21% 30
31. RQ1. What is the frequency of untracked changes? Untracked changes Classes 2% to 15% Methods 10% to 21% 31 Untracked changes are frequent
32. RQ1. What is the frequency of untracked changes? Untracked changes Rename mtd: 26% Extract mtd: 23% Move mtd: 22% Move class: 12% 32
33. RQ1. What is the frequency of untracked changes? Untracked changes Rename mtd: 26% Extract mtd: 23% Move mtd: 22% Move class: 12% 33 Keeping track of renamings is not enough
34. RQ2 What is the extension of untracked changes? 34
35. class Foo { mA() {…} } class Bar { mB() {…} mC() {…} } class Foo { mA() {…} } class Bar { mX() {…} mC() {…} } class Foo { mA() {…} } class Bar { mX() {…} } class Foo { mA() {…} } class Baz { mY() {…} } class Qux { mC() {…} } class Qux { mC() {…} mE() {…} } version 1 version 2 version 3 version 4 7 paths 3 paths: only tracked changes 4 paths: at least one untracked changes RQ2. What is the extension of untracked changes? (example) 35
36. class Foo { mA() {…} } class Bar { mB() {…} mC() {…} } class Foo { mA() {…} } class Bar { mX() {…} mC() {…} } class Foo { mA() {…} } class Bar { mX() {…} } class Foo { mA() {…} } class Baz { mY() {…} } class Qux { mC() {…} } class Qux { mC() {…} mE() {…} } version 1 version 2 version 3 version 4 RQ2. What is the extension of untracked changes? (example) 36 1 2 3 7 paths 3 paths: only tracked changes 4 paths: at least one untracked changes
37. class Foo { mA() {…} } class Bar { mB() {…} mC() {…} } class Foo { mA() {…} } class Bar { mX() {…} mC() {…} } class Foo { mA() {…} } class Bar { mX() {…} } class Foo { mA() {…} } class Baz { mY() {…} } class Qux { mC() {…} } class Qux { mC() {…} mE() {…} } version 1 version 2 version 3 version 4 RQ2. What is the extension of untracked changes? (example) 37 1 2 3 4 7 paths 3 paths: only tracked changes 4 paths: at least one untracked changes
38. class Foo { mA() {…} } class Bar { mB() {…} mC() {…} } class Foo { mA() {…} } class Bar { mX() {…} mC() {…} } class Foo { mA() {…} } class Bar { mX() {…} } class Foo { mA() {…} } class Baz { mY() {…} } class Qux { mC() {…} } class Qux { mC() {…} mE() {…} } version 1 version 2 version 3 version 4 RQ2. What is the extension of untracked changes? (example) 38 1 2 3 4 Not desirable: their histories may be split !!! 7 paths 3 paths: only tracked changes 4 paths: at least one untracked changes
39. RQ2. What is the extension of untracked changes? 39 18% to 41% entities with at least one untracked change in their histories
40. RQ2. What is the extension of untracked changes? 22% to 58% entities with at least one untracked change in their histories Only considering the most changed entities 40
41. RQ2. What is the extension of untracked changes? 22% to 58% entities with at least one untracked change in their histories Only considering the most changed entities 41 Untracked changes cause splits in entity histories
42. RQ3. What is the impact of untracked changes in existing MSR-based approaches? • Approaches • API evolution mining rule (eg, Vector —> List) • API co-usage mining rule (eg, Map —> HashMap) • Results • Amount of mined rules: usually improves when taking into account untracked changes (median: 0% to +7%) • Quality of mined rules: slightly improves when including untracked changes (median: -2% to +2%) 42
43. RQ3. What is the impact of untracked changes in existing MSR-based approaches? • Approaches • API evolution mining rule (eg, Vector —> List) • API co-usage mining rule (eg, Map —> HashMap) • Results • Amount of mined rules: usually improves when taking into account untracked changes (median: 0% to +7%) • Quality of mined rules: slightly improves when including untracked changes (median: -2% to +2%) 43 The impact of untracked changes is difficult to predict, and needs to be evaluated in a case-by-case basis
44. Outline 1. Context 2. Problem 3. Background 4. Study Design 5. Results 6. Final Remarks 44
45. Untracked changes are frequent (10-21% at method level) MSR studies should resolve untracked changes to access potentially relevant new mining data Keeping track of renamings is not enough (≈26%) MSR studies should address “extraction” and “moving” for a more complete resolution of untracked changes Untracked changes cause splits in entity histories (18-41%) MSR studies should resolve untracked changes when performing traceability analysis, for more precise entity lifespans 45
46. Assessing the Threat of Untracked Changes in Software Evolution André Hora, Danilo Silva, Marco Tulio Valente, Romain Robbes ICSE 2018