Learning Effective Changes For Software Projects

Transcript

1. arxiv.org/abs/1708.04589 Learning Effective Changes in Software Engineering Rahul Krishna Home page: rkrsn.us August 31st, 2017
2. arxiv.org/abs/1708.04589 From Prediction to Planning — Panelists at ASE 2015 Actionable Analytics Workshop “ … Most [software analysis] tools tell us what is, But they don’t tell us what to do … ” 2
3. arxiv.org/abs/1708.04589 From Prediction to Planning “ … Most [software analysis] tools tell us what is, But they don’t tell us what to do … ” Prediction What is … ?? — Panelists at ASE 2015 Actionable Analytics Workshop 3
4. arxiv.org/abs/1708.04589 From Prediction to Planning “ … Most [software analysis] tools tell us what is, But they don’t tell us what to do … ” Prediction What is … Planning What to do … 1. Offer clear guidance 2. Least changes most impact ?? — Panelists at ASE 2015 Actionable Analytics Workshop 4
5. arxiv.org/abs/1708.04589 Terminologies • Points to the likelihood of defects • Takes in inputs vectors: ◦ x = (x 1 , x 2 , … , x n ) • Returns ◦ P(Y | x 1 , x 2 , … , x n ) Prediction 5
6. arxiv.org/abs/1708.04589 Terminologies • Points to the likelihood of defects • Takes in inputs vectors: ◦ x = (x 1 , x 2 , … , x n ) • Returns ◦ P(Y | x 1 , x 2 , … , x n ) • Offers a set of actionable plans • Plans return a new vector ◦ x’ = (x 1 ’, x 2 ’, … , x n ’ ) • Such that ◦ P(Y | x’) < P(Y | x) Prediction Planning 6
7. arxiv.org/abs/1708.04589 A Motivating Example
8. arxiv.org/abs/1708.04589 8
9. arxiv.org/abs/1708.04589 9
10. arxiv.org/abs/1708.04589 10
11. arxiv.org/abs/1708.04589 11
12. arxiv.org/abs/1708.04589 Where does this come from? Are these plans actionable? How do developers perceive these plans? 12
13. arxiv.org/abs/1708.04589 Where does this come from? Are these plans actionable? How do developers perceive these plans? This talk 13
14. arxiv.org/abs/1708.04589 Where does this come from? Are these plans actionable? How do developers perceive these plans? Open Issue, to be discussed This talk 14
15. arxiv.org/abs/1708.04589 Planning in SE: Framework 15
16. arxiv.org/abs/1708.04589 Planning in SE: Current Methods • Monte Carlo simulation [Boehm00, Menzies05] ◦ Also computationally expensive ◦ Requires methods to interpret outcomes • Simple clustering techniques [Borges12] ◦ They’re unsupervised ◦ Doesn’t work well for SE • Statistical Thresholding [Alves10, Shatnawi10, Oliveira14] ◦ Recommends lots of spurious changes 16
17. arxiv.org/abs/1708.04589 Planning in SE: Current Methods • Genetic Algorithms [Ouni13, Mkaouer14] ◦ Requires domain models ◦ Offers single point solutions • Deep Learning [Xu16, Fu17] ◦ Again, computationally intensive and slow ◦ Uninterpretable ◦ It’s a black-box 17
18. arxiv.org/abs/1708.04589 XTREE: Motivation • A useful planner must: ◦ Be computationally inexpensive ◦ Be interpretable ◦ Offer a variety of recommendations ◦ Recommend useful changes • To address these, we built XTREE 18
19. arxiv.org/abs/1708.04589 • XTREE is a Cluster+Contrast planner ◦ Cluster parts of the software project ◦ Find contrast sets between neighbouring clusters • These contrast sets are “plans” ◦ Current project falls into cluster C 1 ◦ Neighboring cluster C 2 has better quality ◦ The difference ∆ = C 2 – C 1 is a plan XTREE: Construction 19
20. arxiv.org/abs/1708.04589 XTREE: Construction 20
21. arxiv.org/abs/1708.04589 XTREE: Construction 21
22. arxiv.org/abs/1708.04589 XTREE: Construction 22
23. arxiv.org/abs/1708.04589 XTREE: Construction 23
24. arxiv.org/abs/1708.04589 XTREE: Construction 24
25. arxiv.org/abs/1708.04589 Which current branch does a test case fall in? XTREE: Usage 25
26. arxiv.org/abs/1708.04589 Which desired branch would you want to move to? XTREE: Usage 26
27. arxiv.org/abs/1708.04589 What are the deltas between current & desired branches? XTREE: Usage 27
28. arxiv.org/abs/1708.04589 What are the deltas between current & desired branches? Refactor such that coupling among methods improves XTREE: Usage Ensure a class only contains methods that interact with each other 28
29. arxiv.org/abs/1708.04589 Benchmark Datasets A total of 54 datasets 29
30. arxiv.org/abs/1708.04589 1. Deploy the planner to recommend plans; 2. Alter the test data according to these plans; 3. Apply a verification oracle to the altered datatest to estimate defects; Assessing Plans 30
31. arxiv.org/abs/1708.04589 1. Deploy the planner to recommend plans; 2. Alter the test data according to these plans; 3. Apply a verification oracle to the altered datatest to estimate defects; Assessing Plans Use different data to train a verification oracle 31
32. arxiv.org/abs/1708.04589 Results: Research Questions Does planning with XTREE offer significant improvements? Does using XTREE offer any benefits over other planning methods? How many features does XTREE recommend changing? 32
33. arxiv.org/abs/1708.04589 Results: Research Question 1 Does planning with XTREE offer significant improvements? XTREE offers statistically significant improvements 33
34. arxiv.org/abs/1708.04589 Results: Research Question 2 Does using XTREE offer any benefits over other planning methods? Plans derived from XTREE have significantly better performance and much lower variance compared to other planners. 34
35. arxiv.org/abs/1708.04589 Results: Research Question 3 How many features does XTREE recommend changing? XTREE recommends changing only a fraction of the metrics compared to other methods. 35
36. arxiv.org/abs/1708.04589 Next Steps 36
37. arxiv.org/abs/1708.04589 Next Steps: Industrial Validation 1. Industrial Validation* a. Validate usefulness of plans 37 * Ongoing at RTP Raleigh
38. arxiv.org/abs/1708.04589 • 1600+ live projects at RTP* Next Steps: Industrial Validation 38 * Ongoing at RTP Raleigh
39. arxiv.org/abs/1708.04589 1. Industrial Validation* a. Validate usefulness of plans b. What if plans are not actionable? Next Steps: Actionable Plans 39 * Ongoing at RTP Raleigh
40. arxiv.org/abs/1708.04589 1. Industrial Validation* a. Validate usefulness of plans b. What if plans are not actionable? Recommendation s from XTREE XTREE Next Steps: Actionable Plans 40 * Ongoing at RTP Raleigh
41. arxiv.org/abs/1708.04589 1. Industrial Validation* a. Validate usefulness of plans b. What if plans are not actionable? Next Steps: Actionable Plans Recommendation s from XTREE Recommendations from developers 41 * Ongoing at RTP Raleigh
42. arxiv.org/abs/1708.04589 1. Industrial Validation* a. Validate usefulness of plans b. What if plans are not actionable? Next Steps: Actionable Plans Recommendation s from XTREE Recommendations from developers XTREE Recommendations from developers Future Work 42 * Ongoing at RTP Raleigh
43. arxiv.org/abs/1708.04589 1. Industrial Validation* a. Validate usefulness of plans b. What if plans are not actionable? 2. Developer Surveys a. Willingness to use such tools b. Online vs. post-hoc planning Next Steps: Developer Survey 43 * Ongoing at RTP Raleigh
44. arxiv.org/abs/1708.04589 1. Industrial Validation* a. Validate usefulness of plans b. What if plans are not actionable? 2. Developer Surveys a. Willingness to use such tools b. Online vs. post-hoc planning 3. Scalable planning Next Steps: Scalable Plans 44 * Ongoing at RTP Raleigh
45. arxiv.org/abs/1708.04589 1. Other domains to test plans a. Effect of planning on defect prediction takes time. b. Domains where verification oracles are easier to obtain. 2. Best approaches to developer surveys a. Currently, we plan on A/B testing. b. Avenues for conducting surveys. 3. Alternatives to developer surveys a. Anything else, before surveys? 4. Other comments highly appreciated Possible Discussion Topics * Ongoing at RTP Raleigh 45
46. arxiv.org/abs/1708.04589 Questions & Comments 46