The Role Of Rapid Reviews Supporting Decision-making In Software Engineering Practice

Transcript

1. THE ROLE OF RAPID REVIEWS SUPPORTING DECISION-MAKING IN SOFTWARE ENGINEERING PRACTICE Bruno Cartaxo [email@brunocartaxo.com] Sérgio Soares [scbs@cin.ufpe.br] Gustavo Pinto [gpinto@ufpa.br]
2. In 2004... 2
3. HOWEVER.... MANY ARE CLAIMING THAT SOFTWARE ENGINEERING RESEARCH LACKS CONNECTION WITH PRACTICE IN PARTICULAR SYSTEMATIC REVIEWS 3
5. EVIDENCE BASED MEDICINE (EBM) ALSO FACED A SIMILAR PROBLEM 5
6. EBM HAS BEEN APPROACHING THAT TROUGHT... RAPID REVIEWS 6
7. RRs are lightweight Secondary Studies (SSs) focused on delivering evidence to practitioners in a timely manner 1,2 Some steps of Full Systematic Reviews (FSRs) are deliberately omitted or simplified in RRs to achieve their proposed goal RAPID REVIEWS 1. Tricco et al. A scoping review of rapid review methods. BMC Medicine, 2015 2. Hartling et al. A taxonomy of rapid reviews links report types and methods to specific decision-making contexts. Journal of Clinical Epidemiology, 2016 7
8. GROWING INTEREST ON RAPID REVIEWS IN MEDICINE 8
9. 56% of rapid reviews were conducted on the last 3 years 9
10. RAPID REVIEWS ARE NOT INTENDED TO REPLACE FULL SYSTEMATIC REVIEWS 10
11. CHARACTERISTICS • They reduce costs of heavyweight methods • They deliver evidence in a timely manner • They are performed in close collaboration with practitioners • They aim to provide evidence to issues practitioners are facing in practice • They report results through appealing mediums 11
12. STRATEGIES EMPLOYED 12 TIMELY EVIDENCE AND REDUCED COSTS • Limiting search strategy by date of publication and/or search source; • Using just one person to screen studies; • Not conducting quality appraisal of primary studies; • Presenting results with no formal synthesis COLLABORATION WITH PRACTITIONERS • Bounded to a practical problem, inside a practical context • Practitioners must actively participate in the process
13. APPEALING MEDIUMS 13
14. NO REPORT OF RAPID REVIEWS IN SOFTWARE ENGINEERING 14
15. THE GOAL OF THIS RESEARCH IS TO CONDUCT AND EVALUATE A RAPID REVIEW IN SOFTWARE ENGINEERING CONTEXT 15
16. What are practitioners perception on using Rapid Reviews to support informed decision-making in software engineering practice? RQ 16
17. RESEARCH CONTEXT 17 COMPANY Applied-research institute in Recife, Brazil PROJECT Development of a system to monitor reusable packages during the entire production chain of the automotive industry PARTICIPANTS • Projects coordinator - MSc + 15 years of experience • Project manager - MSc + 2.5 years of experience LIMITATIONS AND THREATS • Perceptions vs. Effects • Collective Interview • Generalizability
18. DIAGNOSIS Semi-Structured Interview PLANNING EXECUTION EVALUATION REFLECTION 18 RESEARCH STEPS
19. DIAGNOSIS RR Protocol Definition RR planned to one week PLANNING EXECUTION EVALUATION REFLECTION 19 RESEARCH STEPS
20. DIAGNOSIS 6 days of RR Execution Workshop to Present Results PLANNING EXECUTION EVALUATION REFLECTION 20 RESEARCH STEPS
21. DIAGNOSIS Introduction Interview Adoption Interview (2 months after) PLANNING EXECUTION EVALUATION REFLECTION 21 RESEARCH STEPS
22. DIAGNOSIS LESSONS LEARNED PLANNING EXECUTION EVALUATION REFLECTION 22 RESEARCH STEPS
23. PRACTITIONERS PROBLEM Low customer collaboration RESEARCH QUESTIONS • RQ1: What are the benefits of customer collaboration in software development practice? • RQ2: What are the problems caused by low customer collaboration in the software development practice? • RQ3: What are the challenges to establish customer collaboration in software development practice? • RQ4: What are the strategies to improve customer collaboration in software development practice? RAPID REVIEW PROTOCOL 23
24. RAPID REVIEW PROTOCOL SEARCH PROCEDURE 24 (customer OR “product owner” OR stakeholder) AND (collaborat* OR participat* OR cooperat* OR relation* OR involvement OR engagement) AND (“software development” OR “software engineering” OR “software project”) SELECTION PROCEDURE
25. RAPID REVIEW PROTOCOL SYNTHESIS PROCEDURE Thematic Analysis LIMITATIONS • Only one search engine was used • Selection procedure was conducted by a solo researcher, which may introduce selection bias; • No quality assessment on the primary studies, which might limit the reliability on the evidence; • The selection of papers by title consists in another threat; 25 EVIDENCE REPORT
27. INTRODUCING RAPID REVIEW IN PRACTICE 27 13 POSITIVE POINTS 4 NEGATIVE POINTS
28. Applicable to software engineering practice 28
29. Offer reliable content 29
30. Improve problem comprehension 30
31. Discussing the rapid reviews’ findings is needed 31
32. Primary studies context far from their findings 32
33. Adopting Rapid Reviews Evidence WHAT WORKED • Some strategies were successfully applied • Evidence Briefing as a reference material • Evidence Briefing to discuss with team members • Motivated them to seek additional knowledge WHAT DID NOT WORK • Some strategies turned out to be not applicable ◦ e-collaboration tools ◦ Just Demos 33 2 MONTHS LATER...
34. REFLECTING BEYOND THE RESULTS... 34
35. Asking the Right Questions! Exploratory Questions • RQ4: What are the strategies to improve customer collaboration in software development practice? Motivational Questions • RQ1: What are the benefits of customer collaboration in software development practice? • RQ2: What are the problems caused by low customer collaboration in the software development practice? 35
36. “it may be time to reverse the perspective and start from the consumer’s perspective: for practitioners of software engineering, what problems, not yet satisfactorily answered by software engineering theory, could benefit, in the search for answers, from empirical studies?” Bertrand Meyer, 2018 IN CONCLUSION... 36
37. Meyer’s voice certainly is not alone Many other researchers are starting to recognize practice orientation as the next long way ahead1,2,3,4 1. Beecham, et al. Making software engineering research relevant. Computer, 2014 2. Duarte et al. Patterns of cooperative technology development and transfer for software-engineering-in-the-large. SERIP, 2015 3. Laird et al. Transferring software engineering research into industry: The stevens way. SERIP, 2015 4. Santos et al. Motivation to perform systematic reviews and their impact on software engineering practice. ESEM, 2013 37 We believe Rapid Reviews can be one of the possible approaches to make software engineering research closer to practice So, considering their inherent limitations
38. FUTURE DIRECTIONS More RRs in SE Graphical synthesis Compare RRs with FSSs Impact of RRs in practice Perception of researchers on RRs Coverage analysis for specific SE areas Creation of EBs templates for different audiences 38
39. THANK YOU Bruno Cartaxo [email@brunocartaxo.com] 39