Mechanisms to Characterize Context of Empirical Studies in Software Engineering

Transcript

1. Mechanisms to Characterize Context of Empirical Studies in Software Engineering Bruno Cartaxo bfsc@cin.ufpe.br Adauto Almeida ataf@cin.ufpe.br Emanoel Barreiros efsb@cin.ufpe.br Juliana Saraiva jags2@cin.ufpe.br Waldemar Ferreira wpfn@cin.ufpe.br Sergio Soares scbs@cin.ufpe.br 1
2. Background Some claim that the study’s context is a central concept in ESE.¹ BUT, Frequently papers report context in a generic and implicit way.² 1 – Dyba, et al. What works for whom, where, when, and why?: On the role of context in empirical software engineering. 2012. 2 – Sjøberg et al. The future of empirical methods in software engineering research. 2007. 2
3. Problem The lack of contextual information is an obstacle to • Replicate experiments.¹ • Transfer knowledge from academy to industry.2 • Integrate evidence on secondary studies.3 • Create families of experiments.4 • Investigate any contextual aspect that possibly had interfered in a research.5 3 1 – Lung, et al. On the difficulty of replicating human subjects studies in software engineering. 2008. 2 – Jedlitschka, et al. Towards evidence in software engineering. 2004. 3 – Lopes, et al. A conceptual framework for the development of applications centred on context and evidence-based practice. 2010. 4 – Basili, et al. Building knowledge through families of experiments. 1999. 5 – Montesi, et al. Software engineering article types: An analysis of the literature. 2008.
4. Goals Identify and classify the mechanisms that support context characterization of empirical studies in SE. 4
5. Context There are various definitions of context in different areas of knowledge. Dijk says that a theory of context could easily become a theory of everything.¹ 5 1 – Dyba, et al. What works for whom, where, when, and why?: On the role of context in empirical software engineering. 2012.
6. Operational definition of context in ESE The set of information that are not the main interest of an empirical research (in other words, they are not the study’s interventions), but they have an influence in the study’s results. 6
7. Research Method 7
8. Research method To identify and classify the mechanisms… We conducted a systematic mapping study following Kitchenham et al. guidelines.¹ 8 Studies Selection 1st Round 2nd Round Studies Search Data Extraction 1 – Kitchenham, et al. Guidelines for performing systematic literature reviews in software engineering. 2007.
9. Studies search 9 Studies Selection 1st Round 2nd Round Studies Search Data Extraction Automatic search sources IEEE ACM ScienceDirect Springer Link Scopus Engineering Village Manual search sources ESEM EASE ESEJ IST Snowballing Independent studies SEARCH SOURCES
10. Search string STRUCTURE <mechanism> AND <context> AND <empirical software engineering> STRING (mechanism OR framework OR method OR methodology OR guideline OR checklist OR process OR technique) AND (context OR contextual OR scope OR environment OR environmental OR settings OR circumstances OR situations OR situational OR population OR variables OR factors OR setup) AND (“empirical software engineering” OR “experimental software engineering” OR “software engineering experimentation” OR “evidence based software engineering”) 10
11. Studies selection – 1st Round 11 Studies Selection 1st Round 2nd Round Studies Search Data Extraction Title + Abstract
12. Studies selection – 2nd Round 12 Studies Selection 1st Round 2nd Round Studies Search Data Extraction 4 researchers pairs + Conflict Meetings 1. The study is a slideshow or an extended abstract. 2. The study is duplicated. 3. The study is not related to SE. 4. The study does not present or only mentions a mechanism to support. 5. Another justified reason.
13. Exclusion criteria evaluation 91.2% of agreement 13 K Agreement Level <0 Less than chance agreement 0,0 Poor Agreement 0,01-0,20 Slight Agreement 0,21-0,40 Fair Agreement 0,41-0,60 Moderate Agreement 0,61-0,80 Substantial Agreement 0,81-0,99 Almost Perfect Agreement Pair K p-value Todas 0,5873544 3,587x10-13 1 0,9097174 6,392x10-10 2 0,8068796 1,307x10-7 3 -0,06353689 0,3311139 4 0,4706173 0,001641
14. Data Extraction 14 Studies Selection 1st Round 2nd Round Studies Search Data Extraction General information + Research questions RQ1: What are the mechanisms to support context characterization? RQ2: What are the types of mechanisms to characterize context? RQ3: What are the types of context covered by the mechanisms? RQ4: What are the empirical methods covered by the mechanisms? RQ5: What are the SE topics covered by the mechanisms?
15. Results 15
16. Results – Selection procedure Sources Initial 1st Round 2nd Round Selected IEEE 312 96 11 5 ACM 1.744 168 17 2 ScienceDirect 996 51 5 1 SpringerLink 2.061 51 5 1 Engineering Village 292 34 5 1 Scopus 4.000 86 13 0 ESEM 373 1 0 0 EASE 137 6 0 0 ESEJ 431 3 0 0 IST 3.009 31 4 0 Snowballing - - - 0 Independent - - - 3 TOTAL 13.355 527 60 13 16
17. Results – Research Question 1 What are the mechanisms to support context characterization … ? 17 CODE TITLE SPS01 An Initial Framework for Research on Pair Programming SPS02 Building Knowledge through Families of Experiments SPS03 Context in industrial software engineering research SPS04 Empirical studies in reverse engineering: state of the art and future trends SPS05 Experimental Context Classification: Incentives and Experience of Subjects SPS06 The situational factors that affect the software development process: Towards a comprehensive reference framework SPS07 Using Context Distance Measurement to Analyze Results across Studies SPS08 What Works for Whom, Where, When, and Why? On the Role of Context in Empirical Software Engineering SPS09 Collecting Feedback During Software Engineering Experiments SPS10 Preliminary Guidelines for Empirical Research in Software Engineering SPS11 Extreme Programming Evaluation Framework for Object-Oriented Languages Version 1.4 SPS12 Rational Unified Process Evaluation Framework Version 1.0 SPS13 Experimentation in Software Engineering: An Introduction
18. Results – Research Question 2 What are the types of mechanisms¹ to characterize context …? 18 Mechanism Studies Framework SPS1, SPS2, SPS4, SPS5, SPS6, SPS11, SPS12 Guidelines SPS4, SPS10 Template SPS7, SPS13 Checklist SPS3, SPS4 Method SPS9 Paradigm SPS8 Processs - Techinique - Lessons Learned - Framework 7 (46,7%) Guideline 2 (13,3%) Method 1 (6,7%) Paradigm 1 (6,7%) Template 2 (13,3%) Checklist 2 (13,3%) 7 out of 13 studies present frameworks 1 – Almeida, et al. Um Mapeamento Sistem´atico de Mecanismos para Guiar Estudos Empíricos em Engenharia de Software. 2011.
19. Results – Research Question 3 What are the types of context¹ covered by the mechanisms? Context Type Studies Discrete SPS1, SPS2, SPS3, SPS4, SPS5, SPS6, SPS7, SPS9, SPS10, SPS11, SPS12, SPS13 Omnibus SPS 8 19 Discrete 12 (92,3%) Omnibus 1 (7,7%) Only one study adopts omnibus context. 1 – Dyba, et al. What works for whom, where, when, and why?: On the role of context in empirical software engineering. 2012.
20. Results – Research Question 4 What are the empirical methods¹ covered by the mechanisms? Method Studies Controlled experiment SPS1, SPS2, SPS4, SPS5, SPS7, SPS9, SPS10, SPS11, SPS12, SPS13 Case study SPS1, SPS3, SPS8, SPS10, SPS11, SPS12 Ethnography SPS8 Action research SPS8 Survey - 20 Controlled experiment 10 (55,6%) Case Study 6 (33,3%) Ethnography 1 (5,6%) Action Research 1 (5,6%) 10 out 13 support controlled experiments. 1 – Easterbrook et al. Selecting empirical methods for software engineering research. 2008.
21. Results – Research Question 5 What are the SE topics¹ covered by the mechanisms? 21 Topic Studies Software construction SPS1, SPS11 Software process SPS6, SPS12 Software maintenance SPS4 Software engineering management SPS5 Software requirements - Software design - Teste de software - Software configuration management - Tools and methods - Software quality - Software construction 2 (33,3%) Software maintenance 1 (16,7%) Software management 1 (16,7%) Software process 2 (33,3%) Only 4 of the 10 SE topics are covered by the mechanisms. 1 – Abran et al. Guide to the Software Engineering Body of Knowledge -SWEBOK. 2007.
22. Summarizing Findings Table to help researchers to choose the mechanism that best fits their empirical method and SE research topic. 22 Empirical Method Software Engineering Topic Studies Controlled experiment Software construction SPS01, SPS11 Software process SPS12 Software maintenance SPS04 Software engineering management SPS05 - SPS02, SPS06, SPS07, SPS09, SPS10, SPS13 Case Study Software construction SPS01, SPS11 Software process SPS12 - SPS03, SPS08, SPS10 Ethnography - SPS08 Action research Software quality SPS08
23. Concluding Remarks We analyzed 13,355 studies, and only 13 were selected. There are many claims about the need for more and better context characterization of empirical studies in SE. 1,2,3,4 BUT, This research showed that there are few mechanisms that support context characterization in SE. 23 1 – Dyba, et al. What works for whom, where, when, and why?: On the role of context in empirical software engineering. 2012. 2 – Jedlitschka, et al. Towards evidence in software engineering. 2004. 3 – Petersen, et al. Context in industrial software engineering research. 2009. 4 – Sjøberg, et al. The future of empirical methods in software engineering research. 2007.
24. Future Work • Investigate how the context of the empirical studies has been characterized. • Define a set of relevant context information for empirical studies focused on each software engineering topic. • Investigate the context characterization for each specific empirical method. 24
25. THANK YOU 25 Bruno Cartaxo bfsc@cin.ufpe.br