A Pilot Study on Software Quality Practices in Belgian Industry

In the context of an ERDF-funded project port- folio, we have carried out a survey to assess the state-of-the- practice in software quality in Belgian companies. With this survey, we wish to find out what are the most common industry practices (processes, techniques and tools) with respect to soft- ware quality, and how these practices vary across companies. Companies could use the results of this study to improve upon their current software quality practices compared to other companies. Researchers could use it to develop better techniques and tools for aspects that have not found sufficient take-up by industry. Teachers may use it to adapt their courses to become more directly relevant to industry practices. Presented at CSMR 2013 (http://csmr2013.disi.unige.it).

Transcript

1. A pilot study on software quality practices in Belgian industry Javier Perez Ansymo group, University of Antwerp (Work done while at University of Mons) Tom Mens Service de Génie Logiciel, University of Mons Presented at CSMR 2013 – Industrial Track March 7, Genova, Italy Flora Kamseu, Naji Habra PRECISE lab, University of Namur
2. Context •  ERDF project - Portefeuille TIC –  CEIQS: Center of expertise in engineering and quality of systems •  aimed at developing a portfolio of innovative techniques allowing local companies to master the diversity, complexity, quality and rapid evolution of information systems –  lead by CETIC (2007-2013) –  Supported by Wallonia –  Collaboration on software quality between universities of Namur and Mons since 2010 2 CSMR 2013 – Industrial Track – Genova, Italy, March 2013
3. Motivation & Objectives •  Which are the most frequently used practices in industry? •  Relate this to what is being taught in academia –  Is there a gap between teaching and industry needs? –  What good methodologies, practices, tools are not being used and why? •  Study quality-related software development practices in industry –  Through online questionnaires –  Compare this across different countries and company sizes 3 CSMR 2013 – Industrial Track – Genova, Italy, March 2013
4. Online Surveys •  Using LimeSurvey –  Respondents = employees of companies involved in software development or software maintenance in the targeted region/ country –  Convenience sampling : invitations sent to potential respondents •  Pilot survey in Belgium –  During June 2012 –  Respondents mainly from Wallonia and Brussels •  Follow-up survey in Europe –  From October till November 2012 –  Respondents mainly from Netherlands, Belgium, Spain, France 4 CSMR 2013 – Industrial Track – Genova, Italy, March 2013
5. Pilot Survey in Belgium •  3 languages (French, Dutch, English) •  188 potential respondents invited by mail •  71 respondents (response rate 37,8%) •  47 companies •  44 complete questionnaires retained for analysis –  Incomplete responses were ignored –  Multiple responses from same companies were aggregated 5 CSMR 2013 – Industrial Track – Genova, Italy, March 2013 PILOT SURVEY IN BELGIUM
6. About the Pilot Survey •  31 questions (15-20 minutes) •  Addressing the following topics –  Development processes –  Structural quality measurement and improvement –  Quality models and quality standards –  Testing –  Development team support 6 CSMR 2013 – Industrial Track – Genova, Italy, March 2013 PILOT SURVEY IN BELGIUM
7. Respondents Origin Company size 7 CSMR 2013 – Industrial Track – Genova, Italy, March 2013 PILOT SURVEY IN BELGIUM
8. Development Processes Usage •  Use of a well-defined and documented development process? 45,5% agile or scrum, Prince 2, RUP, ISO certification, proprietary •  Use of agile practices or methods? 63,6% 17 mention SCRUM, 2 mention Prince2 8 CSMR 2013 – Industrial Track – Genova, Italy, March 2013 PILOT SURVEY IN BELGIUM 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Agile Process Yes No I don't know 45,5% 63,6%
9. Techniques & Tools for Quality Improvement 9 CSMR 2013 – Industrial Track – Genova, Italy, March 2013 PILOT SURVEY IN BELGIUM 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Dynamic Analysis Design Improvement Metrics and Visualisation Refactoring Often or Continuously Scarcely or Never 58,5% <43%
10. Explicit Quality Support 10 Poor adoption of quality (models, standards, processes, certifications…) 36,8% (14/38) 32,4% (12/37) 19,4% (7/36) CSMR 2013 – Industrial Track – Genova, Italy, March 2013 PILOT SURVEY IN BELGIUM
11. Testing – Dedicated Dev. Time 11 CSMR 2013 – Industrial Track – Genova, Italy, March 2013 PILOT SURVEY IN BELGIUM
12. Change or Configuration Management Process 12 CSMR 2013 – Industrial Track – Genova, Italy, March 2013 PILOT SURVEY IN BELGIUM 73,8% (31/42)
13. Tools - Use of Integrated Platform for 13 CSMR 2013 – Industrial Track – Genova, Italy, March 2013 PILOT SURVEY IN BELGIUM 97,6% 92,7% 57,1% 55,6% 60,5%
14. Effect of Company Size 14 § Agile  prac+ces  popular  regardless  of  company  size   § No  clear  difference  in  development  process  across  company  sizes   § Quality  process  mostly  used  by  big  and  medium  companies     § Micro  companies  behave  differently  (for  processes,  agile  and  q.  models)     PILOT SURVEY IN BELGIUM CSMR 2013 – Industrial Track – Genova, Italy, March 2013
15. Summary of Results quality improvement tools and techniques 15 Most  popular   § Version  control  (97,6%)  and  bug  tracking  (92,7%)  plaIorms   § Widespread  use  of  tes+ng:  97,7%   § Design  paMerns:  72,7%   § Refactoring:  58,5%   Moderate  popularity   § Design  improvement  (e.g.  code  smell  reduc+on):  42,9%   § Bad  quality  detec+on  tools:  36,8%   § Metrics  and  visualisa+on  tools:  35,7%   § Dynamic  analysis  tools  (profiling  etc.):  36,6%   Unpopular   § Use  of  quality  models:  19,4%   PILOT SURVEY IN BELGIUM CSMR 2013 – Industrial Track – Genova, Italy, March 2013
16. Summary of Results process related •  Most  popular   –  Agile  prac+ces  (63,6%)   –  Change  and  configura+on  management  processes  (73,8%)   •  Moderate  popularity   –  Development  processes  (45,5%)   –  Test  processes  (46,5%)   –  Quality  support/improvement  process  (32,4%)   All  respondents  agree  that  quality  assurance  and   tes.ng  are  very  important  for  project  success   16 PILOT SURVEY IN BELGIUM CSMR 2013 – Industrial Track – Genova, Italy, March 2013 16 1 2 3 4 5 Quality assurance Testing Average : 4 4,2
17. Other Relevant Observation •  Wide range of –  programming languages •  Java (28), .Net (12), C++ (10), PHP (10), SQL/PL (9), JavaScript (8), Cobol (7), Windev (6), C (5), … –  development environments •  Eclipse (16), Visual Studio (14), NetBeans (4), Windev (3), IntelliJ (2), Delphi (2), Powerbuilder (2), vim (2), … –  testing tools •  unit testing frameworks (8), Mantis (4), HP Quality Center (4), Selenium (3), Hudson (2), Jenkins (2), Quick Test Pro (2), … –  Tools for modeling and documentation •  Visio (10), Word (8), Sparx Enterprise Archictect (5), Doxygen (3), StarUML (2), Confluence (2), … –  Little or no support for model-code synchronisation 25 CSMR 2013 – Industrial Track – Genova, Italy, March 2013 PILOT SURVEY IN BELGIUM
18. Our Main Concern •  Mitigated success of static and dynamic code analysis tools for detecting quality issues, visualisation, computing metrics, profiling, etc… –  popularity between 35% and 43% –  Does not reflect the high perceived importance of quality assurance for software project success 26 CSMR 2013 – Industrial Track – Genova, Italy, March 2013 PILOT SURVEY IN BELGIUM
19. Follow-up survey in Europe •  4 languages (French, Dutch, English, Spanish) •  270 respondents (until 5 December) •  163 useful responses •  80 from NL (49%) •  30 BE (18%) •  29 ES (18%) •  24 FR (15%) •  >117 companies •  Analysis of results still in progress. 27 CSMR 2013 – Industrial Track – Genova, Italy, March 2013 FOLLOW-UP SURVEY IN EUROPE
20. Preliminary Conclusions •  Differences between countries –  No striking differences observed across countries –  Small differences for some particular cases •  Differences in company size –  Usage increases by company size •  Processes, usage and updates of design documents, testing processes and dedicated time, quality processes and models –  No difference observed for agile methodologies 29 FOLLOW-UP SURVEY IN EUROPE CSMR 2013 – Industrial Track – Genova, Italy, March 2013
21. What’s next? •  Analyse results in more detail –  Statistically correlate results according to different criteria •  company size, training level, national differences –  Use data mining techniques to reveal less obvious relations •  Understand the implications of the results –  Confirm results with interviews –  Get feedback from companies 30 CSMR 2013 – Industrial Track – Genova, Italy, March 2013
22. 31 CSMR 2013 – Industrial Track – Genova, Italy, March 2013