Software Testing in 2048 Arie van Deursen Delft University of Technology @avandeursen Symposium Software Engineering Reflections for Practice and Research Vrije Universiteit, Amsterdam, 15 January 2016 

2 Why Testing is Hard • A simple program: 
 3 inputs, 1 output • a,b,c: 32 bit integers • trillion test cases / s. All plants dead All oceans dry Tests done (2.5 bill. y) 

The JUnit Success Story • Open source Java unit testing framework • Major accelerator for 
 (1) developer-written unit tests;
 (2) continuous automated test execution. • "Never in the field of software development have so many owed so much to so few lines of code”. — Martin Fowler. Erich Gamma, Kent Beck. JUnit, A Cook’s Tour. Java Report, May 1999 3 

Design By Contract • During design, explicitly consider • Preconditions • Postconditions • Class and other structural invariants • At run time, use assertions to enforce them: • Raise exception if contract is violated • Also can serve as oracle during test execution 4 Bertrand Meyer, Object-Oriented Software Construction, 1988 

Property-Based Testing • Formulate properties (invariants) • Specify generators that produce (constrained) random data aimed at falsifying invariants. • Search for simplest test case that violates property • QuickCheck for Haskell, ScalaTest, Erlang,, successes at Ericsson, Motorola, Volvo, … 5 Koen Claessen and John Hughes. QuickCheck: A Lightweight Tool for Random Testing, ICFP 2000. 

Learning Likely Invariants • Repeated execution (tests, production)
 of instrumented program points • Discover repeatedly recurring relationships • 0 < x < 100, a list being sorted, set membership, pointer relationships, … • Scaleability, usefulness, stability, ‘interestingness’ 6 Michael D. Ernst et al. Dynamically discovering likely program invariants to support program evolution. IEEE TSE, 2001 

Web Invariants? • Modern web-apps fully JavaScript based • Automated crawling: Click through web app, and build up state diagram • Automated oracle: Generic or application-specific
 invariants on, e.g., the DOM tree. • Tool: Crawljax, used at Google, Fujitsu, eBay, SAP, … 7 Ali Mesbah, Arie van Deursen, Danny Roest. Invariant-based automatic testing of modern web applications. IEEE TSE, 2012 

The State of the Web • Crawled 4000 apps, resulting in 1M unique states • Structural errors: Non-unique ids, doctype declaration, broken layouts. • Performance: header declarations, compression, stylesheets in head, non-blocking scripts, … • Accessibility: navigational assistance, tables, input labels 8 Alex Nederlof, Ali Mesbah, Arie van Deursen. Software engineering for the web: the state of the practice. ICSE SEIP, 2014. Poor 

Model-Based Testing • Use models to determine which test cases to derive • Reason about transition order, state reachability, path independence, … 9 Jan Tretmans. Model based testing with labelled transition systems. FMT, 2008 Arie van Deursen. Testing web applications with state objects. CACM, 2015 

Learning State Machines • Passive learning / log monitoring • Active learning / triggering special cases. • Discover states that permit same event possibilities • Merge candidate states, events, discover repetition, conditions, concurrent or super states,… 10 Fides Aarts, Harco Kuppens, Jan Tretmans, Frits Vaandrager, Sicco Verwer. Improving active Mealy machine learning for protocol conformance testing. ML 2014 

Test Input Generation • Formulate testing as an optimization problem: • Minimize number of test cases;
 Maximize number of detected bugs • Meta-heuristic search:
 Genetic algorithm. 11 Gordon Fraser, Andrea Arcuri. Evosuite: automatic test suite generation for object-oriented software. ESEC/FSE 2011. 

Beyond Coverage: Diversity! • Testing: Try to trigger failures. • Diagnosis: Try to find the corresponding fault. • Automated diagnosis calls for test diversity • Many smaller unit tests;
 Many different interactions in integration tests 12 J. Campos, R. Abreu, G. Fraser, M. d’Amorim. Entropy-Based Test Generation for Improved Fault Localization. ASE 2013 

DevOps: Learning from Production Image Credit: collab.net 

Testing in 2048 • Continuous automated learning
 from running systems, in test or production • Continuous self-triggering, self-checking, 
 and self-diagnosis, in test or production • Supported by explicit abstractions: 
 designed, refined, verified, present at run time. 14