Automation in Software Testing. Humans and Complex Models.

Iosif Itkin, CEO and co-founder, Exactpro

The QA Financial Forum: London 2021
9 November 2021

To learn more about Exactpro, visit our website https://exactpro.com/

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro

Transcript

1. Automation in Software Testing. Humans and Complex Models. Iosif Itkin CEO and co-founder, Exactpro
2. About Exactpro
4. MANAGED SUPPORT SERVICES PROVIDER OF THE YEAR In this Risk.net interview, Alexey Zverev, Exactpro co-CEO and co-founder, talks about mitigating risks while driving innovation during the time of high volatility and explains what made Exactpro stand out among the contestants. Watch the interview with Risk.net via the QR code above.
5. • https://www.linkedin.com/in/iosifitkin/ • Exactpro representative at the Zero Outage Industry Standard association • Head of LOST (Laboratory of Software Testing) at Tomsk Polytechnic University • Co-chair of FIX Trading Community MOST Working Group (Monitoring, Onboarding and Software Testing) About me
6. What is Software Testing
7. Art, Craft, Science, Engineering Where does software testing belongs on this continuum?
8. Science and Engineering https://fs.blog/2013/07/the-difference-between-science-and-engineering/ Abstract Model = theory Concrete Description = data Physical System = object of study flow of information measure compare Abstract Model = design concept Concrete Description = specification Physical System = useful product flow of information design produce Scientific Inquiry Engineering Design The Antiparallel Structures of Scientific Inquiry and Engineering Design
9. Software Testing Software Testing is exploring software with the intent of finding bugs. Software Testing is an empirical technical investigation conducted to provide stakeholders with information about the quality of the product or service under test.
10. Software Testing Abstract Model = theory Concrete Description = data Software = object of study flow of information observe compare Abstract Model = design concept Concrete Description = specification Software = useful product flow of information design produce Software Testing Development The Antiparallel Structures of Software Testing and Development
11. Never invest in a business you cannot understand “ Warren Buffett
12. Understanding Your Technology Assets Models Abstract Model = theory Concrete Description = data Software = object of study flow of information observe compare Abstract Model = design concept Concrete Description = specification Software = useful product flow of information design produce Software Testing Development The Antiparallel Structures of Software Testing and Development
13. All Testing is Model Based Testing A model is a simplified, formal representation of a relationship, process or system. The simplification makes some aspects of the thing modeled clearer, more visible, and easier to work with. All tests are based on models, but many of those models are implicit. When the behavior of the program “feels wrong”, it is clashing with your internal model of the program and how it should behave. Black Box Software Testing Cem Kaner & James Bach
14. Properties of a Model Herbert Stachowiak. Allgemeine Modelltheorie. Springer-Verlag, Wien and New York, 1973. (Cited by Kühne, Thomas. "What is a Model?." In Dagstuhl Seminar Proceedings. Schloss Dagstuhl-Leibniz-Zentrum für Informatik, 2005) mapping feature reduction feature pragmatic feature
15. BA DEV QA Behavior Driven Development Given Everything is Broken When Time is Up Then Build Back Better
16. The best material model of a cat is another, or preferably the same, cat. Arturo Rosenblueth ”
17. Parallel Run / Operational Day Replay
18. Bill Gates The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency. ”
19. Manual vs. Automated Manual Using Tools Mechanization Automation A tool is an object used to extend the ability of an individual to modify features of the surrounding environment Automation is the technology by which a process or procedure is performed with minimal human assistance
20. State of the Art Mohd-Shafie M.L., Kadir W.M.N.W, Lichter H., Khatibsyarbini M., Isa M.A. Model‑based test case generation and prioritization: a systematic literature review. 2021, Springer.
21. State of the Art M. Utting, A. Pretschner, B. Legeard, B., “A taxonomy of model-based testing”, Special Issue: Model-based Testing Volume 1: Foundations and Applications of Model-Based Testing, volume 22, issue 5. August, 2012.
22. Narrow Definition of Model Based Testing Can be both mental representation or a schema or a code-based artifact Requirement/System Specifications MBT Models Abstract Test Case Test Results SUT System Adapters Execution Environment Executable Test Case Mappings Test Selection Criteria 1 2 3 4 5 • more formal models (expressed in some machine-readable, well-defined, notation) • more formal test generation (we are interested in test generation algorithms) • more formal checks for mechanical execution (the generated tests must be sufficiently precise to efficiently use humans’ involvement)
23. A great challenge of life: Knowing enough to think you are right, but not knowing enough to know you are wrong. Neil deGrasse Tyson “
24. Humans and Complex Models
25. George E. P. Box ...all models are approximations. Essentially, all models are wrong, but some are useful. However, the approximate nature of the model must always be borne in mind… ”
26. Snake? Spear? Wall? Rope? Tree? Cut your problems down to size Simplified Models * From Bernard L. Madoff Investment Securities LLC
27. Simplified Models
28. Defense in Depth
29. Microservices Platform SUT DW Diversity
30. MBT Challenges 1. If complex models have to be completed before testing can start, this induces an unacceptable delay for the proper test executions. 2. For complex SUT, like systems of systems, test models need to abstract from a large amount of detail, because otherwise the resulting test model would become unmanageable. 3. The required skills for test engineers writing test models are significantly higher than for test engineers writing sequential test procedures.
31. Scout mindset: the motivation to see things as they are, not as you wish they were. Julia Galef ”
32. Processes Platforms People Managed Software Testing
33. 33 Build Software to Test Software Confidential exactpro.com Build Software to Test Software exactpro.com FOLLOW US ON SOCIAL MEDIA: @exactprosystems TO Thank You #cop26