Automation in Software Testing. Humans and Complex Models.

Iosif Itkin, CEO and co-founder, Exactpro

The QA Financial Forum: London 2021
9 November 2021

To learn more about Exactpro, visit our website https://exactpro.com/

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro

Transcript

1. Automation in Software Testing.
   Humans and Complex Models.
   Iosif Itkin
   CEO and co-founder, Exactpro
   

   View Slide

2. About Exactpro
   

   View Slide

3. View Slide

4. MANAGED SUPPORT
   SERVICES PROVIDER OF THE
   YEAR
   In this Risk.net interview, Alexey Zverev, Exactpro co-CEO and co-founder,
   talks about mitigating risks while driving innovation during the time of
   high volatility and explains what made Exactpro stand out among the
   contestants. Watch the interview with Risk.net via the QR code above.
   

   View Slide

5. ● https://www.linkedin.com/in/iosifitkin/
   ● Exactpro representative at the Zero Outage Industry Standard association
   ● Head of LOST (Laboratory of Software Testing) at Tomsk Polytechnic University
   ● Co-chair of FIX Trading Community MOST Working Group (Monitoring, Onboarding and Software Testing)
   About me
   

   View Slide

6. What is Software Testing
   

   View Slide

7. Art, Craft, Science, Engineering
   Where does software testing belongs on this continuum?
   

   View Slide

8. Science and Engineering
   https://fs.blog/2013/07/the-difference-between-science-and-engineering/
   Abstract Model
   = theory
   Concrete
   Description
   = data
   Physical System
   = object of study
   flow of information
   measure
   compare
   Abstract Model
   = design concept
   Concrete
   Description
   = specification
   Physical System
   = useful product
   flow of information
   design
   produce
   Scientific Inquiry Engineering Design
   The Antiparallel Structures of Scientific Inquiry and Engineering Design
   

   View Slide

9. Software Testing
   Software Testing is exploring software with the intent of finding bugs.
   Software Testing is an empirical technical investigation conducted to provide
   stakeholders with information about the quality of the product or service under test.
   

   View Slide

10. Software Testing
    Abstract Model
    = theory
    Concrete Description
    = data
    Software
    = object of study
    flow of information
    observe
    compare
    Abstract Model
    = design concept
    Concrete Description
    = specification
    Software
    = useful product
    flow of information
    design
    produce
    Software Testing Development
    The Antiparallel Structures of Software Testing and Development
    

    View Slide

11. Never invest in
    a business you cannot
    understand
    “
    Warren Buffett
    

    View Slide

12. Understanding Your Technology Assets
    Models
    Abstract Model
    = theory
    Concrete Description
    = data
    Software
    = object of study
    flow of information
    observe
    compare
    Abstract Model
    = design concept
    Concrete Description
    = specification
    Software
    = useful product
    flow of information
    design
    produce
    Software Testing Development
    The Antiparallel Structures of Software Testing and Development
    

    View Slide

13. All Testing is Model Based Testing
    A model is a simplified, formal
    representation of a relationship,
    process or system. The simplification
    makes some aspects of the thing
    modeled clearer, more visible, and
    easier to work with.
    All tests are based on models, but
    many of those models are implicit.
    When the behavior of the program
    “feels wrong”, it is clashing with your
    internal model of the program and
    how it should behave.
    Black Box Software Testing
    Cem Kaner & James Bach
    

    View Slide

14. Properties of a Model
    Herbert Stachowiak. Allgemeine Modelltheorie. Springer-Verlag, Wien and New York, 1973. (Cited by Kühne, Thomas.
    "What is a Model?." In Dagstuhl Seminar Proceedings. Schloss Dagstuhl-Leibniz-Zentrum für Informatik, 2005)
    mapping feature reduction feature pragmatic feature
    

    View Slide

15. BA DEV QA
    Behavior Driven Development
    Given Everything is Broken
    When Time is Up
    Then Build Back Better
    

    View Slide

16. The best material
    model of a cat is
    another, or preferably
    the same, cat.
    Arturo Rosenblueth
    ”
    

    View Slide

17. Parallel Run / Operational Day Replay
    

    View Slide

18. Bill Gates
    The first rule of any technology used in a
    business is that automation applied to an
    efficient operation will magnify the efficiency.
    The second is that automation applied to an
    inefficient operation will magnify the
    inefficiency.
    ”
    

    View Slide

19. Manual vs. Automated
    Manual Using Tools Mechanization Automation
    A tool is an object used to extend the ability of an individual to modify features of the surrounding environment
    Automation is the technology by which a process or procedure is performed with minimal human assistance
    

    View Slide

20. State of the Art
    Mohd-Shafie M.L., Kadir W.M.N.W, Lichter H., Khatibsyarbini M., Isa M.A. Model‑based test
    case generation and prioritization: a systematic literature review. 2021, Springer.
    

    View Slide

21. State of the Art
    M. Utting, A. Pretschner, B. Legeard, B., “A
    taxonomy of model-based testing”,
    Special Issue: Model-based Testing Volume 1:
    Foundations and Applications of
    Model-Based Testing, volume 22, issue 5.
    August, 2012.
    

    View Slide

22. Narrow Definition of Model Based Testing
    Can be both mental representation or
    a schema or a code-based artifact
    Requirement/System
    Specifications
    MBT Models
    Abstract Test Case
    Test Results SUT
    System Adapters
    Execution Environment
    Executable
    Test Case
    Mappings
    Test Selection
    Criteria
    1 2
    3
    4
    5
    ● more formal models (expressed in some
    machine-readable, well-defined, notation)
    ● more formal test generation (we are interested in test
    generation algorithms)
    ● more formal checks for mechanical execution (the
    generated tests must be sufficiently precise to
    efficiently use humans’ involvement)
    

    View Slide

23. A great challenge of life:
    Knowing enough to think you are
    right,
    but not knowing enough to
    know you are wrong.
    Neil deGrasse Tyson
    “
    

    View Slide

24. Humans and Complex Models
    

    View Slide

25. George E. P. Box
    ...all models are
    approximations. Essentially,
    all models are wrong, but
    some are useful. However,
    the approximate nature of
    the model must always be
    borne in mind…
    ”
    

    View Slide

26. Snake?
    Spear? Wall?
    Rope?
    Tree?
    Cut your problems
    down to size
    Simplified Models
    * From Bernard L. Madoff Investment Securities LLC
    

    View Slide

27. Simplified Models
    

    View Slide

28. Defense in Depth
    

    View Slide

29. Microservices Platform
    SUT
    DW
    Diversity
    

    View Slide

30. MBT Challenges
    1. If complex models have to be completed before testing can start, this induces an unacceptable
    delay for the proper test executions.
    2. For complex SUT, like systems of systems, test models need to abstract from a large amount of
    detail, because otherwise the resulting test model would become unmanageable.
    3. The required skills for test engineers writing test models are significantly higher than for test
    engineers writing sequential test procedures.
    

    View Slide

31. Scout mindset:
    the motivation to see
    things as they are, not
    as you wish they were.
    Julia Galef
    ”
    

    View Slide

32. Processes Platforms People
    Managed Software Testing
    

    View Slide

33. 33 Build Software to Test Software
    Confidential
    exactpro.com
    Build Software to Test Software
    exactpro.com
    FOLLOW US ON SOCIAL MEDIA:
    @exactprosystems
    TO
    Thank You
    #cop26
    

    View Slide