Upgrade to Pro — share decks privately, control downloads, hide ads and more …

TMPA-2021: AUTOMATION IN SOFTWARE TESTING. HUMANS AND COMPLEX MODELS.

TMPA-2021: AUTOMATION IN SOFTWARE TESTING. HUMANS AND COMPLEX MODELS.

TMPA is an annual International Conference on Software Testing, Machine Learning and Complex Process Analysis. The conference will focus on the application of modern methods of data science to the analysis of software quality.

To learn more about Exactpro, visit our website https://exactpro.com/

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro

Exactpro
PRO

November 26, 2021
Tweet

More Decks by Exactpro

Other Decks in Technology

Transcript

  1. 1
    25-27 NOVEMBER
    SOFTWARE TESTING, MACHINE LEARNING
    AND COMPLEX PROCESS ANALYSIS
    Automation in Software Testing.
    Humans and Complex Models.
    Iosif Itkin
    CEO and co-founder, Exactpro

    View Slide

  2. 2
    About Exactpro

    View Slide

  3. 3
    3

    View Slide

  4. 4
    4
    MANAGED SUPPORT
    SERVICES PROVIDER OF
    THE YEAR
    In this Risk.net interview, Alexey Zverev, Exactpro co-CEO and
    co-founder, talks about mitigating risks while driving innovation during
    the time of high volatility and explains what made Exactpro stand out
    among the contestants. Watch the interview with Risk.net via the QR
    code above.

    View Slide

  5. 5
    ● https://www.linkedin.com/in/iosifitkin/
    ● Exactpro representative at the Zero Outage Industry Standard association
    ● Head of LOST (Laboratory of Software Testing) at Tomsk Polytechnic University
    ● Co-chair of FIX Trading Community MOST Working Group (Monitoring, Onboarding and Software Testing)
    ● Co-founder of TMPA (Tools & Methods of Program Analysis)
    ● Software Testing Machine Learning and Complex Process Analysis
    About me

    View Slide

  6. 6
    Zero Outage Industry Standard (ZOIS)
    • Platforms
    • People
    • Processes
    • Security

    View Slide

  7. 7
    Monitoring, Onboarding and Software Testing
    Working Group within FIX Trading Community
    Exactpro Partners
    For over a year, the Monitoring, Onboarding and Software Testing (MOST)
    working group within the FIX Trading Community has been meeting regularly to
    grow a community focused on standardization and innovation in technology
    processes leveraging the FIX protocol. The group’s mission is to raise the quality
    and reliability of platforms operated by the FIX Trading Community Members.
    This is being implemented via developing a set of recommended practices for
    monitoring, client onboarding and software testing of FIX-related financial
    applications as well as creating technical guidelines for the assessment of their
    quality and reliability. The group has presented a detailed mind map of such a
    set of best practices and guidelines, and continues working on the content of the
    document. The MOST members meet on a monthly basis and also collaborate
    with the FIX Orchestra and the Cybersecurity working groups to ensure
    alignment across a number of related topics.

    View Slide

  8. 8
    Software Testing for Complex
    Intelligent Systems and
    Autonomous Vehicles
    Online Course
    Software testing for complex intelligent systems and autonomous vehicles
    - Playlist on the Exactpro YouTube channel
    Laboratory of Software Testing

    View Slide

  9. 9
    Laboratory of Software Testing

    View Slide

  10. 10

    View Slide

  11. 11
    Laboratory of Software Testing - AI Testing

    View Slide

  12. 12
    Laboratory of Software Testing - AI Testing

    View Slide

  13. 13
    Laboratory of Software Testing - AI Testing

    View Slide

  14. 14
    What is Software Testing

    View Slide

  15. 15
    Art, Craft, Science, Engineering
    Where does software testing belong on this continuum?

    View Slide

  16. 16
    https://fs.blog/2013/07/the-difference-between-science-and-engineering/
    Abstract Model
    = theory
    Concrete
    Description
    = data
    Physical System
    = object of study
    flow of information
    measure
    compare
    Abstract Model
    = design concept
    Concrete
    Description
    = specification
    Physical System
    = useful product
    flow of information
    design
    produce
    Scientific Inquiry Engineering Design
    The Antiparallel Structures of Scientific Inquiry and Engineering Design
    Science and Engineering

    View Slide

  17. 17
    Software Testing is exploring software with the intent of finding bugs.
    Software Testing is an empirical technical investigation conducted to provide
    stakeholders with information about the quality of the product or service under test.
    Software Testing

    View Slide

  18. 18
    Abstract Model
    = theory
    Concrete Description
    = data
    Software
    = object of study
    flow of information
    observe
    compare
    Abstract Model
    = design concept
    Concrete Description
    = specification
    Software
    = useful product
    flow of information
    design
    produce
    Software Testing Development
    The Antiparallel Structures of Software Testing and Development
    Software Testing

    View Slide

  19. 19
    Never invest in
    a business you cannot
    understand

    Warren Buffett

    View Slide

  20. 20
    Models
    Abstract Model
    = theory
    Concrete Description
    = data
    Software
    = object of study
    flow of information
    observe
    compare
    Abstract Model
    = design concept
    Concrete Description
    = specification
    Software
    = useful product
    flow of information
    design
    produce
    Software Testing Development
    The Antiparallel Structures of Software Testing and Development
    Understanding Your Technology Assets

    View Slide

  21. 21
    A model is a simplified, formal
    representation of a relationship,
    process or system. The simplification
    makes some aspects of the thing
    modeled clearer, more visible, and
    easier to work with.
    All tests are based on models, but
    many of those models are implicit.
    When the behavior of the program
    “feels wrong”, it is clashing with your
    internal model of the program and
    how it should behave.
    Black Box Software Testing
    Cem Kaner & James Bach
    All Testing is Model Based Testing

    View Slide

  22. 22
    BA DEV QA
    Given Everything is Broken
    When Time is Up
    Then Build Back Better
    Behavior Driven Development

    View Slide

  23. 23
    The best material
    model of a cat is
    another, or preferably
    the same, cat.
    Arturo Rosenblueth

    View Slide

  24. 24
    Parallel Run / Operational Day Replay

    View Slide

  25. 25
    Manual Using Tools Mechanization Automation
    A tool is an object used to extend the ability of an individual to modify features of the surrounding environment
    Automation is the technology by which a process or procedure is performed with minimal human assistance
    Manual vs. Automated

    View Slide

  26. 26
    Bill Gates
    The first rule of any technology used in a
    business is that automation applied to an
    efficient operation will magnify the efficiency.
    The second is that automation applied to an
    inefficient operation will magnify the
    inefficiency.

    View Slide

  27. 27
    Processes Platforms People
    Managed Software Testing

    View Slide

  28. 28
    Mohd-Shafie M.L., Kadir W.M.N.W, Lichter H., Khatibsyarbini M., Isa M.A. Model‑based test
    case generation and prioritization: a systematic literature review. 2021, Springer.
    State of the Art

    View Slide

  29. 29
    Can be both mental representation or
    a schema or a code-based artifact
    Requirement/System
    Specifications
    MBT Models
    Abstract Test Case
    Test Results SUT
    System Adapters
    Execution Environment
    Executable
    Test Case
    Mappings
    Test Selection
    Criteria
    1 2
    3
    4
    5
    ● more formal models (expressed in some
    machine-readable, well-defined, notation)
    ● more formal test generation (we are interested in test
    generation algorithms)
    ● more formal checks for mechanical execution (the
    generated tests must be sufficiently precise to
    efficiently use humans’ involvement)
    Narrow Definition of Model Based Testing

    View Slide

  30. 30
    A great challenge of life:
    Knowing enough to think you are
    right,
    but not knowing enough to
    know you are wrong.
    Neil deGrasse Tyson

    View Slide

  31. 31
    Humans and Complex Models

    View Slide

  32. 32
    George E. P. Box
    ...all models are
    approximations. Essentially,
    all models are wrong, but
    some are useful. However,
    the approximate nature of
    the model must always be
    borne in mind…

    View Slide

  33. 33
    Humans and Complex Models

    View Slide

  34. 34
    Aleatoric vs. Epistemic
    Works
    90% of the time

    View Slide

  35. 35
    Simplified Models
    .01 .09
    .09 .81
    weapons fail independently
    fails works
    weapon 1
    weapon 2
    fails
    works
    .10 0
    0 .90
    weapons fail from a common cause
    fails works
    weapon 1
    weapon 2
    fails
    works

    View Slide

  36. 36
    Snake?
    Spear? Wall?
    Rope?
    Tree?
    Cut your problems
    down to size
    * From Bernard L. Madoff Investment Securities LLC
    Simplified Models

    View Slide

  37. 37
    Defense in Depth

    View Slide

  38. 38
    Defense in Depth

    View Slide

  39. 39
    Simplified Models

    View Slide

  40. 40
    Scout mindset:
    the motivation to see
    things as they are, not
    as you wish they were.
    Julia Galef

    View Slide

  41. 41
    Microservices Platform
    SUT
    DW
    Diversity

    View Slide

  42. 42
    Where does software testing belong on this continuum?
    Art, Craft, Science, Engineering, Journalism, Magic

    View Slide

  43. 43
    The only way of discovering the
    limits of the possible is to
    venture a little way past them
    into the impossible.
    Any sufficiently advanced
    technology is indistinguishable
    from magic.
    Arthur Clarke

    View Slide

  44. 44
    Too many journalists see their work
    as an opportunity to promote their
    own pet political notions, rather
    than a responsibility to inform the
    public and let their readers and
    viewers decide for themselves.
    Thomas Sowell

    View Slide

  45. 45
    45
    FOLLOW US ON SOCIAL MEDIA:
    @exactprosystems
    TO
    #th2net

    View Slide

  46. 46
    Thank You
    Follow TMPA on Facebook
    TMPA-2021 Conference

    View Slide