High Volume Automated Testing

Iosif Itkin
CEO and co-founder, Exactpro

QA Meetup
17.06.2021

The presentation will discuss the general concept of high volume automated testing (HiVAT) – first introduced by Cem Kaner – and the examples of HiVAT techniques.

“In modern high-availability financial technology systems, there is a need for sending a large number of messages into the system under test for a continuous period of time. In order to expose defects related to message flow processing, scrupulous functional verification of outgoing messages and the system’s internal states is required. The set of methods used for it is known as HiVAT – High Volume Automated Testing. These methods are designed to recognize problems that are likely to remain undetected when other testing approaches are used, for example, those involving manual creation of automated testing scenarios and, therefore, producing statistically insufficient quantity of output data.”

https://youtu.be/nPMu3Mce_ts

To learn more about Exactpro, visit our website https://exactpro.com/

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/
Vkontakte https://vk.com/exactpro_llc

Subscribe to Exactpro YouTube channel https://www.youtube.com/c/exactprosystems

Transcript

1. 1 Build Software to Test Software
   17 JUNE 2021 | ONLINE MEETUP
   High Volume Automated Testing
   Iosif Itkin
   Exactpro, co-CEO & co-founder
   

   View Slide

2. 2 Build Software to Test Software
   Exactpro Overview
   • Exactpro specializes in development and implementation of
   high load market infrastructures
   • Headquartered in the UK with operations in the US, Georgia
   and Russia
   • The clients are regulated by the FCA, Bank of England and
   their counterparts from other countries
   • Incorporated in 2009 with 10 people, Exactpro is now
   employing over 630 specialists
   • Exactpro solutions are used in more than 20 countries all
   over the world
   • Exactpro solutions are used by more than a half of top 20
   exchange groups
   

   View Slide

3. 3 Build Software to Test Software
   Global Client Network
   

   View Slide

4. 4 Build Software to Test Software
   Operational Resilience for Mission-Critical Infrastructure
   

   View Slide

5. 5 Build Software to Test Software
   Processes, Platforms, People
   Organizational
   Environment
   Organizational
   Environment
   Organizational
   Environment
   Interpersonal
   Competency
   Methodical
   Competency
   Functional
   Competency
   Sense of Urgency
   Pro-active Work Approach
   Communication
   Teamwork
   Sense for Quality
   Adherence to Processes
   Outcome Oriented
   Learning from Mistakes
   Conflict Resolution and Mediation
   Reliability and Credibility
   Market Mode
   Trading Protocols
   RegSCI Framework
   Reference Data
   Networks and Security
   DevOps and Kubernetes
   Test Design
   Defect Tracking
   Understanding the Code
   Automation in Testing
   

   View Slide

6. 6 Build Software to Test Software
   Problem Statement
   Technology is getting more and more sophisticated and distributed,
   data volumes and complexity levels are increasing
   To survive and thrive in a changing world we need to understand
   technology around us
   Software Testing is our first, last and the only line of defence against
   unknown problems in the software
   Adequate Processes, Platforms and People are required to
   understand, build, transform and operate modern technology
   

   View Slide

7. 7 Build Software to Test Software
   High Volume Automated Testing (HiVAT)
   http://kaner.com/pdfs/highvolCSTER.pdf
   http://kaner.com/?p=278
   HVTA / HiVAT
   

   View Slide

8. 8 Build Software to Test Software
   Types of Defects
   Special Narrow Conditions
   Timing Related Errors
   Corrupt Memory, Stack and Internal States
   Resource Leakage
   Deadlocks
   https://dl.acm.org/doi/10.1145/1022494.1022536
   

   View Slide

9. 9 Build Software to Test Software
   High Volume Automated Testing (HiVAT)
   https://www.slideshare.net/Tech
   WellPresentations/tq-30593280
   

   View Slide

10. 10 Build Software to Test Software
    High Volume Automated Testing (HiVAT)
    https://youtu.be/Upx2f3g4Y1U
    Baldvin Gislason Bern
    

    View Slide

11. 11 Build Software to Test Software
    High Volume Automated Testing (HiVAT)
    https://www.slideshare.net/an
    dytinkham/high-volume-test-a
    utomation-in-practice
    

    View Slide

12. 12 Build Software to Test Software
    The Big Picture
    Software Testing is an
    ● empirical
    ● technical
    ● investigation
    ● conducted to provide stakeholders
    ● with information
    ● about the quality
    ● of the product or service under test
    Software Testing is exploring software with the intent of
    finding bugs
    

    View Slide

13. 13 Build Software to Test Software
    Scientific Method
    Conclusion
    Observe
    Question
    Experiment Hypothesis
    

    View Slide

14. 14 Build Software to Test Software
    Scientific Method
    Conclusion
    Observe
    Question
    Experiment Hypothesis
    It can't be bargained with
    It can't be reasoned with
    It can't do anything intentionally
    

    View Slide

15. 15 Build Software to Test Software
    The Best Test Tool is the Human’s Brain
    

    View Slide

16. 16 Build Software to Test Software
    Small Number of Fixed Tests - Human Capacity Limitations
    

    View Slide

17. 17 Build Software to Test Software
    Automation in Testing
    «The first rule of any technology
    used in a business is that
    automation applied to an efficient
    operation will magnify the
    efficiency. The second is that
    automation applied to an inefficient
    operation will magnify the
    inefficiency».
    Bill Gates
    Automated Testing means
    Computer-Assisted Testing
    

    View Slide

18. 18 Build Software to Test Software
    Working Definitions of HiVAT
    «The first rule of any technology
    used in a business is that
    automation applied to an efficient
    operation will magnify the
    efficiency. The second is that
    automation applied to an inefficient
    operation will magnify the
    inefficiency».
    Bill Gates
    A family of test techniques that use software to generate,
    execute and interpret arbitrarily many tests
    A family of test techniques that empower a tester to focus
    on interesting results while using software to design,
    implement, execute, and interpret a large suite of tests
    (thousands to billions of tests)
    

    View Slide

19. 19 Build Software to Test Software
    Extended Random Regression
    «The first rule of any technology
    used in a business is that
    automation applied to an efficient
    operation will magnify the
    efficiency. The second is that
    automation applied to an inefficient
    operation will magnify the
    inefficiency».
    Bill Gates
    http://www.news.hypercrit.net/2012/11/13/einstein-on-insanity/
    “The definition of insanity is doing
    something over and over again and
    expecting a different result.”
    

    View Slide

20. 20 Build Software to Test Software
    Input / Output Combination
    

    View Slide

21. 21 Build Software to Test Software
    Frequent Job Interview Question
    You are in a room with 3 switches which correspond to 3 bulbs in another
    room and you don't know which switch corresponds to which bulb. You
    can only enter the room with the bulbs once. You can not use any
    external equipment (power supplies, resistors, etc.). How do you find out
    which bulb corresponds to which switch?
    

    View Slide

22. 22 Build Software to Test Software
    Frequent Job Interview Question
    

    View Slide

23. 23 Build Software to Test Software
    Load-Enhanced Functional Testing
    «The first rule of any technology
    used in a business is that
    automation applied to an efficient
    operation will magnify the
    efficiency. The second is that
    automation applied to an inefficient
    operation will magnify the
    inefficiency».
    Bill Gates
    Functional
    Testing
    Non-Functional
    Testing
    at the Confluence of
    Functional and
    Non-Functional Testing
    

    View Slide

24. 24 Build Software to Test Software
    Parameters Variation Can Expand the Testing Scope
    

    View Slide

25. 25 Build Software to Test Software
    How Do We Know That Something is Wrong?
    https://youtu.be/OyDeLYqkD8E
    

    View Slide

26. 26 Build Software to Test Software
    Test Oracle Definition
    https://en.wikipedia.org/wiki/Test_oracle
    In computing, software engineering, and software testing, a test oracle (or just oracle) is a
    mechanism for determining whether a test has passed or failed.[1] The use of oracles
    involves comparing the output(s) of the system under test, for a given test-case input, to
    the output(s) that the oracle determines that product should have
    1. Kaner, Cem; A Course in Black Box Software Testing, 2004
    A complete oracle would have three capabilities and would carry them out perfectly:
    ● A generator, to provide predicted or expected results for each test.
    ● A comparator, to compare predicted and obtained results.
    ● An evaluator, to determine whether the comparison results are sufficiently close to be a pass.
    http://www.testingeducation.org/k04/OracleExamples.htm
    

    View Slide

27. 27 Build Software to Test Software
    All Testing is Model-Based Testing
    «The first rule of any technology
    used in a business is that
    automation applied to an efficient
    operation will magnify the
    efficiency. The second is that
    automation applied to an inefficient
    operation will magnify the
    inefficiency».
    Bill Gates
    

    View Slide

28. 28 Build Software to Test Software
    All Testing is Model Based Testing
    «The first rule of any technology
    used in a business is that
    automation applied to an efficient
    operation will magnify the
    efficiency. The second is that
    automation applied to an inefficient
    operation will magnify the
    inefficiency».
    Bill Gates
    ● Similar System
    ● Previous Version
    ● Constraints & Rules
    ● Diagnostics
    

    View Slide

29. 29 Build Software to Test Software
    Comparison Limitations and Time Variance
    https://exactpro.com/ideas/white-papers/test-automation-ccps-and-
    exchanges-operational-day-replay-limitations
    https://exactpro.com/ideas/research-papers/trading-day-logs-replay-li
    mitations-and-test-tools-applicability
    

    View Slide

30. 30 Build Software to Test Software
    Reference Implementation for HiVAT Platform
    «The first rule of any technology
    used in a business is that
    automation applied to an efficient
    operation will magnify the
    efficiency. The second is that
    automation applied to an inefficient
    operation will magnify the
    inefficiency».
    Bill Gates
    ● Arbitrarily Many Tests, Configurations and
    Permutations
    ● High Volumes and Parallel Tests Execution
    ● Ability to Apply Multiple Oracles to Every Test
    ● Macro-Level Resilience
    Who You Gonna Call?
    

    View Slide

31. 31 Build Software to Test Software
    To Be Continued...
    Thy sea is so Great and
    my boat is so small
    https://www.youtube.com/c/ExactproSystems
    

    View Slide

32. 32 Build Software to Test Software
    Thank you!
    

    View Slide