High Volume Automated Testing

Iosif Itkin
CEO and co-founder, Exactpro

QA Meetup
17.06.2021

The presentation will discuss the general concept of high volume automated testing (HiVAT) – first introduced by Cem Kaner – and the examples of HiVAT techniques.

“In modern high-availability financial technology systems, there is a need for sending a large number of messages into the system under test for a continuous period of time. In order to expose defects related to message flow processing, scrupulous functional verification of outgoing messages and the system’s internal states is required. The set of methods used for it is known as HiVAT – High Volume Automated Testing. These methods are designed to recognize problems that are likely to remain undetected when other testing approaches are used, for example, those involving manual creation of automated testing scenarios and, therefore, producing statistically insufficient quantity of output data.”

https://youtu.be/nPMu3Mce_ts

To learn more about Exactpro, visit our website https://exactpro.com/

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/
Vkontakte https://vk.com/exactpro_llc

Subscribe to Exactpro YouTube channel https://www.youtube.com/c/exactprosystems

Transcript

1. 1 Build Software to Test Software 17 JUNE 2021 | ONLINE MEETUP High Volume Automated Testing Iosif Itkin Exactpro, co-CEO & co-founder
2. 2 Build Software to Test Software Exactpro Overview • Exactpro specializes in development and implementation of high load market infrastructures • Headquartered in the UK with operations in the US, Georgia and Russia • The clients are regulated by the FCA, Bank of England and their counterparts from other countries • Incorporated in 2009 with 10 people, Exactpro is now employing over 630 specialists • Exactpro solutions are used in more than 20 countries all over the world • Exactpro solutions are used by more than a half of top 20 exchange groups
3. 3 Build Software to Test Software Global Client Network
4. 4 Build Software to Test Software Operational Resilience for Mission-Critical Infrastructure
5. 5 Build Software to Test Software Processes, Platforms, People Organizational Environment Organizational Environment Organizational Environment Interpersonal Competency Methodical Competency Functional Competency Sense of Urgency Pro-active Work Approach Communication Teamwork Sense for Quality Adherence to Processes Outcome Oriented Learning from Mistakes Conflict Resolution and Mediation Reliability and Credibility Market Mode Trading Protocols RegSCI Framework Reference Data Networks and Security DevOps and Kubernetes Test Design Defect Tracking Understanding the Code Automation in Testing
6. 6 Build Software to Test Software Problem Statement Technology is getting more and more sophisticated and distributed, data volumes and complexity levels are increasing To survive and thrive in a changing world we need to understand technology around us Software Testing is our first, last and the only line of defence against unknown problems in the software Adequate Processes, Platforms and People are required to understand, build, transform and operate modern technology
7. 7 Build Software to Test Software High Volume Automated Testing (HiVAT) http://kaner.com/pdfs/highvolCSTER.pdf http://kaner.com/?p=278 HVTA / HiVAT
8. 8 Build Software to Test Software Types of Defects Special Narrow Conditions Timing Related Errors Corrupt Memory, Stack and Internal States Resource Leakage Deadlocks https://dl.acm.org/doi/10.1145/1022494.1022536
9. 9 Build Software to Test Software High Volume Automated Testing (HiVAT) https://www.slideshare.net/Tech WellPresentations/tq-30593280
10. 10 Build Software to Test Software High Volume Automated Testing (HiVAT) https://youtu.be/Upx2f3g4Y1U Baldvin Gislason Bern
11. 11 Build Software to Test Software High Volume Automated Testing (HiVAT) https://www.slideshare.net/an dytinkham/high-volume-test-a utomation-in-practice
12. 12 Build Software to Test Software The Big Picture Software Testing is an • empirical • technical • investigation • conducted to provide stakeholders • with information • about the quality • of the product or service under test Software Testing is exploring software with the intent of finding bugs
13. 13 Build Software to Test Software Scientific Method Conclusion Observe Question Experiment Hypothesis
14. 14 Build Software to Test Software Scientific Method Conclusion Observe Question Experiment Hypothesis It can't be bargained with It can't be reasoned with It can't do anything intentionally
15. 15 Build Software to Test Software The Best Test Tool is the Human’s Brain
16. 16 Build Software to Test Software Small Number of Fixed Tests - Human Capacity Limitations
17. 17 Build Software to Test Software Automation in Testing «The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency». Bill Gates Automated Testing means Computer-Assisted Testing
18. 18 Build Software to Test Software Working Definitions of HiVAT «The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency». Bill Gates A family of test techniques that use software to generate, execute and interpret arbitrarily many tests A family of test techniques that empower a tester to focus on interesting results while using software to design, implement, execute, and interpret a large suite of tests (thousands to billions of tests)
19. 19 Build Software to Test Software Extended Random Regression «The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency». Bill Gates http://www.news.hypercrit.net/2012/11/13/einstein-on-insanity/ “The definition of insanity is doing something over and over again and expecting a different result.”
20. 20 Build Software to Test Software Input / Output Combination
21. 21 Build Software to Test Software Frequent Job Interview Question You are in a room with 3 switches which correspond to 3 bulbs in another room and you don't know which switch corresponds to which bulb. You can only enter the room with the bulbs once. You can not use any external equipment (power supplies, resistors, etc.). How do you find out which bulb corresponds to which switch?
22. 22 Build Software to Test Software Frequent Job Interview Question
23. 23 Build Software to Test Software Load-Enhanced Functional Testing «The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency». Bill Gates Functional Testing Non-Functional Testing at the Confluence of Functional and Non-Functional Testing
24. 24 Build Software to Test Software Parameters Variation Can Expand the Testing Scope
25. 25 Build Software to Test Software How Do We Know That Something is Wrong? https://youtu.be/OyDeLYqkD8E
26. 26 Build Software to Test Software Test Oracle Definition https://en.wikipedia.org/wiki/Test_oracle In computing, software engineering, and software testing, a test oracle (or just oracle) is a mechanism for determining whether a test has passed or failed.[1] The use of oracles involves comparing the output(s) of the system under test, for a given test-case input, to the output(s) that the oracle determines that product should have 1. Kaner, Cem; A Course in Black Box Software Testing, 2004 A complete oracle would have three capabilities and would carry them out perfectly: • A generator, to provide predicted or expected results for each test. • A comparator, to compare predicted and obtained results. • An evaluator, to determine whether the comparison results are sufficiently close to be a pass. http://www.testingeducation.org/k04/OracleExamples.htm
27. 27 Build Software to Test Software All Testing is Model-Based Testing «The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency». Bill Gates
28. 28 Build Software to Test Software All Testing is Model Based Testing «The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency». Bill Gates • Similar System • Previous Version • Constraints & Rules • Diagnostics
29. 29 Build Software to Test Software Comparison Limitations and Time Variance https://exactpro.com/ideas/white-papers/test-automation-ccps-and- exchanges-operational-day-replay-limitations https://exactpro.com/ideas/research-papers/trading-day-logs-replay-li mitations-and-test-tools-applicability
30. 30 Build Software to Test Software Reference Implementation for HiVAT Platform «The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency». Bill Gates • Arbitrarily Many Tests, Configurations and Permutations • High Volumes and Parallel Tests Execution • Ability to Apply Multiple Oracles to Every Test • Macro-Level Resilience Who You Gonna Call?
31. 31 Build Software to Test Software To Be Continued... Thy sea is so Great and my boat is so small https://www.youtube.com/c/ExactproSystems
32. 32 Build Software to Test Software Thank you!