Upgrade to Pro — share decks privately, control downloads, hide ads and more …

High Volume Automated Testing

High Volume Automated Testing

Iosif Itkin
CEO and co-founder, Exactpro

QA Meetup
17.06.2021

The presentation will discuss the general concept of high volume automated testing (HiVAT) – first introduced by Cem Kaner – and the examples of HiVAT techniques.

“In modern high-availability financial technology systems, there is a need for sending a large number of messages into the system under test for a continuous period of time. In order to expose defects related to message flow processing, scrupulous functional verification of outgoing messages and the system’s internal states is required. The set of methods used for it is known as HiVAT – High Volume Automated Testing. These methods are designed to recognize problems that are likely to remain undetected when other testing approaches are used, for example, those involving manual creation of automated testing scenarios and, therefore, producing statistically insufficient quantity of output data.”

https://youtu.be/nPMu3Mce_ts

To learn more about Exactpro, visit our website https://exactpro.com/

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/
Vkontakte https://vk.com/exactpro_llc

Subscribe to Exactpro YouTube channel https://www.youtube.com/c/exactprosystems

5206c19df417b8876825b5561344c1a0?s=128

Exactpro
PRO

June 17, 2021
Tweet

Transcript

  1. 1 Build Software to Test Software 17 JUNE 2021 |

    ONLINE MEETUP High Volume Automated Testing Iosif Itkin Exactpro, co-CEO & co-founder
  2. 2 Build Software to Test Software Exactpro Overview • Exactpro

    specializes in development and implementation of high load market infrastructures • Headquartered in the UK with operations in the US, Georgia and Russia • The clients are regulated by the FCA, Bank of England and their counterparts from other countries • Incorporated in 2009 with 10 people, Exactpro is now employing over 630 specialists • Exactpro solutions are used in more than 20 countries all over the world • Exactpro solutions are used by more than a half of top 20 exchange groups
  3. 3 Build Software to Test Software Global Client Network

  4. 4 Build Software to Test Software Operational Resilience for Mission-Critical

    Infrastructure
  5. 5 Build Software to Test Software Processes, Platforms, People Organizational

    Environment Organizational Environment Organizational Environment Interpersonal Competency Methodical Competency Functional Competency Sense of Urgency Pro-active Work Approach Communication Teamwork Sense for Quality Adherence to Processes Outcome Oriented Learning from Mistakes Conflict Resolution and Mediation Reliability and Credibility Market Mode Trading Protocols RegSCI Framework Reference Data Networks and Security DevOps and Kubernetes Test Design Defect Tracking Understanding the Code Automation in Testing
  6. 6 Build Software to Test Software Problem Statement Technology is

    getting more and more sophisticated and distributed, data volumes and complexity levels are increasing To survive and thrive in a changing world we need to understand technology around us Software Testing is our first, last and the only line of defence against unknown problems in the software Adequate Processes, Platforms and People are required to understand, build, transform and operate modern technology
  7. 7 Build Software to Test Software High Volume Automated Testing

    (HiVAT) http://kaner.com/pdfs/highvolCSTER.pdf http://kaner.com/?p=278 HVTA / HiVAT
  8. 8 Build Software to Test Software Types of Defects Special

    Narrow Conditions Timing Related Errors Corrupt Memory, Stack and Internal States Resource Leakage Deadlocks https://dl.acm.org/doi/10.1145/1022494.1022536
  9. 9 Build Software to Test Software High Volume Automated Testing

    (HiVAT) https://www.slideshare.net/Tech WellPresentations/tq-30593280
  10. 10 Build Software to Test Software High Volume Automated Testing

    (HiVAT) https://youtu.be/Upx2f3g4Y1U Baldvin Gislason Bern
  11. 11 Build Software to Test Software High Volume Automated Testing

    (HiVAT) https://www.slideshare.net/an dytinkham/high-volume-test-a utomation-in-practice
  12. 12 Build Software to Test Software The Big Picture Software

    Testing is an • empirical • technical • investigation • conducted to provide stakeholders • with information • about the quality • of the product or service under test Software Testing is exploring software with the intent of finding bugs
  13. 13 Build Software to Test Software Scientific Method Conclusion Observe

    Question Experiment Hypothesis
  14. 14 Build Software to Test Software Scientific Method Conclusion Observe

    Question Experiment Hypothesis It can't be bargained with It can't be reasoned with It can't do anything intentionally
  15. 15 Build Software to Test Software The Best Test Tool

    is the Human’s Brain
  16. 16 Build Software to Test Software Small Number of Fixed

    Tests - Human Capacity Limitations
  17. 17 Build Software to Test Software Automation in Testing «The

    first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency». Bill Gates Automated Testing means Computer-Assisted Testing
  18. 18 Build Software to Test Software Working Definitions of HiVAT

    «The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency». Bill Gates A family of test techniques that use software to generate, execute and interpret arbitrarily many tests A family of test techniques that empower a tester to focus on interesting results while using software to design, implement, execute, and interpret a large suite of tests (thousands to billions of tests)
  19. 19 Build Software to Test Software Extended Random Regression «The

    first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency». Bill Gates http://www.news.hypercrit.net/2012/11/13/einstein-on-insanity/ “The definition of insanity is doing something over and over again and expecting a different result.”
  20. 20 Build Software to Test Software Input / Output Combination

  21. 21 Build Software to Test Software Frequent Job Interview Question

    You are in a room with 3 switches which correspond to 3 bulbs in another room and you don't know which switch corresponds to which bulb. You can only enter the room with the bulbs once. You can not use any external equipment (power supplies, resistors, etc.). How do you find out which bulb corresponds to which switch?
  22. 22 Build Software to Test Software Frequent Job Interview Question

  23. 23 Build Software to Test Software Load-Enhanced Functional Testing «The

    first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency». Bill Gates Functional Testing Non-Functional Testing at the Confluence of Functional and Non-Functional Testing
  24. 24 Build Software to Test Software Parameters Variation Can Expand

    the Testing Scope
  25. 25 Build Software to Test Software How Do We Know

    That Something is Wrong? https://youtu.be/OyDeLYqkD8E
  26. 26 Build Software to Test Software Test Oracle Definition https://en.wikipedia.org/wiki/Test_oracle

    In computing, software engineering, and software testing, a test oracle (or just oracle) is a mechanism for determining whether a test has passed or failed.[1] The use of oracles involves comparing the output(s) of the system under test, for a given test-case input, to the output(s) that the oracle determines that product should have 1. Kaner, Cem; A Course in Black Box Software Testing, 2004 A complete oracle would have three capabilities and would carry them out perfectly: • A generator, to provide predicted or expected results for each test. • A comparator, to compare predicted and obtained results. • An evaluator, to determine whether the comparison results are sufficiently close to be a pass. http://www.testingeducation.org/k04/OracleExamples.htm
  27. 27 Build Software to Test Software All Testing is Model-Based

    Testing «The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency». Bill Gates
  28. 28 Build Software to Test Software All Testing is Model

    Based Testing «The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency». Bill Gates • Similar System • Previous Version • Constraints & Rules • Diagnostics
  29. 29 Build Software to Test Software Comparison Limitations and Time

    Variance https://exactpro.com/ideas/white-papers/test-automation-ccps-and- exchanges-operational-day-replay-limitations https://exactpro.com/ideas/research-papers/trading-day-logs-replay-li mitations-and-test-tools-applicability
  30. 30 Build Software to Test Software Reference Implementation for HiVAT

    Platform «The first rule of any technology used in a business is that automation applied to an efficient operation will magnify the efficiency. The second is that automation applied to an inefficient operation will magnify the inefficiency». Bill Gates • Arbitrarily Many Tests, Configurations and Permutations • High Volumes and Parallel Tests Execution • Ability to Apply Multiple Oracles to Every Test • Macro-Level Resilience Who You Gonna Call?
  31. 31 Build Software to Test Software To Be Continued... Thy

    sea is so Great and my boat is so small https://www.youtube.com/c/ExactproSystems
  32. 32 Build Software to Test Software Thank you!