TMPA-2021: Model-based Testing Approach for Financial Technology Platforms: An Industrial Implementation

Luba Konnova, Ivan Scherbinin, Vyacheslav Okhlopkov, Levan Gharibashvili, Mariam Mtsariashvili and Tiniko Babalashvili, Exactpro

Model-based Testing Approach for Financial Technology Platforms: An Industrial Implementation

TMPA is an annual International Conference on Software Testing, Machine Learning and Complex Process Analysis. The conference will focus on the application of modern methods of data science to the analysis of software quality.

To learn more about Exactpro, visit our website https://exactpro.com/

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro

Transcript

1. 1 25-27 NOVEMBER SOFTWARE TESTING, MACHINE LEARNING AND COMPLEX PROCESS ANALYSIS Model-Based Testing Approach for Financial Technology Platforms: An Industrial Implementation Liubov Konnova, Ivan Scherbinin, Vyacheslav Okhlopkov, Levan Gharibashvili, Mariam Mtsariashvili, and Tiniko Babalashvili, Exactpro
2. 2 25-27 NOVEMBER, TOMSK, RUSSIA SOFTWARE TESTING, MACHINE LEARNING AND COMPLEX PROCESS ANALYSIS Introduction
3. 3 Model-Based Testing (MBT) The process consisting of all lifecycle activities, both static and dynamic, concerned with planning, preparation and evaluation of a component or system and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects. Testing based on or involving models.
4. 4 What is a model?
5. 5 Research on Model-Based testing
6. 6 History of research: milestones • Chow TS. Testing software design modeled by finite-state machines. IEEE Transactions on Software Engineering 1978; 4(3):178–187. • Dias Neto AC., Subramanyan R., Vieira M., Travassos G.H..: A survey on model-based testing approaches: a systematic review. WEASELTech’07: Proceedings of the 1st ACM international workshop on Empirical assessment of software engineering languages and technologies: held in conjunction with the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE) 2007, 31–36 (2007). • Utting M., Legeard B., Bouquet F., Fourneret E., Peureux F., Vernotte A. Chapter two - Recent Advances in Model-Based Testing. Advances in Computers, Volume 101, 2016, Pages 53-120 • Mohd-Shafie M.L., Kadir W.M.N.W, Lichter H., Khatibsyarbini M., Isa M.A. Model-based test case generation and prioritization: a systematic literature review. 2021, Springer. 1978 2007 2016 2021
7. 7 Connections, readers and codecs Scripts, tools, ( JUnit, cucumber etc) th2 scenarios, agents, simulators, monitoring, models API Test management and analytics dashboards Big data/ Machine Learning tools Zephyr Grafana, Jupiter etc Data API Models Data Services Test Data Systems Under Test Objective of the paper
8. 8 th2 - next generation test framework th2 benefits 1. The capacity to execute more functional tests under load to improve test coverage, system quality and resilience 2. Faster processing of output from massive automated test execution to decrease time to market 3. Technology stack with an open interface to facilitate the adoption of digital technology (Cloud, DLT) 4. Unified storage of test results in the Cassandra database to enable better access to test evidence and smart analytics for governance and regulatory compliance https://exactpro.com/test-tools/th2
9. 9 Presentation structure • Characteristics of the MBT approach used at Exactpro: • Directions of future MBT developments in Exactpro • Assessment of the industrial experience with MBT outlined through analysis of survey responses.
10. 10 25-27 NOVEMBER, TOMSK, RUSSIA SOFTWARE TESTING, MACHINE LEARNING AND COMPLEX PROCESS ANALYSIS Taxonomy-based Classification
11. 11 MBT Taxonomy
12. 12 Taxonomy-based Classification Scope Input-Only Input-Output Can generate input scenarios ✔ ✔ Can predict expected results ✔
13. 13 Taxonomy-based Classification Characteristics - Untimed, Deterministic, Discrete Timed / Untimed • Timed models are required for real-time systems, where it is necessary to send input at a particular time or to check that response is received within a particular time frame. Deterministic / Non-deterministic • Non-deterministic systems involve functionality that cannot be predicted in advance. Discrete / Hybrid / Continuous • In discrete models, changes happen only at discrete events. • In continuous and hybrid models, state can continuously change as time passes.
14. 14 Taxonomy-based Classification Paradigm - None Our model doesn't use a formal notation for describing exchange logic. Instead, users come up with scenarios and use model as an oracle. • Pre-Post or Input Domains • Transition-Based • History Based • Functional • Operational • Stochastic • Data Flow
15. 15 Taxonomy-based Classification Test Selection Criteria - Test Case Specifications / Data Coverage • Testers write test case specifications that describe steps that should be performed against the system. • Combinatorial algorithms are used for coming up with input data that will cover intended order configurations.
16. 16 Taxonomy-based Classification Technology - Constraint Solving / Random Generation / Symbolic Execution • Pairwise testing tool with a constraint solver is used for coming up with valid input data, while covering wide range of parameter values. • Random generation is used for parameters that should not affect the test result. • Symbolic execution is used to come up with input data based on some logic.
17. 17 Taxonomy-based Classification Test Execution - Online Offline • Tests are generated strictly before they are run. Online • Tests can be generated on the fly and model can react to actual results from the system.
18. 18 Model Specification Test Generation Test Execution Scope Characteristics Paradigm Test Selection Criteria Technology On/Offline Input- only / Input-Output Untimed / Timed Deterministic / Non-Det. Discrete / Hybrid / Continuous Pre-Post or Input Domains Transition-Based History Based Functional Operational Stochastic Data Flow Structural Model Coverage Data Coverage Requirements Coverage Test Case Specifications Random & Stochastic Fault-Based Online Offline Random generation Search-Based algorithms Mode-checking Symbolic execution Theorem proving Constraint Solving
19. 19 25-27 NOVEMBER, TOMSK, RUSSIA SOFTWARE TESTING, MACHINE LEARNING AND COMPLEX PROCESS ANALYSIS Classification Approach by Dias Neto A.C. et al.
20. 20 25-27 NOVEMBER, TOMSK, RUSSIA SOFTWARE TESTING, MACHINE LEARNING AND COMPLEX PROCESS ANALYSIS Software Domain Trading Platforms - complex, distributed infrastructures
21. 21 25-27 NOVEMBER, TOMSK, RUSSIA SOFTWARE TESTING, MACHINE LEARNING AND COMPLEX PROCESS ANALYSIS Software Domain Test Level Trading Platforms - complex, distributed infrastructures Functional and regression system testing
22. 22 25-27 NOVEMBER, TOMSK, RUSSIA SOFTWARE TESTING, MACHINE LEARNING AND COMPLEX PROCESS ANALYSIS Software Domain Test Level Automation Level Trading Platforms - complex, distributed infrastructures Functional and regression system testing Generally, fully automated, except for steps like reference data upload
23. 23 25-27 NOVEMBER, TOMSK, RUSSIA SOFTWARE TESTING, MACHINE LEARNING AND COMPLEX PROCESS ANALYSIS Software Domain Test Level Automation Level Supporting Tools Trading Platforms - complex, distributed infrastructures Functional and regression system testing Generally, fully automated, except for steps like reference data upload Exactpro does not use any external tools, except for its own testing tool th2
24. 24 25-27 NOVEMBER, TOMSK, RUSSIA SOFTWARE TESTING, MACHINE LEARNING AND COMPLEX PROCESS ANALYSIS Testing Coverage Criteria It is still a question how to find the coverage of such a complex system
25. 25 25-27 NOVEMBER, TOMSK, RUSSIA SOFTWARE TESTING, MACHINE LEARNING AND COMPLEX PROCESS ANALYSIS Testing Coverage Criteria Behavior Model It is still a question how to find the coverage of such a complex system Model is continuously checked at the development stage to make sure it meets the specifications
26. 26 25-27 NOVEMBER, TOMSK, RUSSIA SOFTWARE TESTING, MACHINE LEARNING AND COMPLEX PROCESS ANALYSIS Testing Coverage Criteria Behavior Model Cost and Complexity It is still a question how to find the coverage of such a complex system Model is continuously checked at the development stage to make sure it meets the specifications At the stage of functional testing, it is resource consuming, but at regression stage it brings benefits
27. 27 25-27 NOVEMBER, TOMSK, RUSSIA SOFTWARE TESTING, MACHINE LEARNING AND COMPLEX PROCESS ANALYSIS Future Applications of the Model-Based Test Automation
28. 28 28 Current Model Active model
29. 29 29 Future Plans: Passive model
30. 30 30 Future Plans: • External Exchange Stub • Interactive Model • Generic Model Model automatically provides external exchange responses Exchange 1 Exchange 2 Reusable Part- Generic Model
31. 31 25-27 NOVEMBER, TOMSK, RUSSIA SOFTWARE TESTING, MACHINE LEARNING AND COMPLEX PROCESS ANALYSIS Models Related Challenges and Solutions
32. 32 MBT Challenges 1. If complex models have to be completed before testing can start, this induces an unacceptable delay for the proper test executions. 2. For complex SUT, like systems of systems, test models need to abstract from a large amount of detail, because otherwise the resulting test model would become unmanageable. 3. The required skills for test engineers writing test models are significantly higher than for test engineers writing sequential test procedures.
33. 33 Processes and people
34. 34 Conclusion Systems Under Test SUT testing Models
35. 35 Thank You! Follow TMPA on Facebook TMPA-2021 Conference