Model-Based Testing in Mission-Critical Financial Market Infrastructures Elena Treshcheva Program Manager, Exactpro

Overview ● Definition ● Properties / requirements ● Alternatives to MBT ● 3 problems with MBT ● Processes, platforms and people to make MBT a success What model is a good model? Model-based testing (MBT) MBT challenges and perspectives

About me 2005 - 2018 – General and applied linguistics (Saratov State University): - linguistic database engineer - head of the Laboratory of Applied Linguistics - PhD in general linguistics (2012) - associate professor in computer linguistics 2017 – Software testing, QA analyst (Exactpro, Saratov) 2018 – Business Development and Research (Exactpro, USA) https://www.linkedin.com/in/treshcheva/

A specialist firm focused on functional and non-functional testing services for financial market infrastructures, financial and non-financial institutions. Headquartered in the UK with operations in the US, Georgia, Russia, Lithuania and Sri Lanka. Most of our clients are regulated market infrastructures. Incorporated in 2009 with 10 people, Exactpro now employs over 700 specialists. Part of the London Stock Exchange Group (LSEG) from May 2015 until January 2018, when the Exactpro management proceeded through the buyout of the company from LSEG. Exactpro provides independent software testing services for mission critical technology that underpins global financial markets. The firm is experienced with trading and clearing and settlement platforms, risk management systems, central data warehouses, accounting and treasury systems, regulatory reporting. About Exactpro

Global Client Network – Financial Market Infrastructures

6 Build Software to Test Software exactpro.com Skytra selects Exactpro to test its new derivatives trading software Jeremy Norwood, CIO of Skytra: “Our timescales for launch are challenging, and we needed an experienced IT partner who could help us ensure that the delivered software works as per our business, operational and regulatory requirements. Exactpro will conduct a number of independent, unbiased functional and regression test cycles as part of our overall delivery programme. I am confident that Exactpro, with its proven track record of successful delivery of quality assurance projects for the financial services industry, will provide exceptional service for Skytra.” https://exactpro.com/news/exactpro/skytra-selects-exactpro-test-its-new-derivatives-trading-software

7 Build Software to Test Software exactpro.com Exactpro Supports Launch of MEMX into the U.S. Equity Market with Quality Assurance Testing “MEMX’s close collaboration with Exactpro provided additional operational resilience and contributed to the exchange’s seamless launch, rollout and, ultimately, live trading in all NMS symbols,” said MEMX Chief Technology Officer Dominick Paniscotti. “We greatly appreciate Exactpro’s partnership in working to ensure the exchange’s technology was ready to perform at full capacity from day one.” Read our latest Case Study: MEMX-Exactpro Collaboration on Exchange Quality Assurance https://exactpro.com/news/exactpro/exactpro-supports-launch-memx-us-equity-market-quality-assurance-testing

8 Build Software to Test Software exactpro.com https://exactpro.com/news/exactpro/ledgeredge-selects-exactpro-deliver-resilience-its-distributed -ledger-enabled LedgerEdge Selects Exactpro to Deliver Resilience for its Distributed Ledger Enabled Corporate Bond Trading Ecosystem. Robert Bose, Chief Technology Officer at LedgerEdge: “We are passionate about delivering the future of corporate bond trading. As we get closer to launch, it’s necessary to have a trusted partner to help us test our operational resilience and ensure we can meet the demands of the market. Exactpro’s bespoke tools and methods will enable us to achieve this, and we are excited to see the results of our collaboration.”

What is a model? an interpretation of a theory an artifact aimed to describe a system, formulated in a modeling / programming language real-world system model model

Model-based testing The MBT flow: - Building a model based on a specification - Automated generation of abstract test scenarios - Abstract test scenarios become executable scripts - Scripts are executed against the system - Test results analysis Requirement/System Specifications MBT Models Abstract Test Case Test Results SUT SystemAdapters ExecutionEnvironment Executable Test Case Mappings Test Selection Criteria 1 2 3 4 5

Model-based testing Can be both mental representation or a schema or a code-based artifact Requirement/System Specifications MBT Models Abstract Test Case Test Verdicts SUT SystemAdapters ExecutionEnvironment Executable Test Case Mappings Test Selection Criteria 1 2 3 4 5

Some of the existing test approaches: Feasible alternatives? Developer QA BA BDD

Some of the existing test approaches: Feasible alternatives? Parallel run / Operational day replay approach

All Testing is Model Based Testing

Properties of a model mapping feature reduction feature pragmatic feature Herbert Stachowiak. Allgemeine Modelltheorie. Springer-Verlag, Wien and New York, 1973. (Cited by Kühne, Thomas. "What is a Model?." In Dagstuhl Seminar Proceedings. Schloss Dagstuhl-Leibniz-Zentrum für Informatik, 2005).

Testing vs. Confirmation pragmatic map reduce

The Exactpro Perspective Explicit Pragmatic Multi-Layered

Defense in Depth

State of the Art Mohd-Shafie M.L., Kadir W.M.N.W, Lichter H., Khatibsyarbini M., Isa M.A. Model‑based test case generation and prioritization: a systematic literature review. 2021, Springer.

Taxonomy M. Utting, A. Pretschner, B. Legeard, B., “A taxonomy of model-based testing”, Special Issue: Model-based Testing Volume 1: Foundations and Applications of Model-Based Testing, volume 22, issue 5. August, 2012.

Challenges

Processes Platforms People Challenges

Processes If complex models have to be completed before testing can start, this induces an unacceptable delay for the proper test executions.

Platforms For complex SUT, like systems of systems, test models need to abstract from a large amount of detail, because otherwise the resulting test model would become unmanageable.

Platforms Delivers end-to-end automated functional and non-functional testing of complex financial systems Enables intelligent interaction with many widely adopted network protocols as well as API, UI, DLT and cloud endpoints Executes sophisticated test algorithms Collects and processes distributed test data (for machine learning and other purposes) Performs model-based testing and analyses the behavior of systems under test Integrates with a variety of widely adopted test tools and frameworks via its open interface Is an open-source solution; th2 source code is available on GitHub Platform and Technology Agnostic Customised for Financial Systems Unified Data Warehouse Deliberate Practice of Software Testing AI-driven Strategic Data Acquisition Pervasive Automation Opportunities “th2 represents the next generation of automated testing platforms. We believe its usage will eventually transform the way firms approach their functional and non-functional testing processes. We look forward to rolling the framework out over the coming months and developing further use cases with our clients and partners.” CLICK TO READ MORE

Platforms SUT DW Diversity

People The required skills for test engineers writing test models are significantly higher than for test engineers writing sequential test procedures.

28 BUILD SOFTWARE TO TEST SOFTWARE Build Software to Test Software exactpro.com FOLLOW US ON SOCIAL MEDIA: exactpro.com