1 BUILD SOFTWARE TO TEST SOFTWARE International Summer School on Data Science in Software Engineering Iosif Itkin, Exactpro A Reference Architecture for HiVAT (High Volume Automated Testing) 12.07.2021 — 16.07.2021

2 BUILD SOFTWARE TO TEST SOFTWARE Contents 1. Introduction 2. Build Software to Test Software 3. High Volumes Automated Testing 4. Performance and Scalability 5. Diversity and Unification 6. Resilience and Antifragility 7. Reference Implementation (th2) 8. Q&A

3 BUILD SOFTWARE TO TEST SOFTWARE Introduction ● Started IT career 20+ years ago as a software developer ● Moved into performance and load testing in 2006 ● Co-founded Exactpro on 9th September 2009 ● Head of LOST (Laboratory of Software Testing) at Tomsk Polytechnic University ● Co-chair of FIX Trading Community MOST Working Group (Monitoring, Onboarding and Software Testing) ● TMPA Tools & Methods of Program Analysis Conference

4 BUILD SOFTWARE TO TEST SOFTWARE TMPA Tools & Methods of Program Analysis Conference

5 BUILD SOFTWARE TO TEST SOFTWARE About Exactpro A specialist firm focused on functional and non-functional testing services for financial market infrastructures Headquartered in the UK with operations in the US, Georgia, Russia. This year Exactpro is opening two additional delivery centers in Lithuania and Sri Lanka Exactpro provides independent software testing services for mission critical technology that underpins global financial markets. The firm is experienced with trading and clearing & settlement platforms, risk management systems, central data warehouses, accounting and treasury systems, regulatory reporting. Incorporated in 2009 with 10 people, Exactpro now employs over 650 specialists. Part of the London Stock Exchange Group (LSEG) from May 2015 until January 2018, when the Exactpro management proceeded through the buyout of the company from LSEG.

6 BUILD SOFTWARE TO TEST SOFTWARE Our Clients – 10 of the World’s Top 20 Systemically Important Exchanges and Clearing Houses

7 Build Software to Test Software exactpro.com Confidential exactpro.com Skytra selects Exactpro to test its new derivatives trading software Jeremy Norwood, CIO of Skytra: “Our timescales for launch are challenging, and we needed an experienced IT partner who could help us ensure that the delivered software works as per our business, operational and regulatory requirements. Exactpro will conduct a number of independent, unbiased functional and regression test cycles as part of our overall delivery programme. I am confident that Exactpro, with its proven track record of successful delivery of quality assurance projects for the financial services industry, will provide exceptional service for Skytra.” https://exactpro.com/news/exactpro/skytra-selects-exactpro-test-its-new-derivatives-trading-software

8 BUILD SOFTWARE TO TEST SOFTWARE 8 Build Software to Test Software exactpro.com Confidential exactpro.com Exactpro Supports Launch of MEMX into the U.S. Equity Market with Quality Assurance Testing “MEMX’s close collaboration with Exactpro provided additional operational resilience and contributed to the exchange’s seamless launch, rollout and, ultimately, live trading in all NMS symbols,” said MEMX Chief Technology Officer Dominick Paniscotti. “We greatly appreciate Exactpro’s partnership in working to ensure the exchange’s technology was ready to perform at full capacity from day one.” Read our latest Case Study: MEMX-Exactpro Collaboration on Exchange Quality Assurance https://exactpro.com/news/exactpro/exactpro-supports-launch-memx-us-equity-market-quality-assurance-testing

9 BUILD SOFTWARE TO TEST SOFTWARE Typical Project

10 BUILD SOFTWARE TO TEST SOFTWARE What makes us different The Value of Understanding Your Technology Assets Technology is the main asset of every modern company. We help our clients build, operate and transform their technology to increase its value. Our Deliberate Practice of Software Testing approach is based on modelling the entire system under test (SUT), which allows us to create test libraries that serve as an executable specification for the highly complex platforms we test. This is a ‘deliberate’ practice, meaning that it is systematic and focused on achieving the specific goal of improving our capacity of extracting information about the SUT. https://www.fixtrading.org/wp-content/uploads/2021/06/Pre-Event-Magazine-EMEA-Trading-Confer ence-2021.pdf Read more in Mikhail Kurkov’s article on The Value of Understanding Your Technology Assets in the exclusive FIX Trading Community pre-event EMEA Trading Conference 2021 magazine.

11 BUILD SOFTWARE TO TEST SOFTWARE Software Testing Software Testing is an ● empirical ● technical ● investigation ● conducted to provide stakeholders ● with information ● about the quality ● of the product or service under test Software Testing is exploring software with the intent of finding bugs

12 BUILD SOFTWARE TO TEST SOFTWARE Automated Testing means Computer-Assisted Testing

13 BUILD SOFTWARE TO TEST SOFTWARE High Volume Automated Testing (HiVAT) http://kaner.com/pdfs/highvolCSTER.pdf http://kaner.com/?p=278 HVTA / HiVAT

14 BUILD SOFTWARE TO TEST SOFTWARE High Volume Automated Testing (HiVAT) A family of test techniques that use software to generate, execute and interpret arbitrarily many tests A family of test techniques that empower a tester to focus on interesting results while using software to design, implement, execute, and interpret a large suite of tests (thousands to billions of tests) Automated Testing means Computer-Assisted Testing

15 BUILD SOFTWARE TO TEST SOFTWARE Recent Advances in Machine Learning Data Volumes Computational Power

16 BUILD SOFTWARE TO TEST SOFTWARE Performance and Scalability Daily capacity - 100+ mln transactions Peak rates - 40k+ transactions per second Average round-trip latency - <100 microseconds 3000 trx 2.5 cm <1 mm

17 BUILD SOFTWARE TO TEST SOFTWARE Performance and Scalability

18 BUILD SOFTWARE TO TEST SOFTWARE Performance and Scalability SUT DW

19 BUILD SOFTWARE TO TEST SOFTWARE Diversity and Unification Functional Testing Non-Functional Testing

20 BUILD SOFTWARE TO TEST SOFTWARE Diversity and Unification

21 BUILD SOFTWARE TO TEST SOFTWARE Diversity and Unification

22 BUILD SOFTWARE TO TEST SOFTWARE Diversity and Unification

23 BUILD SOFTWARE TO TEST SOFTWARE Diversity and Unification SUT DW Diversity

24 BUILD SOFTWARE TO TEST SOFTWARE Resilience and Antifragility

25 BUILD SOFTWARE TO TEST SOFTWARE Resilience and Antifragility SUT DW

26 BUILD SOFTWARE TO TEST SOFTWARE Authonomy

27 BUILD SOFTWARE TO TEST SOFTWARE th2 Next-generation Test Automation Framework https://github.com/th2-net

28 BUILD SOFTWARE TO TEST SOFTWARE Delivers end-to-end automated functional and non-functional testing of complex financial systems Enables intelligent interaction with many widely adopted network protocols as well as API, UI, DLT and cloud endpoints Executes sophisticated test algorithms Collects and processes distributed test data (for machine learning and other purposes) Performs model-based testing and analyses the behaviour of systems under test Integrates with a variety of widely adopted test tools and frameworks via its open interface Is an open-source solution; th2 source code is available on GitHub exactpro.com Platform and Technology Agnostic Customised for Financial Systems Unified Data Warehouse Deliberate Practice of Software Testing AI-driven Strategic Data Acquisition Pervasive Automation Opportunities “th2 represents the next generation of automated testing platforms. We believe its usage will eventually transform the way firms approach their functional and non-functional testing processes. We look forward to rolling the framework out over the coming months and developing further use cases with our clients and partners.” has been successfully deployed in a number of use cases. Its first adopter – Airbus' subsidiary Skytra – harnessed the technology for its derivatives trading software. Its most recent implementation was for ML-driven testing of Quod Financial’s technology READ MORE th2 Next-generation Test Automation Framework

29 BUILD SOFTWARE TO TEST SOFTWARE Q&A Thank You