The Third IEEE International Conference On Artificial Intelligence Testing Iosif Itkin, CEO and co-founder, Exactpro Elena Treshcheva, Program Manager, Exactpro DATA-DRIVEN TESTING AND TEST AUTOMATION FOR COMPLEX INTELLIGENT SYSTEMS

2 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 2 The 2021 IEEE Autonomous Driving AI Test Challenge August 23-26, 2021 - Online, Worldwide August 23 - team demos August 26 - selected paper presentations August 26 - Award Ceremony (5:00 pm BST / 9:00 am PST)

BUILD SOFTWARE TO TEST SOFTWARE The Third IEEE International Conference On Artificial Intelligence Testing 3 Overview ● Fintech applications complexity ● Present-day requirements of performance and high availability ● Requirements for next-generation test automation tools ● HiVAT paradigm th2 architecture Open-source model What Why How

4 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 4 A specialist firm focused on functional and non-functional testing services for financial market infrastructures, financial and non-financial institutions. Headquartered in the UK with operations in the US and Eastern Europe. Most of our clients are regulated market infrastructures. Incorporated in 2009 with 10 people, Exactpro is now employing over 650 specialists. Part of the London Stock Exchange Group (LSEG) from May 2015 until January 2018, when the Exactpro management proceeded through the buyout of the company from LSEG About Exactpro Exactpro provides independent software testing services for mission critical technology that underpins global financial markets. The firm is experienced with trading and clearing & settlement platforms, risk management systems, central data warehouses, accounting and treasury systems, regulatory reporting.

5 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 5 Global Client Network – Financial Market Infrastructures

BUILD SOFTWARE TO TEST SOFTWARE The Third IEEE International Conference On Artificial Intelligence Testing 6 WHY

7 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 7 Software complexity Dynamic why? why? Component Coordinative

8 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 8 Component complexity (AI, NLP) why? https://www.stateof.ai/ why?

9 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 9 why? Coordinative and dynamic complexity (fintech platforms)

10 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 10 Continued surge of technology advancements Data Volumes Computational Power why?

11 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 11 Recent Advances in Machine Learning Data Volumes why? https://www.stateof.ai/

12 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 12 Recent Advances in Machine Learning Computational Power why? https://www.stateof.ai/

13 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 13 ● Daily capacity – 300+ mln transactions ● Peakrates – 50k transactions per second per partition ● Average round-trip latency – <60 microseconds ● Availability – high nines 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2 5 6.5 9 12.5 15.5 18 26 33 41 59 74 94 118 149 Data Volume in zettabytes why? Fintech platforms: requirements

14 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 14 Distributed, resilient latency sensitive systems

15 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 15 Non-determinism Traditional Technology AI-based Software why?

16 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 16 Testing tool to match the complexity of the system under test System under test Test Library Complex System – Simple Tools Complex System – Complex Tools why?

BUILD SOFTWARE TO TEST SOFTWARE The Third IEEE International Conference On Artificial Intelligence Testing 17 WHAT

18 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 18 Testing tool to match the complexity of the system under test what?

19 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 19 Requirements Functional Testing Non-Functional Testing what?

20 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 20 Requirements Diversity and Unification Scalability Resilience Data Analysis Modularity Performance Configuration Agility what? Explainability

21 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 21 Performance and Scalability SUT DW what?

22 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 22 Diversity and Unification SUT DW Diversity what?

BUILD SOFTWARE TO TEST SOFTWARE The Third IEEE International Conference On Artificial Intelligence Testing 23 HOW

24 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 24 Open-source model how? https://github.com/th2-net

25 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 25 Open-source model how?

BUILD SOFTWARE TO TEST SOFTWARE 26 The Third IEEE International Conference On Artificial Intelligence Testing Open for Collaboration

27 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 27 TMPA - Tools & Methods of Program Analysis Conference https://tmpaconf.org/events/tmpa-2021/

28 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 28 Exactpro YouTube Channel Software Testing for Complex Intelligent Systems and Autonomous Vehicles Online Course Software testing for complex intelligent systems and autonomous vehicles - Playlist on the Exactpro YouTube channel

29 The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE 29 References 1. Jung, C., Mueller, H., Pedemonte, S., Plances, S., & Thew, O. Machine learning in UK financial services. Joint report by the FCA and the Bank of England. October 2019. 2. LabCFTC (Commodities and Futures Trading Commission). A Primer on Artificial Intelligence in Financial Markets. October 2019. 3. Zhang, J. M., Harman, M., Ma, L., & Liu, Y. Machine Learning Testing: Survey, Landscapes and Horizons. In: IEEE Transactions on Software Engineering. 2020. 4. Treshcheva, E., Yavorsky, R., & Itkin, I. Toward Reducing the Operational Risk of Emerging Technologies Adoption in Central Counterparties through End-to-End Testing. In: Journal of Financial Market Infrastructures 8(3), 2020: 51-74. 5. Wood, R.E. Task Complexity: Definition of a Construct. In: Organizational Behavior and Human Decision Processes, vol. 31, 1986: 60-82. 6. Banker, R. D., Davis, G. B., & Slaughter, S. A. Software development practices, software complexity, and software maintenance performance: A field study. Management science 44(4), 1998: 433-45 7. Protsenko, P., Khristenok, A., Lukina, A., Alexeenko, A., Pavlyuk, T., & Itkin, I. Trading day logs replay limitations and test tools applicability. In Proceedings. International Conference on Tools and Methods of Program Analysis (TMPA 2014), 2014: 46-53. 8. Benaich, N. & Hogarth, I. State of AI Report 2020. October, 2020. 177p.

Thank you! The Third IEEE International Conference On Artificial Intelligence Testing BUILD SOFTWARE TO TEST SOFTWARE