Testing of AI-based Solutions

Iosif Itkin
CEO and co-founder, Exactpro

“AI-based solutions contain many components and model configuration parameters, all of which have to be tested to ensure the quality of the model training process. In addition, intelligent systems have machine learning capabilities that will result in different outcomes, if the same function is called at different moments in time. We invite you to discuss how the multicomponent nature and non-deterministic behaviour of AI-based systems can be mitigated in testing.”

QA Meetup. 25 April 2022

https://exactpro.com/events/external/qa-meetup-25-april?utm_source=speakerdeck&utm_medium=Refferer&utm_campaign=testing-of-ai-based-solutions

---

Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro

Transcript

1. 1 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup BUILD SOFTWARE TO TEST SOFTWARE exactpro.com Testing of AI-based Solutions 25 APRIL 2022 CEO and co-founder, Exactpro Iosif Itkin ONLINE | HATCH WORKS, COLOMBO 1
2. 2 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup Contents • About Exactpro • AI-based Financial Systems • Conversational Assistants • Self Driving Cars • Certified Tester AI Testing • Non-Deterministic Systems
3. 3 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup About Exactpro A specialist firm focused on functional and non-functional testing for complex, distributed, non-deterministic and artificial intelligence platforms Headquartered in the UK with operations in the US, Georgia, Lithuania and Sri Lanka. The company opens additional offices in Yerevan, Abu Dhabi, Toronto, Milan and Sydney in 2022. Exactpro provides software testing services for mission critical technology that underpins global financial markets. The firm is experienced with trading and clearing and settlement platforms, market data systems, collateral management and risk management systems, central data warehouses, regulatory reporting, etc. Incorporated in 2009 with 10 people, Exactpro is now employing over 700 specialists. Most of our clients are regulated market infrastructures. Part of the London Stock Exchange Group (LSEG) from May 2015 until January 2018, when the Exactpro management proceeded through the buyout of the company from LSEG.
4. 4 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup About Exactpro exactpro.com Iosif Itkin, Malta CEO Exactpro Group Alexey Zverev, UK CEO Exactpro Group Natia Sirbiladze, Georgia CEO Exactpro Georgia Kirill Zagorouiko, Canada Chief Operating Officer Alyona Lamash, UK Director, Head of Risk Management Practice Thomas Toller, USA Managing Director Jagath de Silva, Sri Lanka CEO Exactpro Sri Lanka Ian Salmon, UK Business Development Michael Smith, UK Head of Sales Lilia Tira, UK Business Development Hiroshi Matsubara, Japan Director, Business Development Maxim Rudovsky, UK Chief Technology Officer Victoria Leonchik, UK Technology, Data Warehouse Boris Rabinovich, Israel Technology, Derivatives Alyona Bulda, Sri Lanka Head of Global Exchanges, SVP Elena Trescheva, USA Program Manager Natia Sirbiladze, Georgia Chief Financial Officer Asya Legotina, UK Technology, Big Data Rostislav Yavorski, Sri Lanka Head of Research Sona Oganesyan, Armenia CEO Exactpro Armenia Kapila Karunaratne, Sri Lanka Technology, Head of Delivery
5. 5 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup About Exactpro
6. 6 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup Exactpro in the Media
7. 7 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup
8. 8 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 8 Build Software to Test Software exactpro.com Confidential exactpro.com Exactpro Supports the Launch of MEMX into the U.S. Equity Market with Quality Assurance Testing “MEMX’s close collaboration with Exactpro provided additional operational resilience and contributed to the exchange’s seamless launch, rollout and, ultimately, live trading in all NMS symbols,” said MEMX Chief Technology Officer Dominick Paniscotti. “We greatly appreciate Exactpro’s partnership in working to ensure the exchange’s technology was ready to perform at full capacity from day one.” Read our latest Case Study: MEMX-Exactpro Collaboration on Exchange Quality Assurance https://exactpro.com/news/exactpro/exactpro-supports-launch-memx-us-equity-market-quality-assurance-testing
9. 9 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 9 Build Software to Test Software exactpro.com Confidential exactpro.com Skytra selects Exactpro to test its new derivatives trading software Jeremy Norwood, CIO of Skytra: “Our timescales for launch are challenging, and we needed an experienced IT partner who could help us ensure that the delivered software works as per our business, operational and regulatory requirements. Exactpro will conduct a number of independent, unbiased functional and regression test cycles as part of our overall delivery programme. I am confident that Exactpro, with its proven track record of successful delivery of quality assurance projects for the financial services industry, will provide exceptional service for Skytra.” https://exactpro.com/news/exactpro/skytra-selects-exactpro-test-its-new-derivatives-trading-software
10. 10 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 10 Build Software to Test Software exactpro.com https://exactpro.com/news/exactpro/ledgeredge-selects-exactpro-deliver-resilience-its-distributed-ledger-enabled LedgerEdge Selects Exactpro to Deliver Resilience for its Distributed Ledger Enabled Corporate Bond Trading Ecosystem. Robert Bose, Chief Technology Officer at LedgerEdge: “We are passionate about delivering the future of corporate bond trading. As we get closer to launch, it’s necessary to have a trusted partner to help us test our operational resilience and ensure we can meet the demands of the market. Exactpro’s bespoke tools and methods will enable us to achieve this, and we are excited to see the results of our collaboration.”
11. 11 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup
12. 12 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup offers stock exchanges, clearing houses, central securities depositories (CSDs), and other financial infrastructures a comprehensive technology agnostic AI-driven test automation solution. It aims to help regulated capital markets participants stay compliant and resilient to disruption, while focusing on innovation and having the freedom to embrace emerging technologies in response to their clients’ needs. Built with these goals in mind, th2 provides unprecedented flexibility, breadth and depth of software testing to the financial industry. Tackling the ever-increasing complexity of financial platforms, th2 is a next-generation toolkit for automated functional and non-functional testing of distributed transaction processing systems. These include securities trading systems and exchanges, banking and broker systems, post-trade (e.g. clearing, settlement, custody) and payments platforms, and many more. th2 is a Kubernetes-driven microservices solution tailor-made to deliver efficient machine-driven end-to-end test libraries with comprehensive coverage of your system. th2 consolidates the power of the entire Exactpro test tool suite in a single solution. Platform and Technology Agnostic Customised for Financial Systems Unified Data Warehouse Deliberate Practice of Software Testing AI-driven Strategic Data Acquisition Pervasive Automation Opportunities. The th2 source code has been released on GitHub and is open for contributions from the software testing and development community. Professionals working with th2 are Software Development Engineers in Test who combine the roles of programmers, testers and data analysts equally well. ADDRESSING THE NEW LEVELS OF COMPLEXITY IN FINANCIAL SYSTEMS exactpro.com Delivers end-to-end automated functional and non-functional testing of complex financial systems Enables intelligent interaction with many widely adopted network protocols as well as API, UI, DLT and cloud endpoints Executes sophisticated test algorithms Collects and processes distributed test data (for machine learning and other purposes) Performs model-based testing and analyses the behaviour of systems under test Integrates with a variety of widely adopted test tools and frameworks via its open interface Is an open-source solution; th2 source code is available on GitHub Platform and Technology Agnostic Customised for Financial Systems Unified Data Warehouse Deliberate Practice of Software Testing AI-driven Strategic Data Acquisition Pervasive Automation Opportunities 12
13. exactpro.com 13 BUILD SOFTWARE TO TEST SOFTWARE Confidential 13 EXACTPRO ANNOUNCED RUNNER-UP IN SWIFT HACKATHON 2021 The organisers of SWIFT Hackathon 2021 have announced the winners and runners-up of two hackathon challenges launched in early September this year. Exactpro is recognised as runner-up in Challenge 2 – Building ‘synthetic’ data-sets required for AI-based product development, whilst protecting privacy. The announcement came as part of the Sibos 2021 premiere financial services event program that has been taking place this week. From the official kick-off session on 7 September to the wrap-up hours on 24 September, 25 teams from across the industry worked hard to apply their unique expertise, industry’s latest research and best practices to be able to present their solutions to the SWIFT community at this year’s Sibos. The hackathon challenges aim to reflect the increasingly important role artificial intelligence (AI) and machine learning (ML) are playing in financial services, with both tracks serving the purpose of facilitating the adoption of ML and AI technology in the banking space. The two challenges featured were: • Challenge 1: Enhancing the accuracy of anomaly detection in payments • Challenge 2: Building ‘synthetic’ data-sets required for AI-based product development, whilst protecting privacy Exactpro’s solution submitted in Challenge 2 is based on years of experience of growing in-house test data development capabilities. Iosif Itkin, Exactpro CEO and co-founder, comments: “Leveraging the in-house expertise and deep domain knowledge of our team that included business analysts, software testing engineers, developers, researchers and data scientists, Exactpro has been able to produce a scalable solution that can serve the industry in so many ways, from improving anti-money laundering (AML) and fraud detection mechanisms to building a variety of personalised payments products.” Rostislav Yavorsky, Head of Research, Exactpro, says: “Challenge 1 is where the contestants had to be most pedantic about their solutions and relentless about reaching the desired metrics. Challenge 2 is where they had to apply utmost ingenuity and creativity in solving the task at hand. In both cases, the teams had to demonstrate painstaking attention to detail, work hard within a limited timeframe and coordinate exceptionally well, while being miles away from each other. I’m really proud of the results we’ve achieved. The value of reliable GDPR-compliant solutions for generating synthetic data-sets for market infrastructure institutions and fintechs is indisputable.”
14. 14 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup AI-based Financial Systems Machine Learning in financial organizations: - already passed an initial development phase - the usage of live ML applications is about to dramatically increase over the next three years • Market Surveillance Systems • Conversational Assistants • Algo Trading Systems • Pricing Calculators • Machine Readable News • Insurance Claims https://www.bankofengland.co.uk/-/media/boe/files/report/ 2019/machine-learning-in-uk-financial-services.pdf
15. 15 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 15 BUILD SOFTWARE TO TEST SOFTWARE Market Surveillance Systems Market Participants Market surveillance systems are targeted at helping exchanges to maintain orderly markets by analyzing all events in the system to detect malicious participants and prevent market abuse behavior, such as prices or volumes manipulation, money laundering, front-running, etc. With data analytics and pattern recognition involved, market surveillance systems are one of the examples of application of AI to the financial services industry.
16. 16 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 16 BUILD SOFTWARE TO TEST SOFTWARE Market Surveillance Systems Market Participants Information Manipulation Artificiality of control for the value of assets - price (and not a violation of the law in the transaction) Misleading other market participants, falsity of price signals
17. 17 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 17 BUILD SOFTWARE TO TEST SOFTWARE Conversational Assistants • Task-oriented dialogue systems submit an order, setup an alert notification, make a transaction… • Question answering (factoid/non-factoid) When do you use MT569 message? How to change system password? • Chitchat Siri, my brother Lesha says that all robots are stupid. It’s not me. It’s him (Maria, 9 years) • Advanced applications: mix of above ◦ challenges: focus switching/intent detection/mixed initiative/recognition of emotions/maintaining context/personalization
18. 18 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 18 BUILD SOFTWARE TO TEST SOFTWARE Public Relations Disaster vs. Research Breakthrough
19. 19 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 19 BUILD SOFTWARE TO TEST SOFTWARE Self Driving Cars
20. 20 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 20 BUILD SOFTWARE TO TEST SOFTWARE Build Software to Test Software
21. 21 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 21 BUILD SOFTWARE TO TEST SOFTWARE Applying Natural Language Processing
22. 22 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 22 BUILD SOFTWARE TO TEST SOFTWARE Uber Volvo XC90 Crash https://www.ntsb.gov/investigations/Pages/HWY18MH010.aspx
23. 23 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 23 BUILD SOFTWARE TO TEST SOFTWARE Uber Volvo XC90 Crash
24. 24 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 24 BUILD SOFTWARE TO TEST SOFTWARE Production ML Code https://developers.google.com/machine-learning/crash-course/production-ml-systems Sensors Actuators
25. 25 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 25 BUILD SOFTWARE TO TEST SOFTWARE Complex Systems
26. 26 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 26 BUILD SOFTWARE TO TEST SOFTWARE ISTQB Certification
27. 27 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 27 BUILD SOFTWARE TO TEST SOFTWARE Challenges of Testing AI Systems Non-Deterministic Probabilistic Non-Testable Oracle Problem
28. 28 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 28 BUILD SOFTWARE TO TEST SOFTWARE Multi-Threaded Distributed Application
29. 29 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 29 BUILD SOFTWARE TO TEST SOFTWARE Challenges of Testing Self-Learning Systems • Unexpected change • Complex acceptance criteria • Insufficient testing time • Resource requirements • Insufficient specifications of operational environment • Complex test environment • Undesirable behavior modifications
30. 30 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 30 BUILD SOFTWARE TO TEST SOFTWARE Challenges of Testing Self-Learning Systems Certified Tester AI Testing Syllabus defines the following challenges in testing self-learning systems: • Unexpected change • Complex acceptance criteria • Insufficient testing time • Resource requirements • Insufficient specifications of operational environment • Complex test environment • Undesirable behavior modifications The Exactpro teams handle these type of challenges on a daily basis while working with our clients’ trading, clearing, market surveillance and DLT platforms. While testing distributed complex systems, especially related to auctions and implied liquidity, we have developed approaches for testing probabilistic and non-deterministic systems, including metamorphic testing. We deal with the issues of transparency, explainability, interpretability, autonomy and human intervention every time we build and use a large-scale library of automated tests. Exactpro is extremely well positioned for testing AI-based and hybrid systems. We are working on expanding our service offering in this area.
31. 31 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup 31 BUILD SOFTWARE TO TEST SOFTWARE https://www.youtube.com/c/exactprosystems
32. 32 BUILD SOFTWARE TO TEST SOFTWARE QA Meetup Thank You!