Build Software to Test Software exactpro.com Artificial Intelligence, Bias and Fake Software Testing Alexey Zverev, Sibos 2018 Sydney, Australia

2 Build Software to Test Software exactpro.com Build Software to Test Software • A specialist firm focused on functional and non-functional testing of exchanges, clearing houses, depositories and other market infrastructures • Part of London Stock Exchange Group (LSEG) from May 2015 till January 2018. Exactpro management buyout from LSEG in January 2018. • We provide software testing services for mission critical technology that underpins global financial markets. Our clients are regulated by FCA, Bank of England and their counterparts from other countries. • Incorporated in 2009 with 10 people, our company has experienced significant growth as satisfied clients require more services; now employing 550 specialists.

3 Build Software to Test Software exactpro.com Types of systems that we test

4 Build Software to Test Software exactpro.com Fake Testing vs Testing

5 Build Software to Test Software exactpro.com • It is very important to prepare all test cases upfront and have them signed off by the stakeholders before starting test execution • All tests must be traceable to requirements • We must be very careful and avoid raising invalid defects • The metric of our releases quality is test case pass/fail ratio • We rely on an industry-standard test management system to report out test execution Fake Testing Markers

6 Build Software to Test Software exactpro.com • Confirmation bias/Expectancy bias • Illusion of control • Law of the instrument • Ambiguity Effect/Ignorance of outliers • Anchoring • Anthropocentric Principle Example of biases driving Fake Testing

7 Build Software to Test Software exactpro.com Consequences!

8 Build Software to Test Software exactpro.com • We are using a UI automation tool to test complex clearing systems • We have automated 85% of our tests • Performance, load and failover testing are considered to be a totally separate activity which is out of scope of our interests • Unit Tests from developers are our automated tests • We are relying on parallel run testing • We are removing “unrealistic scenarios” from our library • We are automating scenarios provided by “experts” Test Automation (Markers of Fake Testing)

9 Build Software to Test Software exactpro.com Software Testing is Relentless Learning

10 Build Software to Test Software exactpro.com Automated Testing is Machine Learning

11 Build Software to Test Software exactpro.com • Nostradamus • Smart Log Monitor • Smart Load Injector • The Big Button Results Analyzer • Cradle Test Database AI and Machine Learning in Exactpro

12 Build Software to Test Software exactpro.com Cradle Test Database