Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 1 Disruptive Functional Testing The next frontier in Post-Trade systems verification Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 2460, +1 415 830 38 49 www.exactpro.com 16 October 2017, SIBOS Conference Iosif Itkin, Exactpro CEO and co-founder, London Stock Exchange Group

• A specialist firm focused on functional and non-functional testing of systems that process wholesale financial products, particularly market infrastructures • A UK company with operations in the US and four QA & software development centres in Russia • Part of London Stock Exchange Group as of May 29, 2015 • Incorporated in 2009 with 10 people, our company has experienced significant growth as satisfied clients require more services; now employing 536 specialists Front office Listed Derivatives CRM Dark Pool T25 Clearing FX Fixed income SEF Reconciliation Onboarding Ticker Plant Commondities Smart Order Router Trading Desktops Index Dissemination FPGA Equities Connectivity Exchange MTF Swaptions Exactpro Timeline at a Glance

Delivering Quality to London Stock Exchange Group Some of our projects (start date): •Turquoise (2010) •Borsa Italiana (2011) •CCP.A (2011) •FPGA-base Ticker Plant (2012) •LSE (2012) •Oslo Bors (2012) •UnaVista (2013) •FTSE (2013) •EuroTLX (2014) •Russell (2015) •IDEM (2015) •LSEDM (2015) •ELITE (since 2015) •CurveGlobal (2016) •GATElab (2016) •LCH (2016) •TRADEcho (2016) •MTS (2017) •RNS (2017)

Open Access Quality Assurance – a Global Client Network

Faster, not Bigger What is the main difference between incumbent and disruptive? Incumbent means something you already know and expect. Disruptive is when we encounter new things that will re-define what we already know.

Post Trade Systems

Software Testing Challenges

With incumbent testing, you are on the hook of pre-conceptions of how your system under test is supposed to work. It's only when you do disruptive testing that you really gain new knowledge about the system and learn from it. Get Off the Hook

Software Testing is Relentless Learning

Defect Mining The goal of disruptive testing is to find defects that are unlikely to be found if testing is done with other methods. We see ourselves as a mining operation extracting defects hidden in complex scalable technology platforms.

Test Harness

Agile Transformation As in any other large organization in the financial services industry, technology used at the London Stock Exchange Group is going through agile transformation.

Developing a Test Harness for Agile Process

What Agile Development Should Be… and Not NOT LIKE THIS LIKE THIS

Testing Challenges in Agile

Managing Complex Systems The Law of Requisite Variety: a control system must have at least as many possible states as the system it wants to control. It means that Test Harness development is a truly challenging and continuous software development process.

Test Harness Team

Testing Critical Infrastructures

Testing Critical Infrastructures Safety Rule #1 with Submarines: don't open portholes when underwater!

Testing Critical Infrastructures Safety Rule #1 with Submarines: don't open portholes when underwater! Functional testing: iterate through a finite number of scenarios to prove that the porthole won't open Non-Functional testing: iterate through a smaller number of scenarios to prove that it won't open by brute force

Testing Critical Infrastructures Safety Rule #1 with Submarines: don't open portholes when underwater! Functional testing: iterate through a finite number of scenarios to prove that the porthole won't open Non-Functional testing: iterate through a smaller number of scenarios to prove that it won't open by brute force Disruptive testing: 1) iterate through a huge number of random diverse scenarios under load to prove that it won't open

Testing Critical Infrastructures Safety Rule #1 with Submarines: don't open portholes when underwater! Functional testing: iterate through a finite number of scenarios to prove that the porthole won't open Non-Functional testing: iterate through a smaller number of scenarios to prove that it won't open by brute force Disruptive testing: 1) iterate through a huge number of random diverse scenarios under load to prove that it won't open 2) open the porthole

Thank you