Non-Functional Testing at London Stock Exchange

29 November 2016
Harvard Club, New York, QA-Financial Forum
Exactpro is the specialist testing subsidiary of the London Stock Exchange Group. In this presentation,
Iosif Itkin will cover the wide range of testing requirements faced by broker-dealers, banks and
exchanges beyond the requirements of functional testing. Plus: In Conversation with Yann L’Huillier,
CIO, Tradition and Iosif Itkin, CEO, Exactpro
- Iosif Itkin: Exactpro CEO – The London Stock Exchance Group
- Yann L’Huillier: Group Chief Information Officer – Tradition

Transcript

1. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 1 Non-Functional Testing at London Stock Exchange 29 November 2016 Iosif Itkin, Exactpro CEO & Co-Founder London Stock Exchange Group Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 2460, +1 415 830 38 49 www.exactpro.com
2. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 2  London Stock Exchange Group  Exactpro - Open Access Software Testing  Testing Execution Venues  Efficient Load Generation  Latency Measurement  Non-Functional Testing Principles  Other Ways to Produce Load  Approach to Test Automation  Failover Testing Approach  Passive Software Testing  Research in Tools and Methods of Program Analysis Contents Contents
3. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 3 London Stock Exchange Group • A leading diversified international exchange and infrastructure group with assets across the entire exchange value chain • The Group can trace its history back to 1698 • Headquartered in the United Kingdom, with significant global operations, the Group employs approximately 3,500 people • Post trade and risk management services are a significant part of the Group’s business operations • Through FTSE Russell, the Group is a global leader in financial indexing, benchmarking and analytic services with approximately $10 trillion benchmarked to its indexes
4. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 4 Open Access Software Testing Exactpro is: • A specialist firm operating an open access model with focus on functional and operational testing of securities data distribution, trading systems, risk management, market surveillance and post-trade infrastructures. • A UK company with operations in the US and four QA & software development centres in Russia • Part of London Stock Exchange Group as of May 29, 2015 • Incorporated in 2009 with 10 people, our company has experienced significant growth as satisfied clients require more services; now employing over 420 specialists Clients Worldwide:
5. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 5 Open Access Software Testing Toronto New York Tysons Corner Montreal Portland Seattle Dubai Colombo Johannesburg Sydney Singapore Hong Kong Moscow Oslo Paris Milan Athens Vienna London Amsterdam A specialist firm operating an open access model with focus on functional and operational testing for global financial markets worldwide. A UK company with operations in the US and four QA & software development centres in Russia Part of LSEG Since May 29, 2015 Incorporated in 2009 with 10 people, our company has experienced significant growth as satisfied clients require more services; Now employing over 400 specialists
6. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 6 Types of Trading Systems Algo Trading HFT Brokerage and IDB Quality Assurance: test planning and management Process audit and test coverage analysis Intelligent Management of Large Data Sets Test data management Protocol level testing using FIX/FAST, SOAP, HTTP, ITCH, SWIFT, MQ, SQL, proprietary binary and text based data formats, etc. Automated regression testing Gathering requirements and test scenario creation (human, message & reporting interfaces) Creating and productizing state-of-the-art test harnesses Test automation Intelligent functional and exploratory testing Latency & capacity testing Market Infrastructures
7. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 7 Execution Venues Testing Our experience in market infrastructures testing embraces all components and operational processes. Unique tools have been used to test a wide variety of vendor platforms. A holistic integrated automated test solution for trading, clearing, settlement and other systems
8. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 8 Performance Testing Cost Structure System Under Test Production Connectivity Test Tools Infrastructure: ≈ 10 - 100 servers Infrastructure: thousands of servers Infrastructure: 1 - 4 servers
9. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 9 The Idea Behind Efficient Load Generation
10. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 10 The Idea Behind Efficient Load Generation
11. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 11 Latency Measurement
12. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 12 Latency Distribution
13. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 13 Question Everything Do Not Trust: • Test Tools • Average Metrics and Percentiles • That Load Shape is Right • System Requirements • Anything
14. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 14 Keep Calm Have No Fear: • Annihilate the System Under Test • Repeat the Same Tests Many Times • Load Levels that Exceed Presumed Production • Make no Assumptions About Extrapolating Test Results into More Scalable Environments
15. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 15 Be Confident Do Not Beg For: • Test Tools • Confirmation that PROD is Configured as TEST • Max Load Figure • Load Scenarios for Existing Production System • Constant Support
16. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 16 Two Load Generator Types 1. 2.  Simulate real trading participants  Adopt to changing market conditions  Check data consistency from member’s point of view
17. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 17 Achieving Diverse and Relevant Load 1. Random load “Pace Maker” 2. Variety of Passive Liquidity 3. “Aggressors” try to interact with the market to increase the number of test cases happening in the environment 4. Analyze the situation in retrospect. Make sure that what happened is correct. Tick checkboxes against tests that were actually executed Updated test library
18. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 18 Types of Non-Functional Tests
19. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 19 Failover Tests and Catching In-Flight Transactions
20. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 20 Use Passive Testing Tools to Collect Audit Trails
21. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 21 Questions & Answers • Competing conference papers go through a review by a number of leading software development and analysis experts and scientists • The best papers are recommended for their inclusion in Russia’s leading professional and scientific publications The Fourth TMPA Conference Find out more: www.tmpaconf.org Renowned experts in program engineering speaking on: • software test automation • static program analysis • verification • dynamic methods of program analysis • analysis of parallel and distributed systems • testing and analysis of high-availability systems • analysis of hardware and software systems • methods of building quality software • tools for software analysis, testing and verification
22. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 22 Questions & Answers https://ru.linkedin.com/in/iosifitkin The seventh EXTENT conference will take place in London, UK. Find out more: www.extentconf.com – LSEG Technology and Quality Assurance – Risk controls and FPGA – Market Surveillance systems – Blockchain and trading technology trends – Artificial intelligence