EXTENT-2016: Key Challenges and Lessons Learned from Testing a New Trading System

EXTENT-2016: Software Testing & Trading Technology Trends
22 June, 2016, 10 Paternoster Square, London

Key Challenges and Lessons Learned from Testing a New Trading System
Dr. Matthias Burghardt, Head of Business Development, Boerse Stuttgart

Would like to know more?
Visit our website: extentconf.com
Follow us:
https://www.linkedin.com/company/exactpro-systems-llc?trk=biz-companies-cym
https://twitter.com/exactpro
#extent2016
#exactpro

Transcript

1. Key Challenges and Lessons Learned from Testing a New Trading System The Xitaro Project Dr Matthias Burghardt Head of Business Development Boerse Stuttgart London, 22 June 2016
2. • About Boerse Stuttgart • Overview of the Xitaro Project • Ten Lessons Learned 2 Agenda © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System
3. 3 Where exactly is Stuttgart? About Boerse Stuttgart We are here! Stuttgart is the capital and largest city of the state of Baden-Württemberg in southwest Germany. © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System
4. 4 What is Stuttgart known for? About Boerse Stuttgart © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System
5. 5 Boerse Stuttgart is Europe‘s leading market for securitised derivatives About Boerse Stuttgart Source: Respective Stock Exchange. Figures based on volume of securitised derivative orders. Last updated: April 2016. 33,6% 9,7% 16,2% 9,4% 23,7% 1,8% 1,8% 0,4% Stuttgart incl. Nordic Derivatives Exchange Börse Frankfurt Zertifikate AG SIX Swiss Exchange Euronext Borsa Italiana BME NASDAQ Nordic & Baltic Others © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System
6. 1,500,000 500,000 1,000,000 0 2,382 20,305 1995 1,388,600 2015 1,292,471 2010 2005 535,794 70,010 2000 6 Currently more than 1.4m instruments are listed at Boerse Stuttgart This number is driven by the large amount of securitised derivatives About Boerse Stuttgart # of instruments of other asset classes listed at Boerse Stuttgart Debt instruments: 12,186 Equities: 7,973 Investment fund units: 3,410 Number of securitised derivatives listed at Boerse Stuttgart © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System
7. 7 Boerse Stuttgart ranks among Europe’s Top 10 Stock Exchanges in terms of turnover About Boerse Stuttgart Source: Federation of European Securities Exchanges, London Stock Exchange, Borsa Italiana Basis: Order book turnover for whole of 2015 in all asset classes, electronic order book orderbook trades Rank Stock Exchange Turnover (2015) in billion €* 1 London Stock Exchange Group 2,871.5 2 Euronext 2,017.0 3 Deutsche Börse 1,628.7 4 BME (Spanish Exchanges) 1,020.1 5 SIX Swiss Exchange 999.1 6 NASDAQ Nordic & Baltic 711.1 7 Borsa Istanbul 430.1 8 Oslo Børs 135.8 9 Boerse Stuttgart Group 106.6 10 Warsaw Stock Exchange 45.6 11 CEESEG - Vienna 29.6 12 Irish Stock Exchange 19.3 13 Athens Stock Exchange 17.1 14 CEESEG – Budapest 7.2 © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System
8. • About Boerse Stuttgart • Overview of the Xitaro Project • Ten Lessons Learned 8 Agenda © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System
9. 9 The Xitaro Project – replacing Boerse Stuttgart’s exchange system in a complex environment to gain flexibility and independence About the Xitaro Project – Mission Xitaro Boerse Stuttgart‘s new trading system Powered by Nasdaq Genium INET XONTRO • Almost 25 years old • Shared host based system • High degree of dependency Flexible and independent Competitive costs State-of-the-Art Faster innovations Future proof Project Xitaro © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System
10. 10 Five dedicated work streams took the challenge About the Xitaro Project – Project Organisation Joint work stream with Nasdaq BSG work stream (Joint) Steering Committee Risk-Controlling • Management of all stakeholder activities, e.g. trading participants, employees, boards, etc. • Specification of all Xitaro functionalities • interfaces, internal systems, network setup • Solving technical problems and issues • Preparation for the operational phase • setup of processes, resources, infrastructure • Cutover Preparation • Member onboarding • Ensure software and system quality by intensive testing (automated and manual) IT Security & Data protection PMO & Quality Assurance Internal Audit Project Management Legal Advisory Stakeholder Management Functional Design Techn. Design & Customizing Setup Operations & Cutover Management Test and Acceptance © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System
11. 4 % 6 % 33 % 36 % 21 % 11 Accounting for one third of the total effort, Test & Acceptance was Xitaro‘s largest work stream About the Xitaro Project – Test Effort Joint work stream with Nasdaq BSG work stream Stakeholder Management Functional Design Techn. Design & Customizing Setup Operations & Cutover Management Test and Acceptance © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System
12. 1: System Integration Test After a 2 years implementation project, Xitaro went live in September 2015 About the Xitaro Project – Time Line Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 SIT1 Detailed Design Implementation Q3 Member Tests Operations Beta Tests Project Xitaro 2013 2014 2015 Dress Rehearsals Hardware / Network Setup for SIT and UAT Hardware / Network Setup For Production Member Onboarding Member Certification Cutover Concept Test Preparation Regression Tests User Acceptance Tests © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System 12
13. • About Boerse Stuttgart • Overview of the Xitaro Project • Ten Lessons Learned 13 Agenda © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System
14. 14 Key challenges in testing cover a wide range of areas Key Challenges Test automation Outsourcing Regression Testing Test Case Design Beta Testing Early Member Testing Test environments Defect Management Non-functional Testing © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System Acceptance Test Preparation
15. Testing / Contingency 15 You cannot separate functional design and test case design Lesson #1 – Test Case Design Functional Descriptions #1 Functional Descriptions # 2 Functional Descriptions #3 Functional Descriptions #4 Test Case Design Test Case Design #1 Test Case Design #2 Test Case Design #3 Test Case Design #4 Functional Descriptions Testing Testing Test case design requires close alignment between functional and test experts © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System
16. Day 3 Marc Tom Andi Env 1 TC 1 TC 3 Env 2 TC 2 TC 5 Env 3 TC 7 TC 6 … Day 2 Marc Tom Andi Env 1 TC 1 TC 3 Env 2 TC 2 TC 5 Env 3 TC 7 TC 6 … Day 2 16 Take one step at a time – from acceptance test specifications to the detailed acceptance test plan Lesson #2 – Acceptance Test Preparation Acceptance Test Specifications Acceptance Test Procedures Acceptance Test Plan Acceptance Test Plan Detailed Acceptance Test Draft Review Update Approve Day 1 Marc Tom Andi Env 1 TC 1 TC 3 Env 2 TC 2 TC 5 Env 3 TC 7 TC 6 … ATS: •TC1: •TC2: •TC3: •TC4 … ATP: •TC1 • Step 1 • Step 2 • Step 3 • … AT Plan: •Day 1 • TC 1 • TC 2 • TC 3 • … ATP: •TC2 • Step 1 • Step 2 • Step 3 • … ATP: •TC3 • Step 1 • Step 2 • Step 3 • … ATP: •TC3 • Step 1 • Step 2 • Step 3 • … Review Cycle helps aligning customer and supplier expectations 1 2 3 4 © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System
17. 17 Choose wisely which functionality outsiders could test Outsourcing may prove difficult when testing complex functionality Lesson #3 - Outsourcing 36% 64% 43% 57% 43% 57% 44% 56% 46% 54% 52% 48% 70% 30% 91% 9% 100% 100% 100% Functionality 1 100% 93% 7% Functionality 15 17% 83% 22% 78% Functional Experts Test Experts decreasing complexity Test cases executed by Functionalities with higher complexity e.g. Trading Halts Functionalities with lower complexity e.g. GUI User Configuration Overall percentage of executed test cases 53% 47% Test Experts Functional Experts © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System
18. 18 Early beta deliveries are good but you need to test them properly Lesson #4 – Beta Testing 1: User Acceptance Test Q1 2014 Q2 2014 Q3 2014 Q4 2014 Partly included functionalities Completely included functionalities 21.02. 25.03. 09.05. 16.06. 08.09. 13.10. Beta 1.1 Beta 1.2 Beta 1.3 Beta 2.1 Beta 2.2 Beta 2.3 UAT1 5 2 12 2 14 8 7 27 1 61 76 Test Phases Number of functionalities delivered Six Beta Releases were delivered during the project covering 76 functions of varying complexity © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System What we should have done Beta test effort What we actually did
19. • Creating automated test cases is more time-consuming than manual test cases • Not easy to combine several interfaces and reports into one test scenario • Test scripts can have defects and require debugging • Lack of flexibility when functional requirements change • Quick and efficient test case execution • Good for interface tests • Allows for cost-effective frequent test execution during regression tests • More interesting for testers 19 Automating tests frees up valuable experts – if you do it right Lesson #5 – Test automation Used to regression test 8 Releases before Go-Live + - Test automation during the project – the pros & cons + + + - - - Project Xitaro test cases Q1 2015 Q2 2015 Q3 2015 AT 1 1,000 4,000 AT 2 1,700 3,900 Regression Tests 1,800 2,500 Automated Test Cases Manual Test Cases © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System
20. 20 Create different regression test sets for different purposes Lesson #6 – Regression Testing 1 Evaluation 6,500 Test Cases Functional Frequency Frequency of using the functionality in operations Damage Potential Potential damage / impact in the case of defects in the functionality Defect Frequency General frequency of defect occurrence in this functionality / test case 2,200 Test Cases 4,300 Test Cases Possibility for Automation? 2 Selection 3 Categorisation Small regression test set 1,800 Medium regression test set 2,300 Large regression test set 4,300 © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System
21. Start early with member testing to avoid surprises Lesson #7 – Early Member Testing Q4 2014 Q1 2015 Q2 2015 Q3 2015 Early Member Tests Dress Rehearsals Member Onboarding Member Readiness Checks Simulation Period • Pilot & Early Member connection • Participant setup • Contract management • Free and assisted testing • End-to-end Tests • Non-functional tests • Connection and interface tests • Tests of participant configuration • Debugging activities, both in Xitaro and customer systems • 4 Waves of assisted tests to validate Go-Live readiness of all connected members • 4 weekends of Go-Live rehearsals • One participation mandatory © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System 21 Xitaro Member testing activities started about 12 months before Go-Live – early commitment was rewarded Defect reports by members are important to help your test team detect all errors Structured dress rehearsals avoid unpleasant surprises at Go-Live Early member connectivity will give you certainty and remove risks of delays
22. 22 Separating test environments by purpose allows for flexibility during the project and afterwards Lesson #8 – Test Environments Non-Production Systems Xitaro Simulation (production-like hardware) Xitaro External Test (virtualised environment) Xitaro Internal Test (virtualised environment) Xitaro Configuration (virtualised environment) • User Acceptance Test • Performance testing and operational acceptance test • Member testing activities • Automated tests • Regression tests • Planned to be made available to members • Internal test environment for internal software development and integration tests • Temporary test environment for user configuration before Go-Live © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System
23. 23 Plan your non-functional tests from end to end and include all relevant business day activities Lesson #9 – Non-Functional Tests 1 How we planned it SLA Non-Functional Area # Test Cases Security 2 High availability and disaster recovery 79 Performance 34 Audit 3 Operations 20 Integration with external systems 9 2 How we should have planned it • Analyse business day activities according to daily cycle jointly with supplier • Detect all business relevant performance parameters • Plan joint activities for the System Integration Test © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System 3 What we did anyway Xitaro
24. Close defect management and tracking make sure the project survives the User Acceptance Test Lesson #10 – Defect Management 527 149 0 200 400 600 SIT Beta 2.3 Resolved incidents +378 AT1 480 208 0 100 200 300 400 500 Created incidents +272 AT1 SIT Beta 2.3 0 20 40 60 80 100 120 140 160 180 200 220 240 1.2. 1.3. Trend reported vs. resolved incidents ~13.5 incidents resolved per day ~9.7 incidents created per day Reported incidents Resolved incidents Trend reported vs. resolved incidents 15.1. AT2 release Average net reduction of ~4 incidents per day during AT1 ~40–50 open incidents not yet retested End AT1 Average reduction of ~8 incidents estimated after AT1 Resolved but not retested yet © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System 24
25. Questions & Answers © Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System 25
26. Thank you for your attention! Dr. Matthias Burghardt Head of Business Development Tel.: +49 711 222 985-593 Fax.: +49 711 222 985-91-593 Mail: matthias.burghardt@boerse-stuttgart.de Börse Stuttgart Börsenstraße 4 D-70174 Stuttgart