Big Button 2.0 Big Button 2.0 Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 2460, +1 415 830 38 49 www.exactpro.com A new approach to automated regression A new approach to automated regression Alexey Zverev, CEO Exactpro Alexey Zverev, CEO Exactpro 11 November 2015 11 November 2015

2 Goal: to execute a test regression and make it: •as extensive as possible (A) •as efficient as possible (B) How do people usually achieve that? •They develop software which performs the tests and ensure that it requires an adequate amount of maintenance when the software under test changes Test Automation Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com

3 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com Test Automation Classics 1. Design Test Library 2. Develop Test Harness • Script execution framework • GUI injectors • DPI injectors • Upstream / Downstream systems stubs 3. Develop Test Case 4. Develop Many Test Cases 5. Team Of QA Engineers Runs Regression Login A Login B A send Order (…) B Send Order (…) A verify Ask (…) B verify Trade (…) A verify Trade (…) Login A Login B A send Order (…) B Send Order (…) A verify Ask (…) B verify Trade (…) A verify Trade (…)

4 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com The Job is not finished The next task is to make test execution more efficient: •We need a “Big Button”, i.e. software allowing executing a large number of automated test scripts with minimum human attendance •When successfully implemented it can reduce up to 80% effort (in man days) spent on regression

5 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com Big Button 1.0 concept 1. We have many automated test cases 2. Prepare multiple test environment 3. Prepare reference data 4. Redesign test library multiple threads 5. Develop orchestrating test harness Login A Login B A send Order (…) B Send Order (…) A verify Ask (…) B verify Trade (…) A verify Trade (…) 6. Run regression - SHELL - CRON - ... Login A Login B Login C

6 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com Limitations of Big Button 1.0 • Complete control over test environments is required • Automated GUI tests are not stable • Dependent test cases are not stable • Independent test cases require overhead • A lot of copy/paste code • Executing tests consecutively, one by one, takes significant time • Big Button conditions are not similar to production (discussed later)

7 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com Alternative approach - “Load with Trades” • This was a success • We have learned that it is called HiVAT (High volume automated testing) • Unfortunately it is still not a regression

8 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com Meanwhile - Robots! • The Trading Industry lives and breathes High Transaction Volumes • High Volume is created by Trading Algos and we have created a test tool called Minirobots to simulate that • The tool good for simulation of production conditions • Classical test scenarios are still not validated

9 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com Big Button 2.0 • A set of Minirobots Algorithms allowing to perform a complete test regression of a trading system within a few hours • All Robots work concurrently against the test environment • Robots are independent from each other, and their actions are based on current conditions on the order books • Robots use an abstract Test Library and their main goal is to cover all test cases within the library

10 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com How it will work 1. Random load “Pace Maker” 2. Variety of Passive Liquidity 3. “Agressors” try to interact with the market to increase number of test cases happening in the environment 4. Analyze situation in retrospective Make sure that what happened is correct Tick checkboxes against test that was actually executed Updated test library

11 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com Example 1. Read book 2. Search for appropriate test case 3. Try to test the scenario 4. Wait for full trade or auto cancel 5. Iterate through actual event orders: 6. Conclusion •We hit only 2 orders •Another robot cancelled 3rd order •Ticking box against this test case IOC hits 3 passive orders at exact price PFILL PFILL CANCEL ASK

12 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com Discussion + It can work really fast + Maintenance is limited (with adequate robot design) + A challenging and motivating task for QA - May be too complex to implement - It is not clear how to incorporate DLC related scenarios and manual tests

13 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com Q&A Thank You!