Trading Clearing Systems Test Automation

Transcript

1. Trading & Clearing Systems Test Automation Exactpro Systems 21st December 2013
2. STEP 1: Test Server Functionality by FIX or other standard protocol STEP 2: ‘GUI Bypass’ Testing STEP 5: Fully Automated GUI Testing STEP 4: Semi Automated GUI Testing 2 Approach to Test Automation Recommended by Exactpro Given the relative difficulty of achieving efficiencies through automation of GUI testing, Exactpro has evolved a progressive strategy towards test automation that we recommend be followed: • This builds upon the other levels or automation, now it’s about testing the GUI itself not the underlying system • Wherever possible exercise functionality by message injection into FIX or other standard interfaces first - it’s far easier to automate & debug • Connect to the GUI using the best technical interface possible – the technology available makes a big difference in the payback on the automation investment • Partial automation can also be effective, e.g. enter data with automation script and verify manually on screen or script checks database • If the system under test has documented interfaces between the GUI and the server emulate the GUI with a test harness to exercise more of the server functionality that is available through external interfaces STEP 6: ‘Big Button’ Framework • Progressively link all of the automated tests into a framework that allows highly productive control over test execution & reporting across of the entire test set STEP 3: Connect to the GUI GUI Object API Windows Controls Screen Scraping Web Tools
3. 3 STEP 1: Test Server Functionality by FIX or other standard protocol Sample of Test Automation Achieved Using the Approach
4. Exactpro’s Bespoke Test Automation Tools Suite 4 ClearTH: • Post-Trade testing tool • Verifies each stage of the DLC • Integrated schedule • Automated matrices • Can create multiple days test scenarios • Concurrent multiple tests • Integrated simulators • SWIFT ISO protocol support MiniRobots: • Executes multithreaded java code • Complexity of test algorithms is defined by the test developer • Supports multiple client fix connections, order entry and market data via FIX • Can use GUI to iterate through sent and received messages Dolphin: • Model-based testing of market surveillance systems Shsha: • Post-transactional tool • Analyzes clients' activity and forecasts system response • Parses and displays logs in a user-friendly way • Parses messages and then puts each to a data base table where each column corresponds to each message field • Allows making summarized reports, etc • Easy to understand GUI Load Injector: • Simulates multiple client connections with a specified load shape for each connection or a group of connections • Up to 75K messages / second from a single CPU core • Measures latencies in microsecond range • Performance test reports Sailfish: • Can test Order Entry, Market Data and Post Trade connections in one test scenario • Each test scenario is independent • Allows running test scripts in any sequence • Simulation of multiple user connections • Server simulators • All messages are stored into a data base • Generates test reports
5. Explanation of Sailfish 5 Item Description Exactpro Test Strategy Step(s) STEP 1: Test Server Functionality by FIX or other standard protocol; STEP 2: ‘GUI Bypass’ Testing; STEP 4: Semi Automated GUI Testing Testing Type Active Real-Time Target SUT Trading Platforms and Market Data Delivery Systems SUT Interface Back-end (typically connected to message gateways / APIs, and DBs); GUI Testing Capabilities supported via plug-ins to other tools (e.g., Selenium) SUT Interaction Method Message injection and capture for testing of real-time low-latency bi-directional message flows; DB queries for data verification Protocols Extant plug-ins for Industry-standard (FIX and dialects, FAST, SWIFT, ITCH, HTTP, SOAP, etc.) and Proprietary (MIT, SAIL, HSVF, RTF, RV, Reuters, Fidessa OA, Quant House, etc.) protocols. New plug-ins for additional protocols developed by request (codecs are shared between Sailfish and Shsha) Test Scripts Human-readable CSV files; scripts generated manually by test analysts or automatically by test script generator using results of passive testing performed by other tool (e.g., Shsha) Test Management, Execution and Reporting Integrated (Web front-end), allows for multiple simultaneous heterogeneous connections, consecutive execution of multiple planned scripts, test results summary and detailed test reports. REST API supports remote control of Sailfish instances. Optional Big Button framework supported Platform requirements Low footprint cross-platform application, MySQL or other RDBMS Primary Competitor FIX Testing Tools
6. Explanation of Shsha 6 Item Description Exactpro Test Strategy Step(s) STEP 1: Test Server Functionality by FIX or other standard protocol; STEP 2: ‘GUI Bypass’ Testing; STEP 4: Semi Automated GUI Testing Testing Type Passive Real-Time/Batch Target SUT Trading Platforms and Market Data Delivery Systems SUT Interface Back-end (typically connected to message gateways / APIs, and DBs); GUI Testing Capabilities not supported SUT Interaction Method Inputs and outputs monitored by means of message capture and log parsing to analyze client activity and forecast system response; DB queries for data verification; files transfer, upload, export and comparison. Captured messages can be viewed and analyzed in real-time or post-factum Protocols Extant plug-ins for Industry-standard (FIX and dialects, FAST, SWIFT, ITCH, HTTP, SOAP, etc.) and Proprietary (MIT, SAIL, HSVF, RTF, RV, Reuters, Fidessa OA, Quant House, etc.) protocols. New plug-ins for additional protocols developed by request (codecs are shared between Sailfish and Shsha) Test Scripts Certification tests and data reconciliation may be performed by using ordinary SQL queries. Test message traffic generated in real-time or replayed from log files by other tool (e.g., Sailfish) Test Management, Execution and Reporting Integrated (Desktop front-end), allows for multiple simultaneous heterogeneous connections, consecutive execution of multiple planned scripts, test results summary and detailed test reports. Optional Big Button framework supported Platform requirements Low footprint cross-platform application, MySQL Primary Competitor FIX Onboarding Tools
7. Explanation of ClearTH 7 Item Description Exactpro Test Strategy Step(s) STEP 1: Test Server Functionality by FIX or other standard protocol; STEP 2: ‘GUI Bypass’ Testing; STEP 4: Semi Automated GUI Testing Testing Type Active Batch Target SUT Post-Trade Platforms and Middle Office SUT Interface Back-end (typically connected to message gateways / APIs, and DBs); GUI Testing Capabilities supported via plug-ins to other tools (e.g., Selenium) SUT Interaction Method Schedule-driven batch processing (often spanning several business days). Internal scheduler aligned with the SUT scheduler. DB queries for data verification Protocols Extant plug-ins for Industry-standard (FIX and dialects, FAST, SWIFT, ITCH, HTTP, SOAP, FTP, etc.) and proprietary protocols. New plug-ins for additional protocols developed by request Test Scripts Human-readable CSV files; scripts generated manually by test analysts or automatically by test script generator using transactional and reference databases Test Management, Execution and Reporting Integrated (Web front-end), allows for multiple simultaneous heterogeneous connections, concurrent execution of multiple planned scripts linked to particular global steps, test results summary and detailed test reports. Based on Big Button framework principles Platform requirements Low footprint cross-platform application, MySQL or other RDBMS Primary Competitor Custom BPM solutions
8. Explanation of MiniRobots 8 Item Description Exactpro Test Strategy Step(s) STEP 1: Test Server Functionality by FIX or other standard protocol; STEP 2: ‘GUI Bypass’ Testing; STEP 4: Semi Automated GUI Testing Testing Type Active Multi-Participants (applicable for testing at the confluence of functional and non-functional testing) Target SUT Trading Platforms and Market Data Delivery Systems SUT Interface Back-end (typically connected to message gateways / APIs); GUI Testing Capabilities not supported SUT Interaction Method Message injection and capture to emulate multiple participants’ activity in electronic markets (essential when there is a need to reproduce complex scenarios that can be created by trading algorithms) Protocols Extant plug-ins for Industry-standard (FIX and dialects, etc.) and proprietary protocols. New plug-ins for additional protocols developed by request Test Scripts Multi-threaded Java code specifying different liquidity profiles Test Management, Execution and Reporting Integrated (Web front-end), allows for multiple simultaneous heterogeneous connections, concurrent emulation of multiple participants, detailed test reports. Optional Big Button framework supported Platform requirements Low footprint cross-platform application, MySQL or other RDBMS Primary Competitor Custom market and algo trading simulation solutions
9. Test Automation Essential Ingredients 9 Essential Ingredients Bespoke automation tools built up as an investment over several projects, in combination with open source tools Highest calibre development staff on the team Control of test environment back-end by QA and the ability to emulate external systems in order to set the required pre-conditions and increase test automation coverage Pragmatism and determination Experience Principles Low Start Passive & Active Testing Codecs, Drivers and Stubs The “Big Button” Framework High Volume Automated Testing (HiVAT) Type of Tests Transactional Scheduling Reconciliation Participant
10. 10 Design of the Automation Will Embody These Principles Principle Description 1. Low Start • Don’t wait for the full comprehensive test harness to be available; use simple tools as soon as they are ready • Start with getting tools to execute with partial validation & progressively add more validations • Capture logs and executed files. Even semi-manual test execution provides better evidence and serves as the foundation for future scripting 2. Passive & Active Testing • Passive testing is a good way to collect test evidence from other tests and can be used to perform client certification • Messages stored during passive testing can be used to generate active test cases and repeat previously executed activities 3. Codecs, Drivers and Stubs • Codecs are test code capable of parsing a particular protocol or file format; using these allows increasing testing depth, localizing defects and validating the components earlier • A Driver is a test program that can initiate the injection of messages into the system • A Stub is a test program that simulates an absent component called by the system under test • Drivers and Stubs both rely on Codecs • A Test Harness is a set of drivers and stubs for every interface link in the system • An automated test library relies on the Test Harness to execute the test scenarios 4. The “Big Button” Framework • “Big Button” organizes large automated test libraries, so that they can be executed in an unattended mode • It is necessary to carefully design reference/test data sets and plan operational schedule of the test environment • Reduces effort required for regression test execution, but also allows having a better test library structure • The Big Button approach requires more stability from the test tools and test scripts 5. High Volume Automated Testing (HiVAT) • Like other complex distributed real-time systems, Trading and Clearing platforms have intermittent defects that manifest themselves only if load is applied • They can be attributed to race conditions or to rare internal integrity failures • Such defects might not be spotted in the course of ordinary functional test execution due to their low statistical probability under low volumes • HiVAT techniques are based on automated test case generation
11. 11 Automated Tests can Address Four Areas Areas Description Sample of Exactpro Tool 1. Transactional (Active Real-Time) • Processes incoming requests in real-time • Set up controlled transaction data set • Execute volume of independent test rapidly • Independent test cases and is easily supportable 2. Scheduling (Active Batch) • Tests aggregation and batches functionality • Execute scenarios lasting multiple business days • Execute many test scenarios in parallel • Internal scheduler aligned with scheduler of the system under the test 3. Reconciliation (Passive Real-Time / Batch) • Verify system consistency by monitoring inputs and outputs • Agnostic to test input source • Complements active test approaches 4. Participants (Active Multi-Participants) • Emulation of participants activity in electronic markets • Reproduce complex scenarios • Testing at the confluence of functional and non functional testing • Verify system from participant's interfaces