Trading Clearing Systems Test Automation

Transcript

1. Trading & Clearing Systems
   Test Automation
   Exactpro Systems
   21st December 2013
   

   View Slide

2. STEP 1: Test Server Functionality by
   FIX or other standard protocol
   STEP 2: ‘GUI Bypass’ Testing
   STEP 5: Fully Automated GUI Testing
   STEP 4: Semi Automated GUI Testing
   2
   Approach to Test Automation Recommended by Exactpro
   Given the relative difficulty of achieving efficiencies through automation of GUI testing, Exactpro has evolved a progressive strategy towards test automation that we recommend be
   followed:
   • This builds upon the other levels or automation, now it’s about testing the
   GUI itself not the underlying system
   • Wherever possible exercise functionality by message injection into FIX or
   other standard interfaces first - it’s far easier to automate & debug
   • Connect to the GUI using the best technical interface possible – the
   technology available makes a big difference in the payback on the
   automation investment
   • Partial automation can also be effective, e.g. enter data with automation
   script and verify manually on screen or script checks database
   • If the system under test has documented interfaces between the GUI and
   the server emulate the GUI with a test harness to exercise more of the
   server functionality that is available through external interfaces
   STEP 6: ‘Big Button’ Framework
   • Progressively link all of the automated tests into a framework that allows
   highly productive control over test execution & reporting across of the
   entire test set
   STEP 3: Connect
   to the GUI
   GUI Object API
   Windows Controls
   Screen Scraping
   Web Tools
   

   View Slide

3. 3
   STEP 1: Test Server Functionality by
   FIX or other standard protocol
   Sample of Test Automation Achieved Using the Approach
   

   View Slide

4. Exactpro’s Bespoke Test Automation Tools Suite
   4
   ClearTH:
   • Post-Trade testing tool
   • Verifies each stage of the
   DLC
   • Integrated schedule
   • Automated matrices
   • Can create multiple days
   test scenarios
   • Concurrent multiple tests
   • Integrated simulators
   • SWIFT ISO protocol support
   MiniRobots:
   • Executes multithreaded java
   code
   • Complexity of test algorithms is
   defined by the test developer
   • Supports multiple client fix
   connections, order entry and
   market data via FIX
   • Can use GUI to iterate through
   sent and received messages
   Dolphin:
   • Model-based testing
   of market surveillance
   systems
   Shsha:
   • Post-transactional tool
   • Analyzes clients' activity and forecasts system response
   • Parses and displays logs in a user-friendly way
   • Parses messages and then puts each to a data base
   table where each column corresponds to each message
   field
   • Allows making summarized reports, etc
   • Easy to understand GUI
   Load Injector:
   • Simulates multiple client
   connections with a
   specified load shape for
   each connection or a group
   of connections
   • Up to 75K messages /
   second from a single CPU
   core
   • Measures latencies in
   microsecond range
   • Performance test reports
   Sailfish:
   • Can test Order Entry, Market Data and Post
   Trade connections in one test scenario
   • Each test scenario is independent
   • Allows running test scripts in any sequence
   • Simulation of multiple user connections
   • Server simulators
   • All messages are stored into a data base
   • Generates test reports
   

   View Slide

5. Explanation of Sailfish
   5
   Item Description
   Exactpro Test Strategy
   Step(s)
   STEP 1: Test Server Functionality by FIX or other standard protocol; STEP 2: ‘GUI Bypass’ Testing; STEP
   4: Semi Automated GUI Testing
   Testing Type Active Real-Time
   Target SUT Trading Platforms and Market Data Delivery Systems
   SUT Interface Back-end (typically connected to message gateways / APIs, and DBs); GUI Testing Capabilities
   supported via plug-ins to other tools (e.g., Selenium)
   SUT Interaction
   Method
   Message injection and capture for testing of real-time low-latency bi-directional message flows; DB
   queries for data verification
   Protocols Extant plug-ins for Industry-standard (FIX and dialects, FAST, SWIFT, ITCH, HTTP, SOAP, etc.) and
   Proprietary (MIT, SAIL, HSVF, RTF, RV, Reuters, Fidessa OA, Quant House, etc.) protocols. New plug-ins
   for additional protocols developed by request (codecs are shared between Sailfish and Shsha)
   Test Scripts Human-readable CSV files; scripts generated manually by test analysts or automatically by test script
   generator using results of passive testing performed by other tool (e.g., Shsha)
   Test Management,
   Execution and
   Reporting
   Integrated (Web front-end), allows for multiple simultaneous heterogeneous connections,
   consecutive execution of multiple planned scripts, test results summary and detailed test reports.
   REST API supports remote control of Sailfish instances. Optional Big Button framework supported
   Platform requirements Low footprint cross-platform application, MySQL or other RDBMS
   Primary Competitor FIX Testing Tools
   

   View Slide

6. Explanation of Shsha
   6
   Item Description
   Exactpro Test Strategy
   Step(s)
   STEP 1: Test Server Functionality by FIX or other standard protocol; STEP 2: ‘GUI Bypass’ Testing; STEP
   4: Semi Automated GUI Testing
   Testing Type Passive Real-Time/Batch
   Target SUT Trading Platforms and Market Data Delivery Systems
   SUT Interface Back-end (typically connected to message gateways / APIs, and DBs); GUI Testing Capabilities not
   supported
   SUT Interaction
   Method
   Inputs and outputs monitored by means of message capture and log parsing to analyze client activity
   and forecast system response; DB queries for data verification; files transfer, upload, export and
   comparison. Captured messages can be viewed and analyzed in real-time or post-factum
   Protocols Extant plug-ins for Industry-standard (FIX and dialects, FAST, SWIFT, ITCH, HTTP, SOAP, etc.) and
   Proprietary (MIT, SAIL, HSVF, RTF, RV, Reuters, Fidessa OA, Quant House, etc.) protocols. New plug-ins
   for additional protocols developed by request (codecs are shared between Sailfish and Shsha)
   Test Scripts Certification tests and data reconciliation may be performed by using ordinary SQL queries. Test
   message traffic generated in real-time or replayed from log files by other tool (e.g., Sailfish)
   Test Management,
   Execution and
   Reporting
   Integrated (Desktop front-end), allows for multiple simultaneous heterogeneous connections,
   consecutive execution of multiple planned scripts, test results summary and detailed test reports.
   Optional Big Button framework supported
   Platform requirements Low footprint cross-platform application, MySQL
   Primary Competitor FIX Onboarding Tools
   

   View Slide

7. Explanation of ClearTH
   7
   Item Description
   Exactpro Test Strategy
   Step(s)
   STEP 1: Test Server Functionality by FIX or other standard protocol; STEP 2: ‘GUI Bypass’ Testing; STEP
   4: Semi Automated GUI Testing
   Testing Type Active Batch
   Target SUT Post-Trade Platforms and Middle Office
   SUT Interface Back-end (typically connected to message gateways / APIs, and DBs); GUI Testing Capabilities
   supported via plug-ins to other tools (e.g., Selenium)
   SUT Interaction
   Method
   Schedule-driven batch processing (often spanning several business days). Internal scheduler aligned
   with the SUT scheduler. DB queries for data verification
   Protocols Extant plug-ins for Industry-standard (FIX and dialects, FAST, SWIFT, ITCH, HTTP, SOAP, FTP, etc.) and
   proprietary protocols. New plug-ins for additional protocols developed by request
   Test Scripts Human-readable CSV files; scripts generated manually by test analysts or automatically by test script
   generator using transactional and reference databases
   Test Management,
   Execution and
   Reporting
   Integrated (Web front-end), allows for multiple simultaneous heterogeneous connections, concurrent
   execution of multiple planned scripts linked to particular global steps, test results summary and
   detailed test reports. Based on Big Button framework principles
   Platform requirements Low footprint cross-platform application, MySQL or other RDBMS
   Primary Competitor Custom BPM solutions
   

   View Slide

8. Explanation of MiniRobots
   8
   Item Description
   Exactpro Test Strategy
   Step(s)
   STEP 1: Test Server Functionality by FIX or other standard protocol; STEP 2: ‘GUI Bypass’ Testing; STEP
   4: Semi Automated GUI Testing
   Testing Type Active Multi-Participants (applicable for testing at the confluence of functional and non-functional
   testing)
   Target SUT Trading Platforms and Market Data Delivery Systems
   SUT Interface Back-end (typically connected to message gateways / APIs); GUI Testing Capabilities not supported
   SUT Interaction
   Method
   Message injection and capture to emulate multiple participants’ activity in electronic markets
   (essential when there is a need to reproduce complex scenarios that can be created by trading
   algorithms)
   Protocols Extant plug-ins for Industry-standard (FIX and dialects, etc.) and proprietary protocols. New plug-ins
   for additional protocols developed by request
   Test Scripts Multi-threaded Java code specifying different liquidity profiles
   Test Management,
   Execution and
   Reporting
   Integrated (Web front-end), allows for multiple simultaneous heterogeneous connections, concurrent
   emulation of multiple participants, detailed test reports. Optional Big Button framework supported
   Platform requirements Low footprint cross-platform application, MySQL or other RDBMS
   Primary Competitor Custom market and algo trading simulation solutions
   

   View Slide

9. Test Automation Essential Ingredients
   9
   Essential Ingredients
   Bespoke automation tools built up as an investment
   over several projects, in combination with open
   source tools
   Highest calibre development staff on the team
   Control of test environment back-end by QA and the
   ability to emulate external systems in order to set
   the required pre-conditions and increase test
   automation coverage
   Pragmatism and determination
   Experience
   Principles
   Low Start
   Passive & Active
   Testing
   Codecs, Drivers
   and Stubs
   The “Big Button”
   Framework
   High Volume
   Automated
   Testing (HiVAT)
   Type of Tests
   Transactional
   Scheduling
   Reconciliation
   Participant
   

   View Slide

10. 10
    Design of the Automation Will Embody These Principles
    Principle Description
    1. Low Start • Don’t wait for the full comprehensive test harness to be available; use simple tools as soon as they are ready
    • Start with getting tools to execute with partial validation & progressively add more validations
    • Capture logs and executed files. Even semi-manual test execution provides better evidence and serves as the
    foundation for future scripting
    2. Passive &
    Active Testing
    • Passive testing is a good way to collect test evidence from other tests and can be used to perform client
    certification
    • Messages stored during passive testing can be used to generate active test cases and repeat previously executed
    activities
    3. Codecs,
    Drivers and
    Stubs
    • Codecs are test code capable of parsing a particular protocol or file format; using these allows increasing testing
    depth, localizing defects and validating the components earlier
    • A Driver is a test program that can initiate the injection of messages into the system
    • A Stub is a test program that simulates an absent component called by the system under test
    • Drivers and Stubs both rely on Codecs
    • A Test Harness is a set of drivers and stubs for every interface link in the system
    • An automated test library relies on the Test Harness to execute the test scenarios
    4. The “Big
    Button”
    Framework
    • “Big Button” organizes large automated test libraries, so that they can be executed in an unattended mode
    • It is necessary to carefully design reference/test data sets and plan operational schedule of the test environment
    • Reduces effort required for regression test execution, but also allows having a better test library structure
    • The Big Button approach requires more stability from the test tools and test scripts
    5. High
    Volume
    Automated
    Testing (HiVAT)
    • Like other complex distributed real-time systems, Trading and Clearing platforms have intermittent defects that
    manifest themselves only if load is applied
    • They can be attributed to race conditions or to rare internal integrity failures
    • Such defects might not be spotted in the course of ordinary functional test execution due to their low statistical
    probability under low volumes
    • HiVAT techniques are based on automated test case generation
    

    View Slide

11. 11
    Automated Tests can Address Four Areas
    Areas Description Sample of Exactpro Tool
    1. Transactional
    (Active Real-Time)
    • Processes incoming requests in real-time
    • Set up controlled transaction data set
    • Execute volume of independent test rapidly
    • Independent test cases and is easily supportable
    2. Scheduling
    (Active Batch)
    • Tests aggregation and batches functionality
    • Execute scenarios lasting multiple business days
    • Execute many test scenarios in parallel
    • Internal scheduler aligned with scheduler of the system
    under the test
    3. Reconciliation
    (Passive Real-Time
    / Batch)
    • Verify system consistency by monitoring inputs and
    outputs
    • Agnostic to test input source
    • Complements active test approaches
    4. Participants
    (Active
    Multi-Participants)
    • Emulation of participants activity in electronic markets
    • Reproduce complex scenarios
    • Testing at the confluence of functional and non
    functional testing
    • Verify system from participant's interfaces
    

    View Slide