Trading Systems: Testing at the Confluence of FT and NFT

EXTENT Trading Technology Trends & Quality Assurance Conference in Obninsk, 2 March, 2013
Trading Systems:
Testing at the Confluence of FT & NFT
Alexey Zverev, Managing Director
Alyona Bulda, QA Project Lead
Ivan Bobrov, HFT Analyst

Transcript

1. Trading Systems: Testing at the Confluence of FT & NFT Alexey Zverev, Managing Director Alyona Bulda, QA Project Lead Ivan Bobrov, HFT Analyst
2. Introduction 1. Introduction 2. Our topic - how to find / reproduce "toxic issues" which can happen under the load or in real live environment conditions 3. We will talk about: a. A classical approach where FT and NFT are separate streams b. What is missed here c. Some approaches we use or may apply to fill this gap between FT and NFT (applying to trading systems) 4. Our presenters
3. FT focus on: • Test scenarios are based on functional requirements • Test tools which allow to run simplistic tests such as to send an order -> to verify a response • Prepare a functional library starting from simple scenarios and adding more complex ones step by step • Test scenarios are automated • A test Library is enhanced as we proceed with its testing and production clients scenarios • An automated library will be reused as a regression test library which is executed against each release after go live and can be taken for consequent projects NFT focus on: • Testing is based on non-functional requirements • Load scenarios are driven by the technical characteristics in PROD. • The main target of the tests is to assess non-functional requirements: latency, capacity and fault tolerance • A lot of efforts are invested in correct latency measurements and determining conditions when the system breaks • Further regressions are done to compare system performance indicators with the previous releases
4. What do we miss when we keep FT and NFT separated ? - Any complex system must sustain a certain level of concurrency. Some issues appear when several events happen simultaneously (Race conditions). Example 1: Clients send a CxlReplace request and approx. at the same time the ME sends a fill back. These 2 events meet at the Order Management server, and the state of the order can get corrupted.
5. What do we miss when we keep FT and NFT separated ? - Production order books and various system object states are more complex than in our FT scripts, and no amount of test cases can cover all possible scenarios Example 2: A User tries to hit two different implied orders coming from the same parent order which is being restated in the middle of the process.
6. What do we miss when we keep FT and NFT separated ? - Components may behave differently under the load Example 3: The System must support various throttling limitations and the requirements of the system behaviour. The correctness of functional system behavior can be tested only under the load. - Integer memory overflow Example 4: Turnover (VWAP) fields in the statistics messages and incremental OrderID overflow; * the picture was taken from https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcS9nlDdLAmuvmyRqqZsanbyfwL601nJlOvtHGEfJBS4DKnmSTANUA
7. What do we miss when we keep FT and NFT separated ? - Dynamic Mass events which happen during DLC Example 5: Market Open, Market Close, Auctions,.. The state of the system at exact instance when the system goes from one session to another should be verified with the load applied against the system.
8. Detailed Verification Of The System Behavior Under Random Load  We need to make sure that we provide extensive coverage  We need to make sure that the load test mimics what happens in the production system  We need to verify that the system works as expected - basic consistency check - market data must be internally consistent - order book should be available and not be different + Recreation of exact steps of real production issues - It is difficult to record multiple network streams
9. Performing Functional Testing Against The System Under The Load to leave a part of the system intact by the load to adjust functional tests to cope with the activity of the load to make scripts in case when we perform tests actively interacting with the load * scenarios must be flexible * sustain interaction with the script
10. Testing With The Baseline The example of the cycle of a Clearing System A detailed, step by step analysis
11. The “Mini Robots” Solution
12. The “Mini Robots” Solution
13. Questions & Answers-tO Thank You-tO