Trading Systems: Testing at the Confluence of FT and NFT
EXTENT Trading Technology Trends & Quality Assurance Conference in Obninsk, 2 March, 2013
Testing at the Confluence of FT & NFT
Alexey Zverev, Managing Director
Alyona Bulda, QA Project Lead
Ivan Bobrov, HFT Analyst
/ reproduce "toxic issues" which can happen under the load or in real live environment conditions 3. We will talk about: a. A classical approach where FT and NFT are separate streams b. What is missed here c. Some approaches we use or may apply to fill this gap between FT and NFT (applying to trading systems) 4. Our presenters
requirements • Test tools which allow to run simplistic tests such as to send an order -> to verify a response • Prepare a functional library starting from simple scenarios and adding more complex ones step by step • Test scenarios are automated • A test Library is enhanced as we proceed with its testing and production clients scenarios • An automated library will be reused as a regression test library which is executed against each release after go live and can be taken for consequent projects NFT focus on: • Testing is based on non-functional requirements • Load scenarios are driven by the technical characteristics in PROD. • The main target of the tests is to assess non-functional requirements: latency, capacity and fault tolerance • A lot of efforts are invested in correct latency measurements and determining conditions when the system breaks • Further regressions are done to compare system performance indicators with the previous releases
separated ? - Any complex system must sustain a certain level of concurrency. Some issues appear when several events happen simultaneously (Race conditions). Example 1: Clients send a CxlReplace request and approx. at the same time the ME sends a fill back. These 2 events meet at the Order Management server, and the state of the order can get corrupted.
separated ? - Production order books and various system object states are more complex than in our FT scripts, and no amount of test cases can cover all possible scenarios Example 2: A User tries to hit two different implied orders coming from the same parent order which is being restated in the middle of the process.
separated ? - Components may behave differently under the load Example 3: The System must support various throttling limitations and the requirements of the system behaviour. The correctness of functional system behavior can be tested only under the load. - Integer memory overflow Example 4: Turnover (VWAP) fields in the statistics messages and incremental OrderID overflow; * the picture was taken from https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcS9nlDdLAmuvmyRqqZsanbyfwL601nJlOvtHGEfJBS4DKnmSTANUA
separated ? - Dynamic Mass events which happen during DLC Example 5: Market Open, Market Close, Auctions,.. The state of the system at exact instance when the system goes from one session to another should be verified with the load applied against the system.
We need to make sure that we provide extensive coverage We need to make sure that the load test mimics what happens in the production system We need to verify that the system works as expected - basic consistency check - market data must be internally consistent - order book should be available and not be different + Recreation of exact steps of real production issues - It is difficult to record multiple network streams
leave a part of the system intact by the load to adjust functional tests to cope with the activity of the load to make scripts in case when we perform tests actively interacting with the load * scenarios must be flexible * sustain interaction with the script