Exactpro: Non-functional testing approach

Exactpro: Non-functional testing approach

Transcript

1. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 1 Non-functional testing approach Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 2460, +1 415 830 38 49 www.exactpro.com
2. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 2 Contents • NFT Team structure • Tests preparation • Test tools • Latency measurements • Capacity tests • DLC test • Failover tests • Other NFT tests
3. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 3 NFT team structure
4. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 4 LSEG Non-Functional Testing Team
5. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 5 Test plan Sample of an Non-Functional Test plan
6. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 6 Load Generators Classification Measurement-Based Model-Based Closed-Cycle Open-Cycle
7. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 7 Asymmetric Response Model
8. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 8 8 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com Item Description Exactpro Test Strategy Step(s) Load generation in trading systems Testing Type Active Load and Non Functional Testing Target SUT Trading Platforms, Market Data Delivery and Post-Trade Systems and their combinations SUT Interface Back-end (typically connected to message gateways / APIs; data streams generation: mcast/ucast); GUI Testing Capabilities not supported SUT Interaction Method Inputs and outputs are generated based on the configured load shapes, parameters and templates. Captured messages can be viewed and analyzed post-factum using the DB queries (Shsha) or/and performance calculator tool (also developed by Exactpro) Protocols Extant plug-ins for Industry-standard (FIX and dialects, FAST, ITCH, etc.) and Proprietary (MIT, SAIL, HSVF, RTF, RV, Quant House, etc.) protocols. New plug-ins for additional protocols developed by request Test Scripts Capable to stress the system with high rate of transactions including microbursts. Used for Throughput, Bandwidth, Latency tests. Can be used for support of fault tolerance (Failover) tests Test Management, Execution and Reporting Simulation of multiple client connections with specified load shape for each connection or group of connections (configure number of connections, messages templates, Load Shape for each connection or group of connections, messages distribution for each connection or group of connections), throughput up to 75,000 msg per core per second Simulation of market data streams with required SLAs Platform requirements Linux on 64-bit platform Primary Competitor VeriFix TestPilot, HP Load Runner, IBM Rational Performance Tester, JMeter, Yandex.Tank Load Injector test tool
9. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 9 Exchange testing common scheme
10. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 10 Tests preparation • System configuration: - Production setup (Instruments, trading parameters, etc) • Data for building test load:
11. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 11 Tests preparation. Load injector configuration files Load injector configuration files structure
12. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 12 Tests preparation Load shape:
13. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 13 Latency results End-to-end Do we actually send what we thought we send? • Evaluation of message rate ‘per millisecond’ unit and order mix balance: Number of msgs per millisecond % Samples Inbound (into MIT) Outbound (from MIT) <5 55.64% 55.01% 5-8 3.67% 4.05% 8-10 2.60% 2.77% 10-15 5.32% 5.39% 15-20 5.88% 5.95% 20-80 26.85% 26.78% >80 0.05% 0.05% Partition 1 Message Type ME cores Total 0 1 2 3 Order 3.74% 3.02% 2.00% 4.14% 12.89% Cancel 3.56% 2.89% 1.93% 4.02% 12.39% Amend 0.60% 0.53% 0.34% 0.68% 2.16% Quote 0.32% 0.11% 0.16% 0.27% 0.85% Trades 0.24% 0.18% 0.13% 0.29% 0.84% Order mix and load balance across ME cores: Message rate per millisecond: • Internal system stats arbitration (rates per second and total amount of transactions)
14. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 14 Latency results End-to-end % Native avg max 100 82 518 99.99 82 408 99.9 82 139 99 80 103 0 100 200 300 400 500 600 12 29 31 31 45 45 61 61 76 76 134 286 436 519 687 896 1299 2132 2545 3376 4874 8780 17074 21219 29643 35284 40723 29 29 Latency, us rate, mps max_100 max_99_9 max_99_0 0 50000 100000 150000 200000 250000 300000 350000 400000 450000 500000 1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101 106 111 116 121 126 131 136 141 146 151 156 161 166 171 176 Number of samples Latency, us Total new_o amend_o new_q cancel_o Latency distribution: Latency percentiles:
15. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 15 Other NFT tests • Capacity test – Check all external interfaces: • Traiding FIX • Trading Native • PostTrade • DropCopy • ITCH Replay/Recovery – Check hardware resources consumption by system components: • CPU • Memory • Disk • Network
16. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 16 Other NFT tests • DLC test – Pass system through Production like schedule: • All trading cycles • All scheduled sessions – Apply appropriate load during various cycles to trigger extensions, special sessions, etc – Perform some functional tests under load – Data consistency check • All expected messages are disseminated via ITCH during session transitions • All Downstream systems connected and confirmed daily checks
17. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 17 Other NFT tests • Failover tests – Application failovers – Hardware outages • Application failovers – Simulation of various outage types (SIGKILL, SIGSTOP, Stop via SysGuard) – Failovers during different system state (at startup / trading day / EOD) • Hardware failovers – Network equipment failovers (Switches, Ports, NICs) – impact on Application – Server isolations – Oracle instances
18. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 18 Other NFT tests • Rapid user actions tests (connect-disconnect, logon-logout) – System should sustain against such user behavior – HW resources consumption should not grow up • Slow consumer tests – System should handle such users and should has a protection against them – HW resources consumption should not grow up • Intensive usage of recovery channels – System should be able to handle high number of requests on recovery channels and should be able satisfy them • Massive actions from Market Operations (mass order cancels, mass trade cancels, mass instrument halts) – System should handle Market operations’ actions like mass cancel of 10k active orders or trades
19. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 19 Other NFT tests TQ Special latency test – Main market Reference prices processing delay for TQ dark book:
20. Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 20 Thank You!