Tweet
Tweet

Slide 1

Slide 1 text

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 1 Non-functional testing approach Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 2460, +1 415 830 38 49 www.exactpro.com

Slide 2

Slide 2 text

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 2 Contents • NFT Team structure • Tests preparation • Test tools • Latency measurements • Capacity tests • DLC test • Failover tests • Other NFT tests

Slide 3

Slide 3 text

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 3 NFT team structure

Slide 4

Slide 4 text

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 4 LSEG Non-Functional Testing Team

Slide 5

Slide 5 text

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 5 Test plan Sample of an Non-Functional Test plan

Slide 6

Slide 6 text

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 6 Load Generators Classification Measurement-Based Model-Based Closed-Cycle Open-Cycle

Slide 7

Slide 7 text

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 7 Asymmetric Response Model

Slide 8

Slide 8 text

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 8 8 Exactpro Systems, LLC Quality Assurance & Related Software Development for Innovative Trading Systems Tel: +7 495 640 24 60 , +1 415 830 38 49 exactpro.com Item Description Exactpro Test Strategy Step(s) Load generation in trading systems Testing Type Active Load and Non Functional Testing Target SUT Trading Platforms, Market Data Delivery and Post-Trade Systems and their combinations SUT Interface Back-end (typically connected to message gateways / APIs; data streams generation: mcast/ucast); GUI Testing Capabilities not supported SUT Interaction Method Inputs and outputs are generated based on the configured load shapes, parameters and templates. Captured messages can be viewed and analyzed post-factum using the DB queries (Shsha) or/and performance calculator tool (also developed by Exactpro) Protocols Extant plug-ins for Industry-standard (FIX and dialects, FAST, ITCH, etc.) and Proprietary (MIT, SAIL, HSVF, RTF, RV, Quant House, etc.) protocols. New plug-ins for additional protocols developed by request Test Scripts Capable to stress the system with high rate of transactions including microbursts. Used for Throughput, Bandwidth, Latency tests. Can be used for support of fault tolerance (Failover) tests Test Management, Execution and Reporting Simulation of multiple client connections with specified load shape for each connection or group of connections (configure number of connections, messages templates, Load Shape for each connection or group of connections, messages distribution for each connection or group of connections), throughput up to 75,000 msg per core per second Simulation of market data streams with required SLAs Platform requirements Linux on 64-bit platform Primary Competitor VeriFix TestPilot, HP Load Runner, IBM Rational Performance Tester, JMeter, Yandex.Tank Load Injector test tool

Slide 9

Slide 9 text

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 9 Exchange testing common scheme

Slide 10

Slide 10 text

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 10 Tests preparation • System configuration: - Production setup (Instruments, trading parameters, etc) • Data for building test load:

Slide 11

Slide 11 text

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 11 Tests preparation. Load injector configuration files Load injector configuration files structure

Slide 12

Slide 12 text

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 12 Tests preparation Load shape:

Slide 13

Slide 13 text

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 13 Latency results End-to-end Do we actually send what we thought we send? • Evaluation of message rate ‘per millisecond’ unit and order mix balance: Number of msgs per millisecond % Samples Inbound (into MIT) Outbound (from MIT) <5 55.64% 55.01% 5-8 3.67% 4.05% 8-10 2.60% 2.77% 10-15 5.32% 5.39% 15-20 5.88% 5.95% 20-80 26.85% 26.78% >80 0.05% 0.05% Partition 1 Message Type ME cores Total 0 1 2 3 Order 3.74% 3.02% 2.00% 4.14% 12.89% Cancel 3.56% 2.89% 1.93% 4.02% 12.39% Amend 0.60% 0.53% 0.34% 0.68% 2.16% Quote 0.32% 0.11% 0.16% 0.27% 0.85% Trades 0.24% 0.18% 0.13% 0.29% 0.84% Order mix and load balance across ME cores: Message rate per millisecond: • Internal system stats arbitration (rates per second and total amount of transactions)

Slide 14

Slide 14 text

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 14 Latency results End-to-end % Native avg max 100 82 518 99.99 82 408 99.9 82 139 99 80 103 0 100 200 300 400 500 600 12 29 31 31 45 45 61 61 76 76 134 286 436 519 687 896 1299 2132 2545 3376 4874 8780 17074 21219 29643 35284 40723 29 29 Latency, us rate, mps max_100 max_99_9 max_99_0 0 50000 100000 150000 200000 250000 300000 350000 400000 450000 500000 1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101 106 111 116 121 126 131 136 141 146 151 156 161 166 171 176 Number of samples Latency, us Total new_o amend_o new_q cancel_o Latency distribution: Latency percentiles:

Slide 15

Slide 15 text

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 15 Other NFT tests • Capacity test – Check all external interfaces: • Traiding FIX • Trading Native • PostTrade • DropCopy • ITCH Replay/Recovery – Check hardware resources consumption by system components: • CPU • Memory • Disk • Network

Slide 16

Slide 16 text

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 16 Other NFT tests • DLC test – Pass system through Production like schedule: • All trading cycles • All scheduled sessions – Apply appropriate load during various cycles to trigger extensions, special sessions, etc – Perform some functional tests under load – Data consistency check • All expected messages are disseminated via ITCH during session transitions • All Downstream systems connected and confirmed daily checks

Slide 17

Slide 17 text

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 17 Other NFT tests • Failover tests – Application failovers – Hardware outages • Application failovers – Simulation of various outage types (SIGKILL, SIGSTOP, Stop via SysGuard) – Failovers during different system state (at startup / trading day / EOD) • Hardware failovers – Network equipment failovers (Switches, Ports, NICs) – impact on Application – Server isolations – Oracle instances

Slide 18

Slide 18 text

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 18 Other NFT tests • Rapid user actions tests (connect-disconnect, logon-logout) – System should sustain against such user behavior – HW resources consumption should not grow up • Slow consumer tests – System should handle such users and should has a protection against them – HW resources consumption should not grow up • Intensive usage of recovery channels – System should be able to handle high number of requests on recovery channels and should be able satisfy them • Massive actions from Market Operations (mass order cancels, mass trade cancels, mass instrument halts) – System should handle Market operations’ actions like mass cancel of 10k active orders or trades

Slide 19

Slide 19 text

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 19 Other NFT tests TQ Special latency test – Main market Reference prices processing delay for TQ dark book:

Slide 20

Slide 20 text

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 20 Thank You!