Technical Testing Introduction

EXTENT February 2011
Alexey Zverev
Exactpro Systems LLC

Transcript

1. Technical Testing Alexey Zverev EXTENT February 2011
2. Contents • Technical Testing • Our world of trading systems • Approach – Performance Testing • Approach – Stability Testing • Approach – Operability Testing • Approach – Debugging • Tasks – Test Tools • Tasks – Test Preparation/Load Shape • Tasks – Test Execution • Tasks – Result analysis
3. Technical Testing • Technical Testing or NFT – Validation of Non-functional Requirements • Performance • Stability • Operability • These requirements are usually applied to complex distributed server applications – Supports many interactive users – Processing high volume of data coming from and disseminated to several sources • Example: Web Sites or Web Services • Our world – Trading Systems, Exchanges, SOR, DMS, Brokerage, Trading Desks
4. Our world of Trading Systems Trading System Broker/Trader Market Data Provider Downstream Systems ALGO Bad ALGO DMA/OMS SOR Transaction Storage Backoffice Post Trade Market Data Distributor
5. The Approach to Performance Testing • Load Test – Emulate load – Measure KPIs (Latency, Throughput, Capacity) – Determine the load, under which the KPIs become worse than required • Stress Test – Emulate extreme load – Monitor system behaviour – Ensure that we sustain it • Soak Test (Long Running Test) – Emulate load for a long time – Check whether we have any deterioration
6. Approach • Stability Testing – Apply Load • kill • unplug • do something crazy – Operability Testing • Manage the environment – Debugging • Help developers identify issues • Complex vs. Simple load
7. Tasks Test Tools • Load injector • PCAP analyzer • These are some of the most essential pieces of any NFT project: – Correctly emulate the connections, implement protocol, use API – Make sure that KPI measurements are valid and accurate – Make sure that a tool does not experience performance issues itself and doesn’t influence KPI measurements – Developers whose application you are testing will blame your tools! You will require a proof of your figures based on valid sources (as the developers think)
8. Tasks Test Preparation Load Shape • Live production system is the ultimate source of information
9. Tasks Test Execution • Automated system restart and test execution is a good idea • Our system is a complex look at what’s going on • Be ready to run tests again and again
10. Tasks Result Analysis • Monitoring hardware reconciles your latency and hardware results • Be ready to advocate test results • Perform in-depth analysis • Make sure results are repeatable – unicorns are non-existent – Most of your odd results are probably invalid test runs
11. Questions & Answers hank you.