Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Technical Testing Introduction

5206c19df417b8876825b5561344c1a0?s=47 Exactpro
September 27, 2011

Technical Testing Introduction

EXTENT February 2011
Alexey Zverev
Exactpro Systems LLC



September 27, 2011


  1. Technical Testing Alexey Zverev EXTENT February 2011

  2. Contents • Technical Testing • Our world of trading systems

    • Approach – Performance Testing • Approach – Stability Testing • Approach – Operability Testing • Approach – Debugging • Tasks – Test Tools • Tasks – Test Preparation/Load Shape • Tasks – Test Execution • Tasks – Result analysis
  3. Technical Testing • Technical Testing or NFT – Validation of

    Non-functional Requirements • Performance • Stability • Operability • These requirements are usually applied to complex distributed server applications – Supports many interactive users – Processing high volume of data coming from and disseminated to several sources • Example: Web Sites or Web Services • Our world – Trading Systems, Exchanges, SOR, DMS, Brokerage, Trading Desks
  4. Our world of Trading Systems Trading System Broker/Trader Market Data

    Provider Downstream Systems ALGO Bad ALGO DMA/OMS SOR Transaction Storage Backoffice Post Trade Market Data Distributor
  5. The Approach to Performance Testing • Load Test – Emulate

    load – Measure KPIs (Latency, Throughput, Capacity) – Determine the load, under which the KPIs become worse than required • Stress Test – Emulate extreme load – Monitor system behaviour – Ensure that we sustain it • Soak Test (Long Running Test) – Emulate load for a long time – Check whether we have any deterioration
  6. Approach • Stability Testing – Apply Load • kill •

    unplug • do something crazy – Operability Testing • Manage the environment – Debugging • Help developers identify issues • Complex vs. Simple load
  7. Tasks Test Tools • Load injector • PCAP analyzer •

    These are some of the most essential pieces of any NFT project: – Correctly emulate the connections, implement protocol, use API – Make sure that KPI measurements are valid and accurate – Make sure that a tool does not experience performance issues itself and doesn’t influence KPI measurements – Developers whose application you are testing will blame your tools! You will require a proof of your figures based on valid sources (as the developers think)
  8. Tasks Test Preparation Load Shape • Live production system is

    the ultimate source of information
  9. Tasks Test Execution • Automated system restart and test execution

    is a good idea • Our system is a complex look at what’s going on • Be ready to run tests again and again
  10. Tasks Result Analysis • Monitoring hardware reconciles your latency and

    hardware results • Be ready to advocate test results • Perform in-depth analysis • Make sure results are repeatable – unicorns are non-existent – Most of your odd results are probably invalid test runs
  11. Questions & Answers hank you.