Build Software to Test Software exactpro.com TestOps Environments and Monitoring Stanislav Klimakov 20th June 2018

2 Build Software to Test Software exactpro.com What is TestOps? Use of production as test environment Pros: • Very fast product delivery cycle • Wide coverage • High quality for unexposed end users Cons: • Exposed users pay for quality • It is not a solution acceptable in finance System under test Exposed Systems and Users Production Systems and Users Test Cases Analysis and Transformation Quality Assessment Production Usage Active Monitors Exposure Control

3 Build Software to Test Software exactpro.com What is TestOps?

4 Build Software to Test Software exactpro.com What is TestOps for Us?

5 Build Software to Test Software exactpro.com What is TestOps for Us?

6 Build Software to Test Software exactpro.com What is TestOps for Us? Use of a test environment as a production environment Pros: • Fast product delivery cycle • End users are not exposed Cons: • Poorer coverage in comparison with the “traditional” TestOps approach System under test Historical Test Data Anonymous Production Data Test Cases Analysis and Transformation Quality Assessment Production Usage Active Monitors

7 Build Software to Test Software exactpro.com What does “production-like” mean? • Strict configuration and version control • Restricted access to a limited number components • Granted availability and stable work during production hours • Production-only monitoring and operating solutions • Availability – production-like environment must not be idle and must add value • Accessibility – the system must grant testers full control • Additional monitoring

8 Build Software to Test Software exactpro.com Process Flow Product Delivery Test Tools and Data Delivery Testing Results Analysis and Transformation Test Environment Setup Testing tools and data are constantly improving along with product code and may be delivered on time in all environments. Test environment is automatically adapted for new testing conditions. Continuous automated testing with comprehensive result analysis. All the testing data is transformed for further use.

9 Build Software to Test Software exactpro.com Test Tools Delivery and Environment Setup • All the tools must be up to date and delivered on time in all environments • The actual test data must be updated automatically using all accessible information from production • Environment configuration must be updated gradually when required • All changes in scenarios must reflect the current environment setup

10 Build Software to Test Software exactpro.com Test Tools Delivery and Environment Setup Client’s network Version control Test env 1 Test env N … Ansible Data Processor Production data Jenkins Test tools builds Test scenarios Jira QA Server

11 Build Software to Test Software exactpro.com Testing • Tests must be executed 24/7 when possible – an idle system does not help to find issues • Test execution process must be transparent and user-friendly • Put efforts into test coverage and improvement, but not test execution

12 Build Software to Test Software exactpro.com High Touch Testing System under test Tests Results analysis Quality Assessment Am I checking all the conditions? Am I taking into account all the essential results? How did the system behave with previous release candidate installed? Pros: • A tester may notice unusual system behavior, like a real system operator Cons: • A tester may miss an issue while comparing multiple conditions • Low efficiency and coverage • Less time for test improvement

13 Build Software to Test Software exactpro.com Low Touch Testing Automated test management System under test Historical test data Quality assessment Tests Monitors Result analysis and transformation Pros: • Non-stop test execution • Less room for a human error • Less time for analysis • More time for extending test coverage Cons: • Higher tester qualification for improving automated scenarios • Validators may pass an issue that a tester could have noticed in real time I have a free hour for tests improving

14 Build Software to Test Software exactpro.com Automated tests management • High level user friendly scenarios • Event driven automated scenarios execution • Platform independent and ready to go solution • Environment specific plugins support

15 Build Software to Test Software exactpro.com Test scenarios start_load kill -9 MatchingEnginePrimary set smoke_status exec_smoke if ‘PASS’ == #{smoke_status} then goto pass echo FAIL stop_load exit label pass echo PASS 0 1 2 3 4 5 6 7 8 start_load kill -9 MatchingEnginePrimary set smoke_status exec_smoke if ‘PASS’ == #{smoke_status} then goto pass echo PASS 0 1 2 3 8 4 5 6 start_load kill -9 MatchingEnginePrimary set smoke_status exec_smoke if ‘PASS’ == #{smoke_status} then goto pass echo FAIL stop_load exit 0 1 2 3 Every scenario is language agnostic and consists of a sequence of commands (test steps). All the magic is hidden behind abstracted commands like `exec_smoke` which may be provided by an extension or be just an alias for some system script. Even though smoke test may have different logic in a variety of systems, the main scenario logic remains the same.

16 Build Software to Test Software exactpro.com Monitoring Network Purpose: • Monitor system events along with existing monitoring system provided by vendor Requirements: • Standalone tool without 3rd party dependencies for all the tasks • Easy to control, collect and transform data flows Why? • No need to adapt all the environments for our needs • Automation process requires less effort, all the scripts are standardized

17 Build Software to Test Software exactpro.com Monitoring Network ... Management Server QA Server Server 1 Server 2 Server N Router Daemon_M Daemon_S1 Daemon_S2 Daemon_SN Daemon_I SM Daemon_S Collecting system info, logs parsing, commands execution Collecting system info, logs parsing, commands execution Load control and test scripts execution Communication between daemons and controllers Automated execution of test scenarios, collecting and processing test information Daemon_M Daemon_I Router SM Data Processor Database Grafana Data Processor Transform, collect and store data for future use Data visualisation Grafana

18 Build Software to Test Software exactpro.com Data Gathering Management Server SM Data Processor Router Matching Server Daemon MEP MatchingEnginePrimary Matching log Monitoring Server Daemon MON System events log System metrics log MatchingEnginePrimary {PID: 1234, RSS: 500MB, CP Usage: 15%} System MatchingEnginePrimary {STATE: READY} MatchingEnginePrimary {INTERNAL LATENCY: 10} The MEP Daemon parses matching log and provides router with actual system info System {CP Usage: 15%, Free Mem: 50%, Free Disk Space: 80%} The MON Daemon collects system metrics and messages

19 Build Software to Test Software exactpro.com Commands Execution Management Server SM Data Processor Router Matching Server Daemon MEP MatchingEnginePrimary Matching log Monitoring Server Daemon MON System events log System metrics log Command executed {Exit status: 0} System ERROR: MatchingEnginePrimary CRASHED System {CP Usage: 1%, Free Mem: 75%, Free Disk Space: 83%} kill -9 MatchingEnginePrimary 1 ScriptManager reads user’s command and decides which daemon should execute the command and what sort of conditions should be met before execution and after using historical data from previous runs. Once test step is executed it passes all the conditions and essential system info to data processor

20 Build Software to Test Software exactpro.com Behaviour analysis Requirements: • Testing tool cannot completely rely on the deterministic scenario validators while executing tests in a complex distributed environment • Historical data must be stored and used when “almost the same conditions” are met to compare results with Problem: • Exchange operates with thousands internal metrics • Find what kinds of metrics may be compared in a particular conditions A solution must rely on AI because no tester is capable of describing all possible combinations.

21 Build Software to Test Software exactpro.com Thank you!