Alexey Zverev CEO Exactpro Testing Complex Systems within Agile Development Framework

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 2 QA management at Exactpro Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 2460, +1 415 830 38 49 www.exactpro.com Alexey Zverev, Co-CEO, Exactpro

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 3 Introduction

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 4 Agenda 1. Fundamental principles of QA Management 2. The anti-patterns 3. Summary of our principles vs. the anti-patterns 4. Questions & Answers

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 5 Fundamental principles of QA Management QA is a continuous learning and development process 1. Learn how the system should operate in production 2. Learn about the actual behaviour 3. Make sure all discrepancies are either fixed or documented 4. Develop Test Design, Test Tools and an Automated Test Library through a series of small iterations This is a permanent process throughout the delivery cycle and after the actual Go-Live, whilst building the test tools adds more efficiency, and therefore, we find as many issues as possible.

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 6 The anti-patterns 1. “The Requirement Traceability Matrix” 2. “Test Automation” 3. “The Test Library” 4. “The Test Plan” 5. “Non-Functional Testing” 6. “The Test Tool” 7. “The Scrum Team”

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 7 “The Requirements Traceability Matrix” “We need to write Test Cases at the beginning of the project” “All Test Cases must contain direct links to Requirements” “Stakeholders must see test coverage and correspondence between tests cases and software requirements”

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 8 “Test Automation” “We must automate 100% of our Test Library. Let’s start now” “Complete regression must happen overnight as part of the build process” “As a result of test automation we should reduce the QA team to a few people”

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 9 “The Test Library” “We must document all our test scenarios and put them into Test Management Software” “Test Library consists of Test Suites which consist of Test Scenarios which consist of Test Cases” “Once we accomplish this we will start test automation” “Exploratory Testing? What is it? No we don’t do it” “All test cases contain detailed test steps and Testers must put Pass/Fail as they execute them”

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 10 “The Test Plan” “Let’s estimate QA effort required to deliver this project” “Lets plan QA tasks and start implementing them in waterfall model” “Let’s monitor process closely and put our effort into achieving targets and milestones”

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 11 “Non-Functional Testing” “We need to test that the system supports 10,000 orders per second” “We need to make sure that latency is within expected limits” “We will allocate production-like hardware to focus on capacity and latency measurements”

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 12 “The Test Tool” “We have purchased a licence for a magical test automation tool” “We will use it for all of our test automation purposes” “It allows interacting with GUI / via FIX / via SWIFT and has the capacity to record/replay test scenarios” “It is expensive”

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 13 “The Scrum Team” “We are using scrum and will test everything as a part of the sprint” “Testing is done and automated by Developers who can temporarily allocate their time within scrum” “We will put some QA engineers into Scrum team and let them write unit tests” “We are using DevOps we don’t need QA, QA is dead”

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 14 Summary of our principles vs. the anti-patterns 1. Developing team knowledge vs. “The Requirement Traceability Matrix” 2. Building software to test software vs. “Test Automation” 3. Test Design vs. “The Test Library” 4. Agile iterative approach vs. “The Test Plan” 5. Testing at the confluence of Functional and Non-Functional Testing vs. “Non-Functional Testing” 6. Pragmatic in-house development of test tools vs. “The The Test Tool” 7. The QA team is a pacemaker vs. “The Scrum Team”

Open Access Quality Assurance & Related Software Development for Financial Markets Tel: +7 495 640 24 60 , +1 415 830 38 49 www.exactpro.com 15 Questions & Answers Thank you!