Nan Zhang Senior QA Manager GFT Technologies Test Automation and Agile Testing

Nan Zhang June 2016 EXTENT 2016 CONFERENCE Test Automation and Agile Testing

GFT Group 27/06/2016 3 Executive Summary GFT GROUP  This presentation discusses the important role test automation plays in shaping Quality Assurance in Agile delivery methodology and process. The topics include:  Test Automation  Test automation approach and benefits  Test automation framework  Industrialised test automation  Agile Testing  Agile delivery approach overview, activities, roles and responsibilities  Agile testing scope, approach, and activities  Agile testing phases and process  Role of test automation in shaping Agile testing

GFT Group 27/06/2016 4 Test Automation Overview– Approach GFT GROUP  Approach  Test automation solutions can be implemented through either custom-built solution or 3rd party products  Test automation framework should be portable, reusable, easily maintained and designed to be transitioned to client’s team for both GUI and server side automations  Test automation solutions allow financial institutions to significantly increase test effectiveness and test coverage, therefore ensuring a considerable reduction in “time to test” and decreased cost of testing over time  It should be recognised that test automation comes at a cost and the predicted return on that investment should be achievable. For that reason, the early estimation of effort and expected benefits is critical as is the ongoing collection of metrics to validate that ROI  The most practical automation solution should be pursued – whether using a known testing tool (e.g. QTP or Selenium, etc) or through the development of own ‘fit for purpose’ test harness  The approach is to provide deliverables (automated test suites) early and then on a regular basis such that the overall test automation framework provides visible and increasing benefit – end-to-end  Test automated solutions will, wherever feasible, be integrated into the build process to deliver an effective continuous integration solution

GFT Group 27/06/2016 5 Test Automation Overview - Benefits GFT GROUP  Benefits  Increased efficiency and coverage of tests through effective test automation  Reduced overhead of regression testing  Reduced cost of testing over time  Reliability and predictability of tests by eliminating risk of human error  Greater number of test runs possible in regression testing but also as smoke tests or within a continuous build / integration environment  Enables the realisation of Agile development methodology and delivers shippable product after each sprint

GFT Group 27/06/2016 6  What is a “Test Automation Framework”?  The key to successful test automation is to build a solution that is usable by all test analysts, is robust (can handle unexpected events and system errors), reuses common test steps and is easily maintainable.  Using a test automation tool ‘out of the box’ will only cover these criteria to a small extent as the principle behind most tools is to record the steps taken – including all keystrokes, mouse operations, screen navigation – that a manual tester performs to execute a test such that the test can be played back multiple times. This has numerous drawbacks: e.g. every single test has to be recorded and so no reuse; as soon as the system changes so all the tests have to be re-recorded.  The answer therefore is to build libraries of common functions; parameterise the test scripts and externally hold as much as possible in relation to the test actions (using keywords) and test data. In addition build error handling, smart reporting, test dependencies and importantly an interface to the underlying test automation tool that can be used by all test analysts. These are all components of what is referred to as the Test Automation Framework.  The full framework encapsulates the automated regression test capability at both the front (GUI) end and the server side (database/interfaces/messages); the ability to manage the testing and the use of the framework; the tracking of progress through dashboards and reports Test Automation Framework Automated Regression Scripts Automated Smoke Test Scripts Standards Function Libraries Environment Variables Common Libraries Object Repository Error Handling Tools and Utilities Test Automation Tool (e.g. QTP, Selenium) Data Test Management Archive Risk Management Trade Processing Finance & Accounting Pricing & Analytics Reporting & Audit Settlement Upstream Systems Downstream Systems Application Under Test Auto Test GUI  Usability  Portability  Maintainability  Flexibility  Robustness  Reuse GFT GROUP The GFT Test Automation Framework

GFT Group 27/06/2016 7  Description  Industrialised test automation approach harmonises different testing activities applied to the different subsystems in a large scale systems integration project. This harmonization enables reuse of testware, test environments, test simulations, test automation and testing concepts for the systems integration project.  Approach  Reach a testing maturity level where the test process is well established and the corresponding working products are defined and controlled, at which point reuse can take place systematically.  For synchronisation of the test processes between different parties, an independent third party can be utilised which guarantees that each involved party is working on reaching the testing maturity level in sync with all other parties.  Objective  Overall goal is to increase test efficiency by a high degree of reuse and to increase effectiveness by combining different test environments to have more intermediate test stages on the integration path.  Benefits  Many synergies by reusing testware that can be leveraged, reducing test efforts significantly for all involved parties.  Improve the overall quality of the system, since each involved party can focus its test activities on those aspects that are core for their deliverables  Industrialised test automation enables layered automated integration testing. Industrialised Test Automation GFT GROUP  Industrialised test automation approach applies to a large scale systems integration project:

GFT Group 27/06/2016 8 Agile Delivery Approach - Overview Project Kick Off  Ensure that there are clearly defined sprint and release goals (i.e. through to delivery milestone).  Agree and document the definition of (‘Done’) at Sprint level and Release level  Agree and publish the team’s sprint cycle showing the length and schedule for each sprint iteration.  Define and document the triage processes for story readiness and defects  Plan for automated testing and continuous integration Ongoing  Keep the daily stand up brief: what I did yesterday, what I will be doing today, any blockers; take issues offline.  The agreed Definition of ‘Done’ must be adhered to  Do not change the agreed sprint items unless directed to by the Product Owner.  Deploy each sprint to UAT or production (depending on definition of ‘Done’) 2 3 1 4 5 6 7 8 9 10 ACTIVITIES Deliver & Test Stories Business & Regression Test Code Freeze Sprint Planning Show & Tell Retrospective Daily Stand Up Defect Triage REGULAR MEETINGS Project Working Group ARTE- FACTS Delta Report Commitment List Project Status Report Project Status Report Backlog Backlog Refinement Estimate & Prioritise Stories DAY Deliver & Test Stories Code Freeze & Deployment Check Point Project Working Group Scrum of Scrums Show & Tell Rehearsal GFT GROUP

GFT Group 27/06/2016 9  The role of testing within an Agile development environment is of paramount importance with the testers fully integrated within the team so that testing requirements can drive development Agile Testing Approach  The GFT ‘sprint testing’ approach includes:  Test Driven Development (TDD)  Behaviour Driven Development (BDD)  Exploratory Testing  Incremental Testing Test Driven Development Prioritised requirements backlog Prioritised Manual and Automated Test Scripts Daily Meetings Daily Reports (Metrics) Continuous Automated Regression Test Release ready to go live Go-Live Readiness Weekly Test Reports GFT Sprint Testing Approach GFT GROUP  Automation  The implementation of a test automation solution is essential and will bring significant benefits in an Agile environment.  Automating tests as early as possible and building a regression pack to run on a daily basis

GFT Group 27/06/2016 10  GFT implementation teams focus on the following areas in the context of Agile testing. GFT GROUP Agile Testing Scope and Activities  Designing a testing framework and infrastructure can be as complex as designing and implementing the system being tested. Test analysts focus on doing the technical design of testing frameworks that are able to meet requirements for testing individual releases and ongoing system test and certification  Functional test design requires tight alignment between the business analysts that are defining the functional capability of the system and the test analysts that are defining the test cases. Test analysts and engineers should be deployed to work as part of an integrated design and development team and ensure the specification of comprehensive test coverage  Test automation experts engage with clients to design and implement test automation frameworks that are portable, reusable, maintainable, and able to achieve increased test effectiveness, reduced testing time and reduced overall cost. Test automation tools are analysed and selected for both functional and non-functional test.  One of the most difficult aspects of testing is creating and managing the data that drives the testing framework. Test teams deploy for clients a project specific approach to the management and masking of real production data and the generation of comprehensive test data where the use of live data is not possible  The test team is able to provide strategic direction and hands-on implementations around non- functional testing and in particular, performance tuning, performance testing, system availability testing and security Test Design Functional Testing Test Automation Test Data Management Non-Functional Testing

GFT Group 27/06/2016 11  Testing for individual user stories are done within the sprint (i.e. the definition of Done includes the completion of testing)  A continuous integration environment is in place to enable code deployed daily and also on-demand.  Automated test suites kicked off after code deployment  SIT and UAT are run after each sprint in an iterative and progressive manner  Each sprint aims to deliver a shippable product to production environment  Test automation and Continuous Integration plays vital role in the realisation of such approach GFT GROUP Agile Testing Phases and Process Sprint N start Sprint N (Sprint test) Sprint N end Sprint N deployed to SIT env UAT execution Prod Deployable Continuous Integration Test CI Deployment Sprint N+1 start Sprint N+1 end Product Backlog Refinement Sprint N deployed to UAT env Activity Test Phase

GFT Group 27/06/2016 12 Test Automation in Agile Testing GFT GROUP Levels and areas  Unit tests  Backend (DB, services)  UI (validating behaviour or UX design)  Continuous Integration Testing (daily build and automated test)  Full E2E solutions (with all system components and dependency systems)  Non-functional testing (Performance validation, soak test) Implementation  Test framework based on open source, custom solution or mix  Building re-usable solutions (reducing next project kick-off cost)  Detailed test cases prepared by QA  Tests can be written by QA or Dev (fast test coverage increase if needed)  Coverage can be measured by traceability matrix  All tests integrated with CI environment to produce daily reports  Test automation built for both functional and non-functional test

Thank you Any Questions?