Optimize your Automation Design to Improve Test Execution Efficiency - Lessons Learnt for Realizing ‘TRUE’ ROI

This presentation is among the Top 27 Best Papers/Practice/Tutorials selected, out of 460+ submissions received, to be presented @STC 2012.

Presentation Abstract

Today most of the organizations have invested in automated testing solutions as it is critical and strategic necessity to reduce their cost and improve the cycle time on testing activities. Various automated tools and methodologies are integrated to improve the ROI significantly. It is a general misconception that Automated test execution is one click away or there is no or very minimal efforts required to execute the test suite.
In practice we need to splurge 20-30% efforts on test suite design, data setup and test execution, this paper discusses the challenges faced in test design and execution and how to overcome them and minimize the effort spent in test design and execution which in the course improves the ROI.

About the Authors

Pankaj Chhabra is a Test Manager within the independent testing practice at Collabera. He has 10+ years of Industry experience, and is currently responsible for managing a large automation project for a Fortune 100 client. His work includes strategy and solution design for test automation and helping customers build quality products.

Sathya Priya B is a Test Manager within the independent testing practice at Collabera. She has 10+ years of Industry experience, currently works as a Test Automation Architect in the Automation Centre of Excellence. Her work includes strategy and solution design for automation projects and helping the customers to build automation solutions using various tools and framework approaches.

Vamsi is responsible for managing the Testing requirements of all Communication, Media and Supply Chain clients as well as SAP Testing at Collabera. Vamsi has over 18 years of experience in the IT industry spanning application development, application maintenance and Testing. In the Testing domain, he has rich experience of working with leading global clients across banking, healthcare and beverage segments. His forte is in leading manual, automation and SAP testing projects. Before Collabera, he has worked with Accenture and MphasiS. Vamsi has an Executive MBA degree from IIM Bangalore.

Transcript

1. STC 2012 Optimize your Automation approach to improve test execution efficiency - A step towards realizing ‘TRUE’ ROI Optimize your Automation approach to improve test execution efficiency - A step towards realizing ‘TRUE’ ROI Authors Pankaj Chhabra, Test Manager Sathya Priya B, Test Manager Vamsi Krishna Belluri, Sr. Test Manager Authors Pankaj Chhabra, Test Manager Sathya Priya B, Test Manager Vamsi Krishna Belluri, Sr. Test Manager
2. Problem Statement What impedes traditional automation To Speak about… 2 Solution Approach Case Study
3. Organizations have been investing in automation but have they been able to realizing the True ROI? Test cases to High # of Releases Application Stability Test case adequacy Traditional criteria for Test Automation SUCCESS !!! TRUTH Organizations have invested test automation to reduce costs and improve testing cycle time MYTH Once scripted, automated test execution is just one click away to realize ROI ? So what lies under the Hood 3 Automate
4. Traditional Test Automation Process – Do we have the right focus? Test scripts are designed for each manual test case, i.e. One – One mapping approach where each test case is associated to one test script Test scripts are designed overlooking the pre- conditioning, data requirements and business flow Significantly high execution efforts due to more manual intervention required to fulfil the pre-conditions and test Leads to Results In Increased execution time - Script isolation, Repetitive Test Data Increased maintenance effort - Redundancy of code Inability to execute in lights-off mode - Challenges to consolidate into a batch to fulfil the pre-conditions and test data requirements Before actual automation execution To achieve the significant ROI in projected timeline the automated execution efforts should typically be 15-20% of manual test execution efforts. Leads to Results In Poor test case design results in automation execution time to be very close to the manual execution time, hence no ROI 4
5. Revised Test Automation Process – Focus on test design and execution approach/process Application Business Flow Test data Test cases to Automate High # of Releases Application Stability Test case adequacy Test Design Analyse the regression suite Identify end to end business flows Analyse the Test data flow/dependency Structure the test validation and Data as per business/data flow 5 Structure test validation Business Flow Pre-conditioning Test data flow/dependency Checking under the Hood Test Execution Classification of the scripts into Data dependent & independent categories Scripts sequencing Dynamic data generation
6. Optimize Test Design approach/process Business Case Test scripts are designed for each manual test case by overlooking the re- conditioning, data requirements and business flow Each script becomes isolated which lead to redundancy of code and increases the maintenance efforts Increases the execution time as the similar set of action will be executed in each script. Execution efforts will be significantly high as there would be comparatively more manual intervention required to fulfil the pre-conditions and test data requirements. All of the above leads to increase of effort and hence reduces the ROI. Easy script maintenance reduces scripting effort Reduces scripting Effort and Execution time Effort saved in Test Data creation by reusing existing test data Solution Approach – Deep Dive ROI. Implementation of Solution Analysis the application business and data flow and Inter- dependency of module. Test validations were merged/ demerged to fit in a business flow and dependency. Design approach fused the test suite in such a way that the outcome of one scripts becomes the input for other script instead of an isolated unit. Test data was injected by automated scripts to eliminate the manual intervention by reusing existing test data Highly coupled Business components can reduce flexibility Factors considered for optimized approach/process Create Reusable Components Consolidate the test validation as per business flow Structure the test validation as per data flow 6
7. Optimize Test Execution approach/process Business Case Test scripts are designed by overlooking the pre-conditioning, data requirements. More manual intervention required to fulfill the pre-conditions and test data requirements difficulties' to consolidate into a batch and execute in lights-off mode. Execution efforts will be significantly high as there would be comparatively more manual intervention required to fulfil the pre-conditions and test data requirements. All of the above leads to increase of effort and hence reduces the ROI. Scripts can be readily executed in lights-off mode Reduces test data dependency Significantly reduced the manual intervention during automated test execution and improved the execution efficiency Solution Approach – Deep Dive Implementation of Solution A streamlined execution process can help to reduce the test execution cost and improve the cycle time on testing activities: Classification of the scripts into Data dependent & independent categories. Scripts sequencing - Data dependent scripts can be sequenced such that the output/data generated by the previous script will be consumed by the next script for execution, thus saving the data preparation time. Automated process like dynamic data fetching from a database using a SQL, to create test data will significantly save effort and increases the accuracy of test data. execution efficiency All scripts in the sequence need to be executed.Cascading failure Factors considered for optimized approach/process Classification of the scripts Scripts sequencing Dynamic data creation 7
8. Case Study - Optimization of Automation Design approach/process Design Approach Adopted In order to realise the outcome the optimised approach/process, the solution was implemented on running as well as new projects. Design approach was optimized for new projects, initiated with Feasibility Analysis. There were around 150 Test cases picked up for an application and after thorough analysis of test cases, existing design approach was optimized by considering the application business and data flow and Inter-dependency of module. 8 Design Approach Adopted Test validations were merged/ demerged to fit in a business flow and dependency of test data was eliminated by automating the test pre- conditions and by dynamic data fetching. This design approach fused the test suite in such a way that the outcome of one scripts becomes the input for other script instead of an isolated unit. As the data was fusion by automated scripts, and by that manual intervention was eliminated, the scripts execution in batch reduced the execution efforts significantly. 60-70% test execution time saved post- optimization of design approach compare to similar nature of other project.
9. Sample Scenarios - Optimization of Automation Design approach/process Add Loan Add Loan Login Delete Loan Search Loan Login Modify Loan Search Loan Login Add/Modify Loan Validation_A Validation_B Validation_C Financial Analysis Validation_A Validation_Y Validation_N Settle Option Validation_Y Validation_S Validation_N Payment Option DB Scenario 1: Scenario 2: 9 View Loan Logout Delete Loan Logout Modify Loan Logout Business Units are built in such a way that they can be reused across multiple business flows. Payment Option Validation_S Validation_P Validation_Q Sample Business Units: Add/Delete/Update Loan Financial Analysis - Yes/No Settle Option – Short Sale / Non Short Sale Payment option -1/2/3/4 Manager Approve/Decline
10. Case Study - Optimization of Automation Execution approach/process Execution Approach Adopted Solution was implemented on completed/running projects. 900+ Test cases considered for this exercise. To start with, scripts were analyzed and categorized into data dependent/independent. Pre-conditions and data requirement was analyzed for data dependent scripts. To eliminate the dependency, a feasibility analysis to automate the pre-conditions and data creation was done. Code was injected to the existing test suite for data requirements and to fulfill the test pre-conditions. 10 Overall test execution effort savings are around 70% using automation scripts Around 35% improvement in automated test execution savings post-optimization of execution process Execution Approach Adopted Alternatively, existing automated scripts were sequenced with some code correction to work as input for data dependent scripts. This significantly reduced the manual intervention during automated test execution and improved the execution efficiency. To further reduce the execution time, automated suite was fragmented in small batches (considering data dependencies) and executed on multiple systems simultaneously. The automated test execution efforts (in Hours) reduced by 50% which in turn reduced the ROI time frame.
11. Sample Scenarios - Optimization of Automation execution approach/process Scenario 1: Module X TestCase_X1 TestCase_X2 TestCase_X3 TestCase_X4 TestCase_X5 TestCase_X6 TestCase_X7 TestCase_X8 TestCase_X9 TestCase_X10 Module X TestCase_X1 TestCase_X3 TestCase_X5 TestCase_X7 TestCase_X8 Module X TestCase_X2 TestCase_X4 TestCase_X6 TestCase_X9 TestCase_X10 Independent Scripts Dependent Scripts Scenario 3: Independent Scripts Dependent Scripts Module X TestCase_X1 TestCase_X3 TestCase_X5 TestCase_X7 TestCase_X8 TestCase_X2 TestCase_X4 TestCase_X9 TestCase_X10 Module X TestCase_X2 TestCase_X4 TestCase_X6 TestCase_X9 TestCase_X10 Code DB 11 Scenario 2: Module X TestCase_X1 TestCase_X3 TestCase_X5 TestCase_X7 TestCase_X8 Module X TestCase_X2 TestCase_X4 TestCase_X6 TestCase_X9 TestCase_X10 Independent Scripts Dependent Scripts Module X TestCase_X1++ TestCase_X2 TestCase_X3 TestCase_X5 TestCase_X7 TestCase_X8++ TestCase_X9 Scenario 1 : Classification of scripts Scenario 2 : Scripts Enhancement Scenario 3 : Data control
12. The focus is to leverage the Automation benefits by streamlining the design and execution process. This will help the organization to achieve an effective automation ROI. However, since every project is unique in nature, the cost, time and effort involved in automation must be assessed to ensure acceptable ROI. Conclusion 12 Advice for Implementing This paper is not intended as a primer on implementation best practices, but here are a few “lessons learned” and considerations for other companies to adopt the streamlined test design and execution process Conclusion s to achieve the significant ROI. .
14. Thank You 14