Test Size Driven Productivity Tracking

Best of the Best Award - 2nd Runner Up presentation by Robert Bosch Engineering and Business Solution @STC 2012.
Authors - KrishnaKishore P.V, Srividya Ponnusamy

Presentation Abstract

Test Case execution with stringent timelines is a major challenge for every testing project, environment and build stability would add to this and are more rigorous in agile environments.
Majority of the projects would adapt to test case number based productivity tracking, which is often unrealistic due to the variations in each test case size and its complexity.

This paper is going to present test case size driven productivity tracking and its benefits, test case size can be derived by considering the complexity factors, number of test steps, test data dependencies and other influencing parameters specific to project. Project progress can be closely tracked based on the actual size delivered per week, irrespective of the test case count.

This process can be further optimized by developing an excel macro based tool, this tool will fully automate the process of deriving actual size from the execution results and compare with expected size to arrive at the project status., further to this we can also integrate the dashboard reports generated from test management tools with this process.

About the Authors

P.V Krishna Kishore holds M.Tech in Electronics & Control from Sathyabama Deemed University and started his career with Cognizant as Test Engineer. Later joined Infosys Hyderabad as Test Lead in Automation. Presently he is working for Robert Bosch Engineering and Business Solutions Limited (RBEI) as Senior Project Manager, leading the Validation services group for Bosch TeleHealth Products. He carries 10+ years of experience in software testing both in manual and automation testing, extensive knowledge and domain expertise in TeleHealth, Hybrid Automation Framework Implementation and Risk Based Testing. He initiated to have more conventional method of productivity measurement and tracking for testing projects, which has now evolved as a solution and can be extended to all the testing projects.

Srividya works for Robert Bosch Engineering and Business Solutions Limited (RBEI) as Technical Specialist. She has completed her B.E (Electronics and Communications Engineering) from Coimbatore Institute of Technology and has interest in learning new technology. She has been in industry for 8 years and has worked across various domains like Embedded Medical, Embedded Semiconductor and Insurance where she has been involved in development as well as manual and automation testing. She has done the core implementation by providing automation solution to integrate the dashboard results, measure the actual size and plot the trends.

Transcript

1. Test Size Driven Productivity Tracking Program Management 1 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
2. Abstract Test case execution, a major challenge ! Issues with test environment, build stability etc., Unrealistic ‘Traditional Approaches’ 2 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Unrealistic ‘Traditional Approaches’ New approach: Measure Test Size
3. Challenges in traditional methods Need to measure delivered test size Implement TSDPT Introduction - Focus Areas 3 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Demonstrate benefits
4. Intended Audience, Key Takeaways Traditional approaches: Case Study #1 Our Approach/Solution/Benefits: TSDP Tracking Contents 4 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. TSDP Tracking: Case Study #2 Analysis: Case Study #1 & #2 Improvements, References, Q&A
5. Test Management Professionals QA Managers Intended Audience 5 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Test Leads Test Engineers
6. Importance of size & complexity Productivity tracking based on test size Key Takeaways 6 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Establish, track and improve productivity Integrate this solution with test management tools
7. Test Execution Tracking Based on fixed timelines Uses test management tool dashboard Traditional Approaches 7 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. No of Test cases delivered
8. Drawbacks Incorrect progress tracking Size, Complexity not accounted Traditional Approaches.. 8 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Individual SQA productivity - Not tracked Schedule slippage unnoticed Expensive corrective measures
9. Case Study #1: Test Case Count Approach 9 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Graph 1: Test Execution Progress Tracking By Test Case Count Approach Microsoft Excel Worksheet
10. Key Observations Good productivity during initial weeks Case Study #1.. 10 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Deviations observed from 5th week only Control measures planned from 5th week
11. Analysis Not accounted the test case size Case Study #1.. 11 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Not accounted the test case size Could not adhere to test suite time frames Results are not trusted
12. Conclusions Need to measure the test size delivered Case Study #1.. 12 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Productivity should be defined Evaluate results for corrective measures
13. Productivity = Size of a work product / Total Effort Our Approach: TSDPT Test Size = (Complexity * Design Steps)/100 Complexity = 1/0.5/0.25 13 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Complexity = 1/0.5/0.25 DB Access Subsystem Testing, H/W, S/W Interface Test Data & Test Environment Documentation etc..,
14. Planned size (Total size) = 100 Estimated Effort = 125 days Productivity Baseline = 100/125 = 0.8 Define size: 5 Week Plan with 5 Tester TSDPT - Approach .. Productivity Goals 14 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Estimated Effort = 25 days (125/5) Size Expected = 20 per Tester (100/5) Size to be delivered per Week = 4 (20/5) Define size: 5 Week Plan with 5 Tester
15. Productivity Baseline Derive size Assignments Excel Solution End-to-End Collect, analyse & track TSDPT - Solution 15 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Automation solution Excel macro tool Computes the actual size delivered per week Design Dashboard Reports Test management tool Excel Report with actual test cases executed
16. Productivity baselines Effective tracking mechanism Apply control measures TSDPT - Benefits 16 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Uncovers challenges with traditional approaches Highly adoptable Test effectiveness tracked per tester closely
17. Step1: Derive Test Size Table 1: Test Size Table Test # No Steps Complex Size TC1 14 1 0.14 TC2 28 0.5 0.14 TC3 27 0.25 0.07 Case Study #2: Test Case Size Approach 17 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Table 1: Test Size Table Step2: Define Productivity/Week Table 2: Productivity Table Tester Ind Size Effort Size/Week SQA1 36.40 44.00 3.64 SQA2 32.40 48.20 3.24
18. Step3: Collect Dashboard Report TESTER TC STATUS SIZE Case Study #2.. 18 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. SQA1 TC1 P 0.02 Table 3: Test Execution Report
19. Step4: Run Automation Macro Compute Size Per Tester / Week Table 4: Test Size Report SQA1 SQA2 TOTAL E TOTAL A E A E A 3.6 3.72 3.3 3.3 6.9 7.02 Case Study #2.. 19 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Table 4: Test Size Report Compute Cumulative Size Table 5: Cumulative Test Size Report Week # SQA1 SQA2 TOTAL E TOTAL A E A E A 6 21.6 19.2 19.4 16.1 41.0 35.3
20. Case Study #2.. Step5: Plot the results as chart 20 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Chart 1: Week 6 Graph Per Tester
21. Case Study #2.. 21 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Chart 2: Week 6 Cumulative Graph
22. Case Study #2.. 22 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Chart 3: Overall % Execution Graph
23. Case Study #2.. 23 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Graph 2: Test Execution Progress Tracking By Test Size Approach
24. Case Study #2.. Key observations Deviations 24 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Early Control Measures
25. Case Study #1 & #2 25 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Graph 3: Test Execution Progress Tracking: Test Case Count (Vs) Test Size Approach
26. Case Study #1 & #2 Key observations 10 to 15 % Variations 26 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights. Misleading Results Ineffective Tracking
27. Improvements End-To-End tool solution Extend to automation & other areas 27 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
28. References From own experience No reference 28 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.
29. Q&A Thank You! 29 Department | 09/10/2012 | © Robert Bosch Engineering and Business Solutions Limited 2012. All rights reserved, also regarding any disposal, exploitation, reproduction, editing, distribution, as well as in the event of applications for industrial property rights.