Compatibility Testing - A Must Do of the WebApps

This presentation is among the Top 27 Best Papers/Practice/Tutorials selected, out of 460+ submissions received, to be presented @STC 2012.

Presentation Abstract

The major changes occurring in the new century involve computerization and automation. Until some years ago there were not many different versions of web browsers to choose from. Now there are over 250 distinct browser versions.
In today’s scenario, software needs to be compatible with many platforms, browsers, peripherals, network, databases and different environment. If the compatibility issue arises, we may lose a potential customer. Proliferation of browsers, operating systems and devices has mandated organizations to ensure that all software is rendered as intended.

When software is compatible with different platform it increases the sales and improve the company branding. After all, the enduser wants high quality software that works without any glitch.
This paper aims at sharing our experience in testing in different OS with different browser version. The objective of this paper to share our strategy, challenges, approaches in Compatibility testing.

Our approach is (and not limited to) test environment selection based on market strategies, prioritization, virtualization technologies, end to end test coverage, etc.. To decide what to test we need to understand what is likely to break. The current batch of web browsers has a set of commonly known bugs and differences.

A software product compatible with multiple OS & browser environments will be the most successful one, both commercially and structurally.

Key Takeaways
• Strategic approach to compatibility testing
• Highlighting cross-browser issues
• Performance dashboard
• Reduce testing costs
• Reduce hardware maintenance
• Minimize user dissatisfaction resulting from compatibility issues
• Testing results which becomes a good reference for continual improvement

About the Authors

Premalatha, Project Manager at Indium Software (India) Ltd., comes with 8 years of experience in the IT industry focusing on software testing and managing delivery. She holds anM.Phil in Computer Science and has experience in education and life sciences domain. This expertise in compatibility testing comes from managing the compatibility testing project management and delivery for many years.

Kokila, Test Engineer, Indium Software (India) Ltd., works with 2+ years of experience and she is into compatibility testing project. She holds a B.Tech in Information Technology. She has contributed many presentations internally which includes topics like cloud computing, virtualization. These efforts have lead the way to this paper. She has good knowledge in analyzing test scenarios and MS Excel macros.

Transcript

1. Compatibility Testing - A Must Do of the Web Apps Premalatha Shanmugham & Kokila Elumalai Premalatha Shanmugham & Kokila Elumalai
2. ENHANCING SOFTWARE QUALITY Agenda The Need The Impact The Challenges The Strategy The Checklist Metrics 2 Metrics Inferences The Road Ahead 2 © 2012 Indium Software
3. ENHANCING SOFTWARE QUALITY The Need Operating Systems Windows XP, Vista, Win7, MAC, Unix, Linux, etc., Browsers IE, Firefox, Chrome, Safari, Opera, etc., Databases Oracle, Sybase, DB2, Mysql, SqlServer, etc., 3 3 © 2012 Indium Software Mysql, SqlServer, etc., System Software Web server, networking, messaging tool Platforms PC, PDA, Laptop, Mobile, PAD Choices are in abundance for users
4. ENHANCING SOFTWARE QUALITY The Impact Impact:- • High cost of customer support • Decrease in user willingness to recommend That was a bad choice 4 4 © 2012 Indium Software recommend • Negative client feedback • Increase channel cost • Tarnish company’s reputation • Lost sales
5. ENHANCING SOFTWARE QUALITY The Challenges Many different browsers Many different versions New versions released frequently Business Setting up test environment Huge resource allocation Testing 5 5 © 2012 Indium Software New versions released frequently User preferences changes Standard follows, still there’s always variation Huge resource allocation Test coverage in multiple browser Quality deliverable in short time
6. ENHANCING SOFTWARE QUALITY The Strategy Resource Management Reporting • Browsers, Platforms and Plugins Tracking • Systems & contact maintenance • Record test results • Dashboard with compatibility data 6 Targeting Planning Testing Communication 6 © 2012 Indium Software • Platform Identification • Current deployment core list • Scope, Objectives • Define & Schedule • Virtualization • Configurations and goals • Schedule & status • Testing platforms of top priority • Pilot deployments
7. ENHANCING SOFTWARE QUALITY The Checklist Compatibility testing should ensure that Users have the same visual experience irrespective of the browsers In terms of functionality, the application must behave & respond the same way across browsers UI Validations Functionality Validations 7 7 © 2012 Indium Software CSS Rendering JavaScript Copyrights Applets Dropdown controls Check-in / Checkout Tabs Export / Import Clipboard Links
8. 8 © 2012 Indium Software
9. ENHANCING SOFTWARE QUALITY Inferences Inferences (based on the test results of our case study) Application is faster in Firefox compared to other browsers Testing in Windows 8 developer edition/ IE 10 Beta brought out more bugs as in IE6 Application is more stable in Firefox versions, IE latest versions and Safari Java applets are not supported in Chrome. Some features of the application throws UI issues. The following browsers are suggested for inclusion in the test matrix : 9 9 © 2012 Indium Software Browser Reason for inclusion in test matrix IE 8, IE 9 Top browser with the highest market share of 46% Chrome Most common browser in use today & has significant rising market share of 22% Firefox Fastest browser next to Chrome in use with 19% share iPhone & iPad Uses webkit, the largest single mobile browser Safari 5.x, Safari 6.x Default browser on OS X
10. ENHANCING SOFTWARE QUALITY The Road Ahead Our compatibility testing is based on proven testing methodologies and experience, powered with Framework comes with the following features: Automated, Re-usable Framework to reduce Compatibility Testing overheads Seamless updates to compatibility matrix to ensure high quality end user experience Key Levers With Compatibility Framework Value Additions Before After Saving USD Delivery Cycle Reduced by 30% Streamlined process cycle time * 8 – 10 5–7 days 10 10 © 2012 Indium Software Delivery Cycle Reduced by 30% Streamlined process cycle time Ensures high quality service delivery * 8 – 10 days 5–7 days ~USD 100,000 Test Execution Phase Faster by 20% Automated, Re-usable Framework Execute Automated tests * 5 – 7 days 4–6 days Reporting Faster by 40% Proactive monitoring of end-user device configurations * 2 days 1 day Defect Detection Increased by 25% Jump Start Kits-Known failure patterns across different OS/Browser combinations 100 defects /Qtr 125 defects / Qtr ~USD 50,000 Testing ROI 50-60% Savings Reduces compatibility Testing overheads ~USD 200,000 USD 100,000 ~USD 100,000 -> Total cost saving is ~ USD 300,000 * Test cycle - 10 days
11. ENHANCING SOFTWARE QUALITY Thank You In God we Trust... Everyone else’s code we Test! 11 11 11 In God we Trust... Everyone else’s code we Test! © 2012 Indium Software