How to Achieve Complete Test Coverage for an e-Commerce Mobile Application with Reduced Effort

This presentation is among the Top 27 Best Papers/Practice/Tutorials selected, out of 460+ submissions received, to be presented @STC 2012.

Presentation Abstract

Large number of e-commerce users are trending from desktops towards mobile devices for their day-to-day shopping needs. Accordingly several e-commerce websites in market are supporting their businesses on multiple mobile devices/platforms through mobile compatible websites and hybrid apps.

Hence there is a tremendous need for qualifying these web sites and applications for compatibility on various mobile platforms and device models. The success of any e-commerce mobile application or a website very much lies in the quality in terms of look and feel, ease of use, robustness, usability, performance, security and many more factors.

Considering plethora of factors involved in this exercise like OS types, versions, device models and browser types, achieving exhaustive test coverage requires enormous efforts. The intent of this white paper is to provide proven techniques to reduce the effort of testing while providing complete coverage of business needs. This paper also talks about Tools, techniques, best practices, challenges and myths that can be leveraged in mobile e-commerce testing space.

About the Author

Vishnu Bangar Raju C is a Senior Test Manager in e-commerce vertical with Collabera. He has 12+ years of experience in the IT industry. His areas of interest are predominantly in Automation, Performance and Mobile application testing spanning across domains including Financial Services, E-Commerce, Health care and Utilities.

Suresh Babu V is a Test Manager with Collabera in Mobile Testing COE. He has 13+ years of experience in the IT industry. He has expertise in Automation, Functional and Mobile application testing spanning across domains including Financial Services, Utilities, Telecom and Desktop applications.

Padma Podduturi is a Test Manager with Collabera in Mobile Testing COE. She has 8 years of experience in the IT industry. Her areas of interest are predominantly in Automation, Performance and Mobile application testing spanning across domains including Financial Services and E-commerce.

Transcript

1. STC 2012 How to Achieve Complete Test Coverage for an e-Commerce Mobile Application with Reduced Effort Authors Vishnu Bangar Raju C – Senior Test Manager Suresh Babu V – Test Manager Padma Podduturi – Test Manager
2. Agenda Context Business Trends in e-commerce mobile application space Challenges in testing An approach to minimize testing effort/budget Tools & Techniques Automation Tools – Functional Testing Best Practices 2 Best Practices Case Studies
3. Context Large number of e-commerce users shifting from desktops towards mobile devices. E-commerce companies started supporting their businesses on mobile platforms. 3 2012 US Thanks Giving's Day: 24% of consumers used a mobile device to visit a retailer's site, up from 14.3% in 2011. Mobile sales exceeded 16%, up from 9.8% in 2011.
4. Business Trends in e-commerce mobile application space 4
5. Challenges in testing a mobile application Plethora of devices, OS, Form Factors Multiple devices with different specs OS Fragmentation Tablets and Smart phones Backward compatibility to OS versions App store certification guidelines Major app stores has their own 5 Major app stores has their own guidelines which need to be adhered to Compliance guidelines (E.g.: UTC from Android forum) Rapid Rollout Schedule Rapid rollout of application Improvements and bug fixes. Multiple versions & frequent releases Optimized QA cycles Adequate coverage in limited time
6. An Approach to Minimize Testing Effort/Budget Scenario 1 - Mobile Hybrid e-Commerce Application Factor Ideal Scenario No. of device combinations 150 No. of test cases 100 Execution time per cycle 7500 Optimized scenario Comments 10 Popular smart phones, tablets on various OS flavors 100 500 At the rate of 0.5 hrs per 6 OS flavors & versions Resolution Speed RAM Features User base & Statistics Execution time per cycle (hrs) 7500 Execution time in PMs 46.9 Criteria for smart device selection Edge & Commonality Strategy 500 At the rate of 0.5 hrs per test case 3.1 Effort Reduction by 15 times (Indicative)
7. An Approach to Minimize Testing Effort/Budget Scenario 2 - Mobile Web Site of an e-Commerce Store Factor Ideal scenario No. of device combinations 50 No. of test cases 100 Execution time per cycle (hrs) 2500 Optimized Scenario Comments 5 Popular browsers on various platforms 100 250 At the rate of 0.5 hrs per test case 7 OS flavors & versions Resolution Browser type User base & Statistics (hrs) 2500 Execution time in PMs 15.6 Criteria for smart device selection Edge & Commonality Strategy 250 test case 1.55 Effort Reduction by 10 times (Indicative)
8. Factors for Smart Device Selection Factor Selection Criteria OS flavors & versions Select top 2 versions of the operating system based on usage statistics. Resolution Device models with mix of resolution (One device in each of the resolution categories – LDPI(120), MDPI(160), HDPI(240), XHDPI(320)) Speed One device with low CPU speed(500 MHz), one with medium CPU speed(800 MHz) and one with high CPU speed (1200 MHz). RAM One device with low RAM (128 MB), one with medium sized RAM (512 MHz) 8 RAM One device with low RAM (128 MB), one with medium sized RAM (512 MHz) and one with high RAM size (1 GB). Features Select one or more devices that has commonly available features that most of the devices has(Camera, GPS, accelerometer) to conduct network related testing, interrupt testing etc User base Website analytics data available in the market provides the list of device models used for online shopping & their popularity. From this data, the top 3 (indicative) devices can be selected. Browser type & Version Depending on the OS, top 2 browser versions can be included
9. Tools & Techniques 9
10. Best Practices Emulators Majority of the functional testing can be accomplished on emulators. Usage emulators saves considerable amount of testing budget. RDAs RDA subscription provides access to wide variety of devices with less cost. Include one or two real devices into testing to eliminate the limitations of RDAs Tools Utilize online tools for mobile web compliance testing like GOMO, DotMobi, iphoney, W3C mobileOK checker Early Testing Initiate testing during the early stages of development to avoid critical issues 10 Initiate testing during the early stages of development to avoid critical issues in final stages. Trainings Mobile application testers should have knowledge about devices, Emulators, Device specifications, features, standards, browser specifications and settings. Process A quality standard process document should be prepared and published with all testing guidelines, and it should be followed strictly by the test team. Non functional Testing The performance (installation, launch, navigation etc), non-functional (battery consumption, memory consumption etc) should be verified before certification
11. Testing phases – Strategy to be adopted Type of Testing Manual Testing Automated Testing Using Devices Using Emulators Unit Testing No Yes No Integration Testing No Yes No 11 System Testing Yes Partial No Regression Testing Yes Partial Yes Compatibility Testing Yes No Yes GUI Testing Yes No No
12. Case Study – Largest Futures Brokerage Firm in USA Client’s Problem Statement • To support the Hybrid trading application on various iOS and android versions • App should attract new clients who demand a premiere mobile experience • Extensive Test coverage of Apps on devices & OS • Very aggressive deadlines • No clue on devices to be considered for testing • To support the Hybrid trading application on various iOS and android versions • App should attract new clients who demand a premiere mobile experience • Extensive Test coverage of Apps on devices & OS • Very aggressive deadlines • No clue on devices to be considered for testing • Smart device selection applied • Factors : Client web analytics, Form Factors, Os versions, Hard ware specifications, Network carriers. • Smart device selection applied • Factors : Client web analytics, Form Factors, Os versions, Hard ware specifications, Network carriers. • Samsung Galaxy S III • Galaxy Nexus • Samsung Galaxy S III • Galaxy Nexus Device Set 12 Approach Network carriers. • After applying smart device selection technique, the devices list is optimized to 15 • Extensive usage of DAW services Network carriers. • After applying smart device selection technique, the devices list is optimized to 15 • Extensive usage of DAW services Benefits • Reduced testing cost through smart device selection • Reduced list of devices ( from 150 to 15) • Achieved extensive test coverage • Testing on real devices using DAW • Reusable test cases helped in reduction of test design efforts • Reduced testing cost through smart device selection • Reduced list of devices ( from 150 to 15) • Achieved extensive test coverage • Testing on real devices using DAW • Reusable test cases helped in reduction of test design efforts • Galaxy Nexus • Samsung Galaxy S II • Motorola Droid • HTC inspire 4G • Samsung I997 Infuse 4G • HTC Hero • Samsung Intercept • Huawei Fusion U8652 • iPhone 3G • iPhone 4 & iPhone 4S • iPad 2, 3 & 4 • Galaxy Nexus • Samsung Galaxy S II • Motorola Droid • HTC inspire 4G • Samsung I997 Infuse 4G • HTC Hero • Samsung Intercept • Huawei Fusion U8652 • iPhone 3G • iPhone 4 & iPhone 4S • iPad 2, 3 & 4
13. Case Study – Leading Consumer Electronics Retailer in USA Client’s Problem Statement • To support e-commerce web site on all popular mobile devices & mobile browsers • Aggressive deadlines • To support e-commerce web site on all popular mobile devices & mobile browsers • Aggressive deadlines Smart device • Smart device selection applied • Factors : Client web analytics, Form Factors, OS versions, Mobile browsers. • Smart device selection applied • Factors : Client web analytics, Form Factors, OS versions, Mobile browsers. Device Set 13 Smart device Selection OS versions, Mobile browsers. • After applying smart device selection technique, the devices list is optimized to 6 • Extensive usage of DAW services OS versions, Mobile browsers. • After applying smart device selection technique, the devices list is optimized to 6 • Extensive usage of DAW services Benefits • Reduced testing cost through smart device selection • Reduced list of devices ( from 150 to 6) • Achieved extensive test coverage • Testing on real devices using DAW • Reduced testing cost through smart device selection • Reduced list of devices ( from 150 to 6) • Achieved extensive test coverage • Testing on real devices using DAW • Samsung Galaxy S III • Motorola Droid • HTC Hero • Huawei Fusion U8652 • iPhone 4 • iPad 3 • Samsung Galaxy S III • Motorola Droid • HTC Hero • Huawei Fusion U8652 • iPhone 4 • iPad 3
14. Questions
15. Thank You