Mobile App Testing

Introduce test architecture for mobile app and some useful tools

Transcript

1. Mobile App Testing By Proko Chou
2. About Me • 8+ years mobile application testing experience in HTC and CloudMosa • OS: ◦ Android ◦ iOS ◦ Windows ◦ Brew • Applications: ◦ Browser ◦ Galery (Album) ◦ Camera ◦ Music ◦ Video player ◦ Voice recorder ◦ MMS ◦ ……
3. Agenda • Before Testing: Test Sample • Test Coverage ◦ Basic Function ◦ Compatibility ◦ Integration ◦ Stress ◦ Power Consumption • Q & A
4. Before testing: Prepare Test Sample • Different Type: ◦ Album: png, jpeg, bmp,gif, etc. ◦ Music: mp3, wav, aiff, etc. • Different Size: ◦ Album: 0 bytes, normal size, big size image ◦ Video player: 0 bytes, normal size, big size video • Different Environment: ◦ Camera: yellow light, sparkled item, colorful item ◦ Internet: metallic box, 2G/3G/ 4G SIM, VPN account Sparkled Item Colorful Item Metallic box
5. Before testing: Device Precondition • Create datas on devices under testing (DUT) ◦ Log in mail accounts ◦ Messenger: Line, Whatsapp or Facebook messenger ◦ Many SMS/ MMS messages ◦ Many photos in ablum ◦ Enable lockscreen ◦ …...
6. Test Coverage
7. Basic Function • 使用者介面 (UI / Multi-Language / IME Check) • 邊界 (Boundary) • 記憶體釋放 (Memory leakage) • 錯誤處理 (Error Handling) • 安裝(Install) 、反安裝 (Uninstall)、儲存至SD card (Move to SD card) • 使用者資料保存 (Data Migration) • 重啟 (After relaunching) • 清除App資料 (After clearing application’s data) • 清除整台機器的資料 (After resetting the device)
8. Basic Function • 使用者介面 (UI / Multi-Language / IME Check) • 邊界 (Boundary) • 記憶體釋放 (Memory leakage) • 錯誤處理 (Error Handling) • 重啟 (After relaunching) • 使用者資料保存 (Data Migration) • 取消該功能的使用 (Stop feature function) • 安裝(Install) 、反安裝 (Uninstall) • 清除App資料 (After clearing application’s data) • 清除整台機器的資料 (After resetting the device)
9. Test Coverage: Compatibility
10. Compatibility: Different Devices ◦ Device: ▪ iOS: ◦ iPad: iPad 2 ◦ iPad with Retina: iPad 3 or iPad 4 ◦ iPad with split function: iPad air 2, iPad pro ◦ iPhone ◦ Device with 3D touch function ▪ Android: ◦ Tablet /Phone ◦ Device with Pen and Keyboard ◦ Different devices test on Cloud: (250 mins free) ▪ AWS Test: https://aws.amazon.com/tw/device-farm/
11. Compatibility: Different OS Version • Support old and current version: ◦ iOS: ▪ iOS5~6 ▪ iOS 7 ▪ iOS8~ iOS9 ▪ iOS 10 ◦ Android: ▪ Gingerbread 2.3–2.3.7 ▪ Honeycomb (Tablet only) 3.0–3.2.6 ▪ Ice Cream Sandwich 4.0–4.0.4 ▪ Jelly Bean 4.1–4.3.1 ▪ KitKat 4.4–4.4.4, 4.4W–4.4W.2 ▪ Lollipop 5.0–5.1.1 ▪ Marshmallow 6.0–6.0.1 • Beta version test during July ~ September each year！！
12. Test Coverage: Integration
13. Integration: Internet
14. Integration: Internet (1/2) Airplane Mode VPN Under connecting or loading No Connection (Use Metallic box) Connect via proxy Weak signal (Use Elevator) Normal Connection Data Connection
15. Integration: Internet (2/2) Emulate network connectivity: • Facebook Augmented Traffic Control, ATC: Reference: Link 1, Link 2 • iOS Developer Network Link Conditioner: Reference: Link Facebook ATC iOS Developer Tool
16. Integration: Keyboard • Android: Google, Samsung, Asus, bluetooth keyboard, pen, voice input, etc. • iOS: iAcess, voice input and split keyboard (Link) iOS: Split keyboard Bluetooth keyboard Pen
17. Integration: Other Apps’ functions ◦ Other applications: ▪ Incoming phone call ▪ Alarm ▪ Voice assistant (e.g., Siri) ▪ Album ▪ Camera ◦ Device: ▪ Power off/ Resume device ▪ Support split screen function ▪ Support 3D touch function
18. Test Coverage: Stress and Power Consumption
19. Stability: Stress Test • Keep using over X times • Keep using over X hours • Keep relaunching X times • Criteria: MTBF (Reference: link) • Monkey test ◦ Monkey script for Android: https://developer.android.com/studio/test/monkey.html ◦ Monkey script for iOS: https://github.com/search?utf8=%E2%9C%93&q=ios+monkey
20. Power Consumption • How many hours does an application keep working? • Criteria: Compare to other applications