How we continuously delivery LINE TODAY App with high agility and high quality

Kuan-Wei Lin
LINE Taiwan Senior Quality Engineer
https://linedevday.linecorp.com/jp/2019/sessions/F2-4

Transcript

1. 2019 DevDay How We Continuously Delivery LINE TODAY App With High Agility and High Quality > Kuan-Wei Lin > LINE Taiwan Senior Quality Engineer
2. Kuan-Wei Lin About Me LINE LINE SDK LINE MUSIC (Taiwan) LINE TODAY
3. LINE TODAY App News Movie Entertainment Video Personalization
4. CI/CD Puzzle iOS Android
5. CI/CD Puzzle Pull Request Code Review Build Unit Tests Integration / API Tests iOS Android Parallel Tasks
6. CI/CD Puzzle Pull Request Code Review Build Unit Tests Internal Deploy External Deploy UI Tests Integration / API Tests Release Parallel Tasks Regression Tests Exploratory Tests Device Lab Parallel Tasks iOS Android
7. CI/CD Puzzle Pull Request Code Review Build Unit Tests Internal Deploy External Deploy UI Tests Integration / API Tests Release Regression Tests Exploratory Tests Automation Parallel Tasks Device Lab Parallel Tasks iOS Android
8. Automation Pull Request Code Review Build Unit Tests Integration / API Tests Parallel Tasks External Deploy UI Tests Release Regression Tests Exploratory Tests Device Lab Parallel Tasks iOS Android CI/CD Puzzle Internal Deploy xcodebuild test -only-testing:Tests ./gradlew assemble testUnitTest
9. Automation Parallel Tasks Internal Deploy External Deploy UI Tests Release Regression Tests Exploratory Tests Device Lab Parallel Tasks iOS Android Integration / API Tests Code Review Build Unit Tests CI/CD Puzzle Pull Request xcodebuild test -only-testing:UITests -destination* ./gradlew connectedAndroidTest
10. CI/CD Puzzle Pull Request Code Review Build Unit Tests Internal Deploy External Deploy UI Tests Integration / API Tests Release Regression Tests Exploratory Tests Device Lab Parallel Tasks iOS Android Parallel Tasks Automation
11. CI/CD Puzzle Pull Request Code Review Build Unit Tests Internal Deploy External Deploy UI Tests Integration / API Tests Release Regression Tests Exploratory Tests Device Lab Parallel Tasks iOS Android Parallel Tasks Automation
12. Unit Tests UI Tests > What Is Good Code Coverage To Have? > How To Have Reliable UI Automation Tests?
13. Agenda > UI Automation Tests Guideline > Unit Tests & Code Coverage Strategy Unit Tests UI Tests
14. Number of Tests Execute Time Unit Integration User Interface
15. Testing Spectrum Integration / API Tests UI Tests Regression Tests Unit Tests Unit End To End Resource • API automation testing • Load testing • Database integration tests • Native test frameworks • Device lab • UI automation guideline • Agile testing • Exploratory testing • Logic & UI separation • Test double (Stub/Mock/Fake) • TDD (Test-Driven Development) • Code coverage strategy Automation Manual
16. UI Automation Tests Guideline
17. Flaky Tests
18. Why Flaky Tests Are Blockers? 1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th Test A Test B Test C … … Result Automation Runs
19. Why Flaky Tests Are Blockers? 1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th Test A Test B Test C … … Result Automation Runs
20. 1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th Test A ? Test B ? Test C ? … 100 More UI Tests? Result ? Why Flaky Tests Are Blockers? Automation Runs
21. UI Automation Testing Is… Easy To Develop but Expensive To Maintain
22. > Known issues of test frameworks (XCUITest / Espresso) > Different device dependency > Cross App testing instability > Parallel testing instability > IDE run v.s. Command line run > And more… External Flaky Factors > UI may change in every release > Test environment instability > Test data instability > Dynamic UI elements design > Poorly written test code Internal Flaky Factors UI Automation Testing Vulnerability
23. Be Aware of Maintenance Cost and Flaky Tests Before Starting UI Automation Tests UI Automation Testing Is Smoke Testing
24. Acceptance Criteria Search Bar Search Result Filter Tabs
25. Test Suite Layer Page Object Layer SearchPage MainPage Acceptance Criteria
26. Smoke Cases • Most common user scenarios • Mature functionality • Suitable to automate Regression Cases • Features which are keep developing • Hard / High cost to automate UI Automation Tests Regression Tests Automation Manual Acceptance Criteria
27. 0% 20% 40% 60% 80% 0 400 800 1200 1600 App Versions 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 Manual Regression Tests UI Automation tests Ratio 12.9% 11% 20.1% 19.3% 30.5% 38.2% 43% 51.5% 50.1% 50.4% 914 950 1,279 1,519 1,186 1,074 1,054 536 622 400
28. Unit Tests & Code Coverage Strategy
29. Unit Tests UI Tests Logic Classes UI Classes Code Coverage
30. API Server API + MVC API Client Models Controllers View User action Update Notify Update Notify / Parsing Update Unit Tests UI Tests SERVER CLIENT Request Response
31. API Server Models Notify UIViewController View Controller API Client Notify / Parsing Update ? SERVER CLIENT iOS MVC Request Response Unit Tests UI Tests
32. API Server iOS MVVM Models Notify UIViewController View Controller ViewModel Binding API Client Update Notify / Parsing SERVER CLIENT Request Response Unit Tests UI Tests
33. Models UIViewController View Controller ViewModel API Client Unit Tests UI Tests Manager Helper DB Animation Cell View iOS Code Coverage Formula
34. Code Coverage Unit Tests UI Tests Functionalities UI
35. Code Coverage Functionalities ViewModel Model API Client Manager Helper… UI ViewController View Cell Animation… Unit Tests UI Tests
36. Code Coverage UI Not in UI Tests Functionalities ViewModel Model API Client Manager Helper… UI in UI Tests ViewController View Cell Animation… Unit Tests UI Tests
37. Code Coverage New Functionalities New UI Functionalities ViewModel Model API Client Manager Helper… Unit Tests UI Tests UI Not in UI Tests UI in UI Tests ViewController View Cell Animation…
38. Code Coverage New UI Not in UI Tests New UI in UI Tests Functionalities ViewModel Model API Client Manager Helper… New Functionalities Unit Tests UI Tests UI Not in UI Tests UI in UI Tests ViewController View Cell Animation…
39. Code Coverage System Callback, Extension, Utility… Functionalities ViewModel Model API Client Manager Helper… New Functionalities Unit Tests UI Tests New UI Not in UI Tests New UI in UI Tests UI Not in UI Tests UI in UI Tests ViewController View Cell Animation…
40. Code Coverage New UI in UI Tests Functionalities ViewModel Model API Client Manager Helper… System Callback, Extension, Utility… New Functionalities Unit Tests UI Tests UI in UI Tests ViewController View Cell Animation… New UI Not in UI Tests UI Not in UI Tests
41. Code Coverage ?% Unit Tests UI Tests System Callback, Extension, Utility… New UI Not in UI Tests UI Not in UI Tests
42. Code Coverage ?% Unit Tests UI Tests ?%
43. Code Coverage Unit Tests UI Tests X%
44. Code Coverage Unit Tests UI Tests Y% X%
45. Code Coverage Unit Tests UI Tests Y% X% > New Code > Refactoring
46. Number of Tests Execute Time Unit Integration User Interface
47. Unit Integration Fatal Issues Functional Issues UI Issues Edge case Issues User Interface
48. Unit Integration Fatal Issues Functional Issues UI Issues User Interface Edge case Issues
49. Unit Integration User Interface Fatal Issues Functional Issues UI Issues Edge case Issues
50. Unit Integration User Interface Fatal Issues Functional Issues UI Issues Regression Tests Exploratory Tests Automation Manual Edge case Issues
51. CI/CD Pipeline | Testing Spectrum In Progress Other Projects Now > UI automation tests guideline > Unit tests & code coverage strategy • Web Application • … LINE TODAY LINE LINE SDK LINE MUSIC (Taiwan) > Automated pipeline
52. Thank You