How we continuously delivery LINE TODAY App with high agility and high quality

Slide 1

Slide 1 text

2019 DevDay How We Continuously Delivery LINE TODAY App With High Agility and High Quality > Kuan-Wei Lin > LINE Taiwan Senior Quality Engineer

Slide 2

Slide 2 text

Kuan-Wei Lin About Me LINE LINE SDK LINE MUSIC (Taiwan) LINE TODAY

Slide 3

Slide 3 text

LINE TODAY App News Movie Entertainment Video Personalization

Slide 4

Slide 4 text

CI/CD Puzzle iOS Android

Slide 5

Slide 5 text

CI/CD Puzzle Pull Request Code Review Build Unit Tests Integration / API Tests iOS Android Parallel Tasks

Slide 6

Slide 6 text

CI/CD Puzzle Pull Request Code Review Build Unit Tests Internal Deploy External Deploy UI Tests Integration / API Tests Release Parallel Tasks Regression Tests Exploratory Tests Device Lab Parallel Tasks iOS Android

Slide 7

Slide 7 text

CI/CD Puzzle Pull Request Code Review Build Unit Tests Internal Deploy External Deploy UI Tests Integration / API Tests Release Regression Tests Exploratory Tests Automation Parallel Tasks Device Lab Parallel Tasks iOS Android

Slide 8

Slide 8 text

Automation Pull Request Code Review Build Unit Tests Integration / API Tests Parallel Tasks External Deploy UI Tests Release Regression Tests Exploratory Tests Device Lab Parallel Tasks iOS Android CI/CD Puzzle Internal Deploy xcodebuild test -only-testing:Tests ./gradlew assemble testUnitTest

Slide 9

Slide 9 text

Automation Parallel Tasks Internal Deploy External Deploy UI Tests Release Regression Tests Exploratory Tests Device Lab Parallel Tasks iOS Android Integration / API Tests Code Review Build Unit Tests CI/CD Puzzle Pull Request xcodebuild test -only-testing:UITests -destination* ./gradlew connectedAndroidTest

Slide 10

Slide 10 text

CI/CD Puzzle Pull Request Code Review Build Unit Tests Internal Deploy External Deploy UI Tests Integration / API Tests Release Regression Tests Exploratory Tests Device Lab Parallel Tasks iOS Android Parallel Tasks Automation

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Unit Tests UI Tests > What Is Good Code Coverage To Have? > How To Have Reliable UI Automation Tests?

Slide 13

Slide 13 text

Agenda > UI Automation Tests Guideline > Unit Tests & Code Coverage Strategy Unit Tests UI Tests

Slide 14

Slide 14 text

Number of Tests Execute Time Unit Integration User Interface

Slide 15

Slide 15 text

Testing Spectrum Integration / API Tests UI Tests Regression Tests Unit Tests Unit End To End Resource • API automation testing • Load testing • Database integration tests • Native test frameworks • Device lab • UI automation guideline • Agile testing • Exploratory testing • Logic & UI separation • Test double (Stub/Mock/Fake) • TDD (Test-Driven Development) • Code coverage strategy Automation Manual

Slide 16

Slide 16 text

UI Automation Tests Guideline

Slide 17

Slide 17 text

Flaky Tests

Slide 18

Slide 18 text

Why Flaky Tests Are Blockers? 1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th Test A Test B Test C … … Result Automation Runs

Slide 19

Slide 19 text

Why Flaky Tests Are Blockers? 1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th Test A Test B Test C … … Result Automation Runs

Slide 20

Slide 20 text

1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th Test A ? Test B ? Test C ? … 100 More UI Tests? Result ? Why Flaky Tests Are Blockers? Automation Runs

Slide 21

Slide 21 text

UI Automation Testing Is… Easy To Develop but Expensive To Maintain

Slide 22

Slide 22 text

> Known issues of test frameworks (XCUITest / Espresso) > Different device dependency > Cross App testing instability > Parallel testing instability > IDE run v.s. Command line run > And more… External Flaky Factors > UI may change in every release > Test environment instability > Test data instability > Dynamic UI elements design > Poorly written test code Internal Flaky Factors UI Automation Testing Vulnerability

Slide 23

Slide 23 text

Be Aware of Maintenance Cost and Flaky Tests Before Starting UI Automation Tests UI Automation Testing Is Smoke Testing

Slide 24

Slide 24 text

Acceptance Criteria Search Bar Search Result Filter Tabs

Slide 25

Slide 25 text

Test Suite Layer Page Object Layer SearchPage MainPage Acceptance Criteria

Slide 26

Slide 26 text

Smoke Cases • Most common user scenarios • Mature functionality • Suitable to automate Regression Cases • Features which are keep developing • Hard / High cost to automate UI Automation Tests Regression Tests Automation Manual Acceptance Criteria

Slide 27

Slide 27 text

0% 20% 40% 60% 80% 0 400 800 1200 1600 App Versions 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 Manual Regression Tests UI Automation tests Ratio 12.9% 11% 20.1% 19.3% 30.5% 38.2% 43% 51.5% 50.1% 50.4% 914 950 1,279 1,519 1,186 1,074 1,054 536 622 400

Slide 28

Slide 28 text

Unit Tests & Code Coverage Strategy

Slide 29

Slide 29 text

Unit Tests UI Tests Logic Classes UI Classes Code Coverage

Slide 30

Slide 30 text

API Server API + MVC API Client Models Controllers View User action Update Notify Update Notify / Parsing Update Unit Tests UI Tests SERVER CLIENT Request Response

Slide 31

Slide 31 text

API Server Models Notify UIViewController View Controller API Client Notify / Parsing Update ? SERVER CLIENT iOS MVC Request Response Unit Tests UI Tests

Slide 32

Slide 32 text

API Server iOS MVVM Models Notify UIViewController View Controller ViewModel Binding API Client Update Notify / Parsing SERVER CLIENT Request Response Unit Tests UI Tests

Slide 33

Slide 33 text

Models UIViewController View Controller ViewModel API Client Unit Tests UI Tests Manager Helper DB Animation Cell View iOS Code Coverage Formula

Slide 34

Slide 34 text

Code Coverage Unit Tests UI Tests Functionalities UI

Slide 35

Slide 35 text

Code Coverage Functionalities ViewModel Model API Client Manager Helper… UI ViewController View Cell Animation… Unit Tests UI Tests

Slide 36

Slide 36 text

Code Coverage UI Not in UI Tests Functionalities ViewModel Model API Client Manager Helper… UI in UI Tests ViewController View Cell Animation… Unit Tests UI Tests

Slide 37

Slide 37 text

Code Coverage New Functionalities New UI Functionalities ViewModel Model API Client Manager Helper… Unit Tests UI Tests UI Not in UI Tests UI in UI Tests ViewController View Cell Animation…

Slide 38

Slide 38 text

Code Coverage New UI Not in UI Tests New UI in UI Tests Functionalities ViewModel Model API Client Manager Helper… New Functionalities Unit Tests UI Tests UI Not in UI Tests UI in UI Tests ViewController View Cell Animation…

Slide 39

Slide 39 text

Code Coverage System Callback, Extension, Utility… Functionalities ViewModel Model API Client Manager Helper… New Functionalities Unit Tests UI Tests New UI Not in UI Tests New UI in UI Tests UI Not in UI Tests UI in UI Tests ViewController View Cell Animation…

Slide 40

Slide 40 text

Code Coverage New UI in UI Tests Functionalities ViewModel Model API Client Manager Helper… System Callback, Extension, Utility… New Functionalities Unit Tests UI Tests UI in UI Tests ViewController View Cell Animation… New UI Not in UI Tests UI Not in UI Tests

Slide 41

Slide 41 text

Code Coverage ?% Unit Tests UI Tests System Callback, Extension, Utility… New UI Not in UI Tests UI Not in UI Tests

Slide 42

Slide 42 text

Code Coverage ?% Unit Tests UI Tests ?%

Slide 43

Slide 43 text

Code Coverage Unit Tests UI Tests X%

Slide 44

Slide 44 text

Code Coverage Unit Tests UI Tests Y% X%

Slide 45

Slide 45 text

Code Coverage Unit Tests UI Tests Y% X% > New Code > Refactoring

Slide 46

Slide 46 text

Number of Tests Execute Time Unit Integration User Interface

Slide 47

Slide 47 text

Unit Integration Fatal Issues Functional Issues UI Issues Edge case Issues User Interface

Slide 48

Slide 48 text

Unit Integration Fatal Issues Functional Issues UI Issues User Interface Edge case Issues

Slide 49

Slide 49 text

Unit Integration User Interface Fatal Issues Functional Issues UI Issues Edge case Issues

Slide 50

Slide 50 text

Unit Integration User Interface Fatal Issues Functional Issues UI Issues Regression Tests Exploratory Tests Automation Manual Edge case Issues

Slide 51

Slide 51 text

CI/CD Pipeline | Testing Spectrum In Progress Other Projects Now > UI automation tests guideline > Unit tests & code coverage strategy • Web Application • … LINE TODAY LINE LINE SDK LINE MUSIC (Taiwan) > Automated pipeline

Slide 52

Slide 52 text

Thank You