How we continuously delivery LINE TODAY App with high agility and high quality

How we continuously delivery LINE TODAY App with high agility and high quality

Kuan-Wei Lin
LINE Taiwan Senior Quality Engineer
https://linedevday.linecorp.com/jp/2019/sessions/F2-4

Be4518b119b8eb017625e0ead20f8fe7?s=128

LINE DevDay 2019

November 21, 2019
Tweet

Transcript

  1. 2019 DevDay How We Continuously Delivery LINE TODAY App With

    High Agility and High Quality > Kuan-Wei Lin > LINE Taiwan Senior Quality Engineer
  2. Kuan-Wei Lin About Me LINE LINE SDK LINE MUSIC (Taiwan)

    LINE TODAY
  3. LINE TODAY App News Movie Entertainment Video Personalization

  4. CI/CD Puzzle iOS Android

  5. CI/CD Puzzle Pull Request Code Review Build Unit Tests Integration

    / API Tests iOS Android Parallel Tasks
  6. CI/CD Puzzle Pull Request Code Review Build Unit Tests Internal

    Deploy External Deploy UI Tests Integration / API Tests Release Parallel Tasks Regression Tests Exploratory Tests Device Lab Parallel Tasks iOS Android
  7. CI/CD Puzzle Pull Request Code Review Build Unit Tests Internal

    Deploy External Deploy UI Tests Integration / API Tests Release Regression Tests Exploratory Tests Automation Parallel Tasks Device Lab Parallel Tasks iOS Android
  8. Automation Pull Request Code Review Build Unit Tests Integration /

    API Tests Parallel Tasks External Deploy UI Tests Release Regression Tests Exploratory Tests Device Lab Parallel Tasks iOS Android CI/CD Puzzle Internal Deploy xcodebuild test -only-testing:Tests ./gradlew assemble testUnitTest
  9. Automation Parallel Tasks Internal Deploy External Deploy UI Tests Release

    Regression Tests Exploratory Tests Device Lab Parallel Tasks iOS Android Integration / API Tests Code Review Build Unit Tests CI/CD Puzzle Pull Request xcodebuild test -only-testing:UITests -destination* ./gradlew connectedAndroidTest
  10. CI/CD Puzzle Pull Request Code Review Build Unit Tests Internal

    Deploy External Deploy UI Tests Integration / API Tests Release Regression Tests Exploratory Tests Device Lab Parallel Tasks iOS Android Parallel Tasks Automation
  11. CI/CD Puzzle Pull Request Code Review Build Unit Tests Internal

    Deploy External Deploy UI Tests Integration / API Tests Release Regression Tests Exploratory Tests Device Lab Parallel Tasks iOS Android Parallel Tasks Automation
  12. Unit Tests UI Tests > What Is Good Code Coverage

    To Have? > How To Have Reliable UI Automation Tests?
  13. Agenda > UI Automation Tests Guideline > Unit Tests &

    Code Coverage Strategy Unit Tests UI Tests
  14. Number of Tests Execute Time Unit Integration User Interface

  15. Testing Spectrum Integration / API Tests UI Tests Regression Tests

    Unit Tests Unit End To End Resource • API automation testing • Load testing • Database integration tests • Native test frameworks • Device lab • UI automation guideline • Agile testing • Exploratory testing • Logic & UI separation • Test double (Stub/Mock/Fake) • TDD (Test-Driven Development) • Code coverage strategy Automation Manual
  16. UI Automation Tests Guideline

  17. Flaky Tests

  18. Why Flaky Tests Are Blockers? 1st 2nd 3rd 4th 5th

    6th 7th 8th 9th 10th Test A Test B Test C … … Result Automation Runs
  19. Why Flaky Tests Are Blockers? 1st 2nd 3rd 4th 5th

    6th 7th 8th 9th 10th Test A Test B Test C … … Result Automation Runs
  20. 1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th

    Test A ? Test B ? Test C ? … 100 More UI Tests? Result ? Why Flaky Tests Are Blockers? Automation Runs
  21. UI Automation Testing Is… Easy To Develop but Expensive To

    Maintain
  22. > Known issues of test frameworks (XCUITest / Espresso) >

    Different device dependency > Cross App testing instability > Parallel testing instability > IDE run v.s. Command line run > And more… External Flaky Factors > UI may change in every release > Test environment instability > Test data instability > Dynamic UI elements design > Poorly written test code Internal Flaky Factors UI Automation Testing Vulnerability
  23. Be Aware of Maintenance Cost and Flaky Tests Before Starting

    UI Automation Tests UI Automation Testing Is Smoke Testing
  24. Acceptance Criteria Search Bar Search Result Filter Tabs

  25. Test Suite Layer Page Object Layer SearchPage MainPage Acceptance Criteria

  26. Smoke Cases • Most common user scenarios • Mature functionality

    • Suitable to automate Regression Cases • Features which are keep developing • Hard / High cost to automate UI Automation Tests Regression Tests Automation Manual Acceptance Criteria
  27. 0% 20% 40% 60% 80% 0 400 800 1200 1600

    App Versions 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 Manual Regression Tests UI Automation tests Ratio 12.9% 11% 20.1% 19.3% 30.5% 38.2% 43% 51.5% 50.1% 50.4% 914 950 1,279 1,519 1,186 1,074 1,054 536 622 400
  28. Unit Tests & Code Coverage Strategy

  29. Unit Tests UI Tests Logic Classes UI Classes Code Coverage

  30. API Server API + MVC API Client Models Controllers View

    User action Update Notify Update Notify / Parsing Update Unit Tests UI Tests SERVER CLIENT Request Response
  31. API Server Models Notify UIViewController View Controller API Client Notify

    / Parsing Update ? SERVER CLIENT iOS MVC Request Response Unit Tests UI Tests
  32. API Server iOS MVVM Models Notify UIViewController View Controller ViewModel

    Binding API Client Update Notify / Parsing SERVER CLIENT Request Response Unit Tests UI Tests
  33. Models UIViewController View Controller ViewModel API Client Unit Tests UI

    Tests Manager Helper DB Animation Cell View iOS Code Coverage Formula
  34. Code Coverage Unit Tests UI Tests Functionalities UI

  35. Code Coverage Functionalities ViewModel Model API Client Manager Helper… UI

    ViewController View Cell Animation… Unit Tests UI Tests
  36. Code Coverage UI Not in UI Tests Functionalities ViewModel Model

    API Client Manager Helper… UI in UI Tests ViewController View Cell Animation… Unit Tests UI Tests
  37. Code Coverage New Functionalities New UI Functionalities ViewModel Model API

    Client Manager Helper… Unit Tests UI Tests UI Not in UI Tests UI in UI Tests ViewController View Cell Animation…
  38. Code Coverage New UI Not in UI Tests New UI

    in UI Tests Functionalities ViewModel Model API Client Manager Helper… New Functionalities Unit Tests UI Tests UI Not in UI Tests UI in UI Tests ViewController View Cell Animation…
  39. Code Coverage System Callback, Extension, Utility… Functionalities ViewModel Model API

    Client Manager Helper… New Functionalities Unit Tests UI Tests New UI Not in UI Tests New UI in UI Tests UI Not in UI Tests UI in UI Tests ViewController View Cell Animation…
  40. Code Coverage New UI in UI Tests Functionalities ViewModel Model

    API Client Manager Helper… System Callback, Extension, Utility… New Functionalities Unit Tests UI Tests UI in UI Tests ViewController View Cell Animation… New UI Not in UI Tests UI Not in UI Tests
  41. Code Coverage ?% Unit Tests UI Tests System Callback, Extension,

    Utility… New UI Not in UI Tests UI Not in UI Tests
  42. Code Coverage ?% Unit Tests UI Tests ?%

  43. Code Coverage Unit Tests UI Tests X%

  44. Code Coverage Unit Tests UI Tests Y% X%

  45. Code Coverage Unit Tests UI Tests Y% X% > New

    Code > Refactoring
  46. Number of Tests Execute Time Unit Integration User Interface

  47. Unit Integration Fatal Issues Functional Issues UI Issues Edge case

    Issues User Interface
  48. Unit Integration Fatal Issues Functional Issues UI Issues User Interface

    Edge case Issues
  49. Unit Integration User Interface Fatal Issues Functional Issues UI Issues

    Edge case Issues
  50. Unit Integration User Interface Fatal Issues Functional Issues UI Issues

    Regression Tests Exploratory Tests Automation Manual Edge case Issues
  51. CI/CD Pipeline | Testing Spectrum In Progress Other Projects Now

    > UI automation tests guideline > Unit tests & code coverage strategy • Web Application • … LINE TODAY LINE LINE SDK LINE MUSIC (Taiwan) > Automated pipeline
  52. Thank You