$30 off During Our Annual Pro Sale. View Details »

Testing in weekly release

Testing in weekly release

Kyohei Kato

April 18, 2019
Tweet

More Decks by Kyohei Kato

Other Decks in Technology

Transcript

  1. Testing in weekly release
    Cookpad Inc.
    Tech dept, Quality Improvement Group
    Kyohei Kato
    1
    週次リリースにおけるテストとは

    View Slide

  2. Introduction
    ● Kyohei Kato (Cookpad Inc.)
    ● Quality Improvement Group
    (Tech department)
    ● Software Engineer in Quality
    ● e.g. Implementing automated UI tests,
    QA, building test environment in mobile
    apps
    2

    View Slide

  3. Agenda
    ● What is Cookpad
    ● Weekly release in mobile apps development
    ● What is testing in weekly release
    ● Problems in testing through weekly release
    ● Ideal image of testing
    3

    View Slide

  4. What is Cookpad
    ● Recipe service originally web-only
    ● Posting and searching recipes
    ● Released apps in Android and iOS
    ● Have been doing weekly releases for half a
    year
    4

    View Slide

  5. Weekly release
    5

    View Slide

  6. The concept is “Human adjusts to machine”
    ● Release following a fixed schedule
    ● Release tasks are run automatically
    ● This automatic scheduling is set every Friday at 2 AM
    ○ Week is the best so far
    6
    クックパッドアプリはみんなが寝ている間にサブミットされる(
    https://techlife.cookpad.com/entry/2018/09/14/090000)

    1
    1

    View Slide

  7. 7

    View Slide

  8. ● Changes merged until Friday will be target for release of next week
    ● Then the app containing changes will progress to just before release
    ● Necessary tasks for release are automated
    ○ Automate tasks such as submitting reviews to Apple on iOS
    ○ Progress to the condition that the app is released just by pushing a release button
    ● People decide whether to release or skip
    ○ The department responsible for service development takes the lead in making decisions
    ● Develop apps back from the release time when you develop
    Main development flow
    8

    View Slide

  9. View Slide

  10. Whether to release or skip
    ● App review by Apple (iOS only)
    ● Checking changes by each developers
    ○ Check their own changes
    ● Pre-release testing by Quality Improvement Group
    ○ It takes a few days
    10

    View Slide

  11. View Slide

  12. What is pre-release testing
    12

    View Slide

  13. Quality Improvement Group
    ● Group focusing on quality
    ● In relation with all departments contributing to the Cookpad mobile apps
    ● Mainly responsible for Cookpad mobile apps
    ○ Background and other movement will be omitted
    ● Implementing automated UI tests, QA, building test environment
    13

    View Slide

  14. 14
    Quality Improvement Group
    ● Implement automated UI tests
    ● QA
    ● Build testing environment

    View Slide

  15. What we do in pre-release testing
    ● Combination of automated UI testing and manual testing
    a. Regression tests to check the function by automated UI test
    b. Tests difficult to automate are done manually
    c. Manual exploratory testing against changes for each changes
    15

    View Slide

  16. a. Automated UI tests
    ● Automated UI tests using Appium
    ● Appium is a test automation tool based on Selenium widely used for
    mobile apps or PCs
    ○ Both emulator (simulator) and real device can be used
    ● Regression test to check existing features
    ○ e.g. login, recipe search, recipe browsing
    16

    View Slide

  17. Check UI difference with image comparison
    ● Take screenshots in automated UI tests
    ● Compare with past ones with human eyes
    17

    View Slide

  18. 18

    View Slide

  19. b.Tests that cannot be covered by
    automated test
    ● Manually cover technically difficult tests that are difficult to automate
    ○ Payment flow due to complex status of user status or device account
    ■ In-app purchase, carrier payments…
    ■ Difficult to automate
    19

    View Slide

  20. c. Exploratory testing against changes
    ● Of course each development team is testing
    ○ Assuming that each team tests normal use cases
    ● Exploratory testing against changes
    ○ Confirm specifications of changes from issues and developers
    ○ Check abnormal use cases and the impact on related features
    20

    View Slide

  21. Summary up to here
    ● Weekly releases improve release frequency
    ○ It became possible to turn service development quickly with short cycles
    ● Pre-release testing by Quality Improvement Group
    ○ Pre-release testing is one of the release decision materials
    ○ Combination of automatic and manual
    ○ Need to be done weekly for weekly release
    21

    View Slide

  22. Problems of current testing
    22

    View Slide

  23. Quality Improvement Group approach
    ● Get information about development from issue or directly from developers
    ○ Both current changes and future developments
    ● That information is used for pre-release testing
    ○ What does regression testing cover with automated UI testing?
    ○ What are unintended UI changes?
    ○ What should I focus on in exploratory testing?
    23

    View Slide

  24. View Slide

  25. Problems that could have been avoided
    ● Failure to understand the background of specifications
    ● It leads to omit to consider of complex user status
    ● Aware problems in the specification after implementation
    ○ It was a problem that could be understood from the viewpoint of conducting the test
    ○ It was finally discovered in the test stage
    25

    View Slide

  26. Problems emerging due to weekly release
    schedule
    ● A lot of problems presented here are not limited to weekly releases
    ○ They will occur if the locations responsible for development and testing are separated
    ● Improvement of development speed with weekly release raises the
    difficulty to keep up
    ● The speed of development was not enough for problems to emerge until
    now
    ○ Release once a month -> once a week
    26

    View Slide

  27. 27
    Four times the number of releases in the same period
    (about 3 months)

    View Slide

  28. Why does these problems come up?
    28

    View Slide

  29. Testing is a part of service development
    ● We cannot know the truth of service development unless we participate in it
    ○ Service development is not only the following flow:
    (specification -> implementation -> test -> release)
    ■ How to analyze user issues
    ■ How to solve the problem
    ■ How to find the solution
    ■ How is the solution realized (here)
    ■ etc…
    ● We cannot understand service development just by being involved in only a part
    of this flow
    29

    View Slide

  30. Limit of communication
    ● We cannot understand service development
    only by issues and occasional communication
    ● Especially in fast development speed such as
    weekly release schedule
    ● Getting closed to limit of communication
    30

    View Slide

  31. Ideal image of testing
    31

    View Slide

  32. Testing in service development
    ● To keep up, you should run together
    ● Image that members of Quality Improvement
    Group are part of service development
    32

    View Slide

  33. Change testing and also service development
    ● Make service development think about quality and testing
    ○ What affects the quality of the release, what are defects?
    ○ What quality do you want to guarantee in service development?
    ○ Awaken implicit expectations sleeping inside service development
    ■ I don’t want a to put out a bug which make recipe sharing impossible!!!
    ■ I want to protect the target user range for measures!!
    33

    View Slide

  34. View Slide

  35. View Slide

  36. Required moves
    ● People who think about quality as part of service development
    ○ Think of quality as a member of service development rather than a different team
    ○ Always think about what points of quality to focus on testing in service development
    ○ Raise issues that are likely to be barriers to specification and implementation
    ○ Ability to propose flexible testing
    ● Close to my idea of an “agile tester”
    36

    View Slide

  37. System to promote service development
    ● We need people who are responsible for quality and testing as members of
    service development
    ● Break through the limits of communication between service development
    and Quality Improvement Groups
    ● Improve service development speed!!!
    37

    View Slide

  38. Summary
    ● Weekly releasing has been carried out in Cookpad mobile apps
    development
    ● There is pre-release testing by Quality Improvement Group
    ● Limitations of communication between service development and Quality
    Improvement Group through weekly releasing
    ● As a member of service development, we need to think about the quality of
    service and testing against it
    ● To a world where service development that thinks about quality, and a
    testing team can run side by side
    38

    View Slide