Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Technology supporting the release of HOME'S

Technology supporting the release of HOME'S

We accelerated release frequency
from two times a week to four times a week.

We achieved this goal by solving some problems in the automated system testing which is executed before release.

Naoto KISHINO

April 19, 2019
Tweet

Other Decks in Technology

Transcript

  1. Technology supporting
    the release of HOME'S
    Naoto Kishino (Software Engineer in Test at LIFULL)
    April 19th, 2019 at Selenium Conf Tokyo

    View full-size slide

  2. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    2
    Introduction
    Naoto Kishino
    (木住野 奈夫人)
    Software Engineer in Test Group
    (3 members)
    Test Automation/CI/CD
    Group vision :
    自動化と高速化を進め、プロとして最高のDXを実現する
    Create the best DX (Developer Experience) in the world.
    @naotospace
    NA070

    View full-size slide

  3. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    3
    Overview
    We accelerated release frequency
    from two times a week to four times a
    week.
    We achieved this goal by solving some
    problems in the automated system testing
    which is executed before release.

    View full-size slide

  4. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    4
    Agenda
    1.About LIFULL HOME’S
    2.About development process
    3.Problems in our automated system testing
    4.Solutions
    5.Conclusion

    View full-size slide

  5. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    5
    About LIFULL HOME’S
    Do you know LIFULL HOME’S?

    View full-size slide

  6. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    6
    About LIFULL HOME’S

    View full-size slide

  7. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    7
    About LIFULL HOME’S

    View full-size slide

  8. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    8
    About LIFULL HOME’S
    ● Information service about Real Estate
    in Japan.
    ● Registration number’s No.1 in Japan

    View full-size slide

  9. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    9
    Development process

    View full-size slide

  10. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    10
    About development process
    Team A Team B Team C
    Multiple teams create
    many new features at the same time.

    View full-size slide

  11. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    11
    About development process
    Team B Team C
    These new features become
    one release package.
    Team A

    View full-size slide

  12. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    12
    Test
    About development process
    Team B Team C
    Team A
    Release!
    Teams do some tests for their function.
    After passing all the tests, package will be release.

    View full-size slide

  13. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    13
    Release!
    Test
    About development process
    Hundreds of new features release per month.
    It was able to release on Tuesday and Thursday.
    Monday Tuesday Wednesday Thursday Friday
    ◯ ◯

    View full-size slide

  14. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    14
    About development problem
    Large-scale features can only released on Tuesday.
    Monday Tuesday Wednesday Thursday Friday
    X (Delay) → → →
    → Release
    Release!
    Too Slow….

    View full-size slide

  15. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    15
    About development Team
    Change the number of releasable day in a week
    Monday Tuesday Wednesday Thursday Friday
    ◯ ◯
    Monday Tuesday Wednesday Thursday Friday
    ◯ ◯ ◯ ◯

    View full-size slide

  16. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    16
    Problems in our
    automated system testing

    View full-size slide

  17. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    17
    Problems in our automated system testing
    Release!
    • E2E(End-to-End) test using Selenium.
    • Executed before release.
    • To make sure no incident is increasing by degrades.
    System testing

    View full-size slide

  18. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    18
    High cost!
    We almost spent 4 person-day on per test.
    Bottleneck!!!
    Problems in our automated system testing

    View full-size slide

  19. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    19
    Why bottleneck?
    1. Too many test cases
    2. Too flaky
    3. Maintainability was too low
    Problems in our automated system testing

    View full-size slide

  20. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    20
    1. Too many test cases
    ● No test plan
    ● What we want to assure is not clear
    Problems in our automated system testing

    View full-size slide

  21. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    21
    2. Too flaky
    ● Test code were implemented
    in complex processing
    ● Some cases include unsuitable step
    in automated testing
    Problems in our automated system testing

    View full-size slide

  22. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    22
    3. Maintainability was too low
    ● Complex test code
    ● Low readability
    Problems in our automated system testing

    View full-size slide

  23. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    23
    Solution 1
    Recreate test plan

    View full-size slide

  24. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    24
    Solutions
    1. Too many test cases -> Test plan
    2. Too flaky
    3. Maintainability was too low
    Solution 1 Recreate test plan

    View full-size slide

  25. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    25
    Solution 1 Recreate test plan
    Define the scopes about
    manual testing and automated testing
    • Manual:Specification based
    – New features and regression test
    • Automated:Risk based
    – Comprehensive regression test

    View full-size slide

  26. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    26
    Solution 1 Recreate test plan
    All functions in LIFULL HOME’S
    Manual
    Testing
    Automated
    Testing
    New
    Feature

    View full-size slide

  27. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    27
    Solution 1 Recreate test plan
    All functions in LIFULL HOME’S
    Manual
    Testing
    Automated
    Testing
    New
    Feature

    View full-size slide

  28. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    28
    Solution 1 Recreate test plan
    All functions in LIFULL HOME’S
    Manual
    Testing
    Automated
    Testing
    New
    Feature

    View full-size slide

  29. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    29
    Solution 1 Recreate test plan
    Prioritize test scopes
    • Page views and page’s values on Google Analytics
    – Business critical
    • Incidents of the production

    View full-size slide

  30. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    30
    Solution 1 Recreate test plan
    ● Decreased to suitable number
    ○ 1000 cases -> 600 cases
    ● Defined the rule of adding test cases
    ○ Unsuitable test case prevented.

    View full-size slide

  31. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    31
    Solution 2
    New testing framework “Bucky”

    View full-size slide

  32. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    32
    Solutions.
    1. Too many test cases -> Test plan
    2. Too flaky -> Bucky
    3. Maintainability was too low -> Bucky
    Solution 2 New testing framework “Bucky”

    View full-size slide

  33. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    33
    Solution 2 New testing framework “Bucky”
    $ bucky run -case case_1
    Data
    base
    Save
    test result
    Execute test
    TestC
    ode
    TestC
    ode
    Test
    Case
    Page
    object
    Element
    Locater
    Element
    Location
    Page
    object
    Make
    test report
    Bucky-core Bucky-management

    View full-size slide

  34. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    34
    Solutions.
    1. Too many test cases -> Test plan
    2. Too flaky -> Retest
    3. Maintainability was too low
    Solution 2 New testing framework “Bucky”

    View full-size slide

  35. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    35
    Improve retest function
    Solution 2 New testing framework “Bucky”
    $ bucky run -r 3 $ bucky rerun --job_id 1 -r 3

    View full-size slide

  36. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    36
    Solutions.
    1. Too many test cases -> Test plan
    2. Too flaky -> Retest
    3. Maintainability was too low-> Test code in YAML
    Solution 2 New testing framework “Bucky”

    View full-size slide

  37. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    37
    Test code in YAML
    Solution 2 New testing framework “Bucky”

    View full-size slide

  38. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    38
    What’s YAML?
    https://yaml.org/spec/1.2/spec.html
    Solution 2 New testing framework “Bucky”
    # YAML example
    american:
    - Boston Red Sox
    - Detroit Tigers
    - New York Yankees
    national:
    - New York Mets
    - Chicago Cubs
    - Atlanta Braves

    View full-size slide

  39. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    39
    Solution 2 New testing framework “Bucky”
    - case_name: favorite_1
    func: Save to favorite.
    desc: Save Real Estate to favorite from detail page
    procs:
    - proc: Open Real Estate’s detail page.
    exec:
    operate: go
    url: www.homes.co.jp/chintai/b-0000000000000/
    - proc: Save to favorite.
    exec:
    operate: click
    page: bukken_detail
    part: favorite
    - proc: Check whether target Real Estate is saved as favorite.
    exec:
    verify: assert_exist_part
    page: bukken_detail
    part: complete_favorite

    View full-size slide

  40. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    40
    Test code in YAML
    ● Very simple
    ○ No programming skills dependence
    ○ No if and loop
    ○ Reduce complexity
    -> Improved maintainability
    -> Stabilization test execution
    Solution 2 New testing framework “Bucky”

    View full-size slide

  41. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    41
    Solution 2 New testing framework “Bucky”
    # Open Real Estate’s detail page.
    self.driver.get('www.homes.co.jp/chintai/b-0000000000000/’)
    # Save to favorite.
    page = bukkenluist.Page(self.driver)
    page.favorite.click
    # Check whether target Real Estate is saved as favorite.
    self.verify.is_true(self.is_exist_part(page, 'complete_favorite’))
    def is_exist_part(page, part_name):
    try:
    getattr(page, part_name)
    return True
    except NoSuchElementException:
    return False
    except Exception as e:
    raise Exception(e.message)
    - case_name: favorite_1
    func: Save to favorite.
    desc: Save Real Estate to favorite from detail page
    procs:
    - proc: Open Real Estate’s detail page.
    exec:
    operate: go
    url: www.homes.co.jp/chintai/b-0000000000000/
    - proc: Save to favorite.
    exec:
    operate: click
    page: bukken_detail
    part: favorite
    - proc: Check whether target Real Estate is saved as
    favorite.
    exec:
    verify: assert_exist_part
    page: bukken_detail
    part: complete_favorite
    Bucky-core
    Old(Python)

    View full-size slide

  42. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    42
    Conclusion

    View full-size slide

  43. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    43
    Solutions.
    1. Too many test cases -> Test plan
    2. Too flaky -> Retest
    3. Maintainability was too low-> Test code in YAML
    Conclusion

    View full-size slide

  44. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    44
    Testing cost per day decreased.
    Conclusion
    0
    5
    10
    15
    20
    25
    30
    35
    2 days 4 days
    32h (4 people * 8h)
    6h (3 people * 2h)

    View full-size slide

  45. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    45
    The number of releasable days increase!!
    Conclusion
    Monday Tuesday Wednesday Thursday Friday
    ◯ ◯
    Monday Tuesday Wednesday Thursday Friday
    ◯ ◯ ◯ ◯

    View full-size slide

  46. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    46
    Bucky have many useful functions!
    We published as OSS.
    Please take a look and have a try of Bucky!
    https://github.com/lifull-dev/bucky-core
    https://github.com/lifull-dev/bucky-management

    View full-size slide

  47. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。
    47
    Thank you for listening.
    Have a good Selenium life.

    View full-size slide