Technology supporting the release of HOME'S

Technology supporting the release of HOME'S

We accelerated release frequency
from two times a week to four times a week.

We achieved this goal by solving some problems in the automated system testing which is executed before release.

E5c3d36c97f9ea64cceddbd7a04a4bb8?s=128

Naoto KISHINO

April 19, 2019
Tweet

Transcript

  1. 1.

    Technology supporting the release of HOME'S Naoto Kishino (Software Engineer

    in Test at LIFULL) April 19th, 2019 at Selenium Conf Tokyo
  2. 2.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 2 Introduction Naoto Kishino (木住野 奈夫人)

    Software Engineer in Test Group (3 members) Test Automation/CI/CD Group vision : 自動化と高速化を進め、プロとして最高のDXを実現する Create the best DX (Developer Experience) in the world. @naotospace NA070
  3. 3.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 3 Overview We accelerated release frequency

    from two times a week to four times a week. We achieved this goal by solving some problems in the automated system testing which is executed before release.
  4. 4.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 4 Agenda 1.About LIFULL HOME’S 2.About

    development process 3.Problems in our automated system testing 4.Solutions 5.Conclusion
  5. 8.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 8 About LIFULL HOME’S • Information

    service about Real Estate in Japan. • Registration number’s No.1 in Japan
  6. 10.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 10 About development process Team A

    Team B Team C Multiple teams create many new features at the same time.
  7. 12.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 12 Test About development process Team

    B Team C Team A Release! Teams do some tests for their function. After passing all the tests, package will be release.
  8. 13.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 13 Release! Test About development process

    Hundreds of new features release per month. It was able to release on Tuesday and Thursday. Monday Tuesday Wednesday Thursday Friday ◯ ◯
  9. 14.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 14 About development problem Large-scale features

    can only released on Tuesday. Monday Tuesday Wednesday Thursday Friday X (Delay) → → → → Release Release! Too Slow….
  10. 15.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 15 About development Team Change the

    number of releasable day in a week Monday Tuesday Wednesday Thursday Friday ◯ ◯ Monday Tuesday Wednesday Thursday Friday ◯ ◯ ◯ ◯
  11. 17.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 17 Problems in our automated system

    testing Release! • E2E(End-to-End) test using Selenium. • Executed before release. • To make sure no incident is increasing by degrades. System testing
  12. 18.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 18 High cost! We almost spent

    4 person-day on per test. Bottleneck!!! Problems in our automated system testing
  13. 19.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 19 Why bottleneck? 1. Too many

    test cases 2. Too flaky 3. Maintainability was too low Problems in our automated system testing
  14. 20.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 20 1. Too many test cases

    • No test plan • What we want to assure is not clear Problems in our automated system testing
  15. 21.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 21 2. Too flaky • Test

    code were implemented in complex processing • Some cases include unsuitable step in automated testing Problems in our automated system testing
  16. 22.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 22 3. Maintainability was too low

    • Complex test code • Low readability Problems in our automated system testing
  17. 24.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 24 Solutions 1. Too many test

    cases -> Test plan 2. Too flaky 3. Maintainability was too low Solution 1 Recreate test plan
  18. 25.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 25 Solution 1 Recreate test plan

    Define the scopes about manual testing and automated testing • Manual:Specification based – New features and regression test • Automated:Risk based – Comprehensive regression test
  19. 26.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 26 Solution 1 Recreate test plan

    All functions in LIFULL HOME’S Manual Testing Automated Testing New Feature
  20. 27.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 27 Solution 1 Recreate test plan

    All functions in LIFULL HOME’S Manual Testing Automated Testing New Feature
  21. 28.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 28 Solution 1 Recreate test plan

    All functions in LIFULL HOME’S Manual Testing Automated Testing New Feature
  22. 29.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 29 Solution 1 Recreate test plan

    Prioritize test scopes • Page views and page’s values on Google Analytics – Business critical • Incidents of the production
  23. 30.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 30 Solution 1 Recreate test plan

    • Decreased to suitable number ◦ 1000 cases -> 600 cases • Defined the rule of adding test cases ◦ Unsuitable test case prevented.
  24. 32.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 32 Solutions. 1. Too many test

    cases -> Test plan 2. Too flaky -> Bucky 3. Maintainability was too low -> Bucky Solution 2 New testing framework “Bucky”
  25. 33.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 33 Solution 2 New testing framework

    “Bucky” $ bucky run -case case_1 Data base Save test result Execute test TestC ode TestC ode Test Case Page object Element Locater Element Location Page object Make test report Bucky-core Bucky-management
  26. 34.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 34 Solutions. 1. Too many test

    cases -> Test plan 2. Too flaky -> Retest 3. Maintainability was too low Solution 2 New testing framework “Bucky”
  27. 35.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 35 Improve retest function Solution 2

    New testing framework “Bucky” $ bucky run -r 3 $ bucky rerun --job_id 1 -r 3
  28. 36.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 36 Solutions. 1. Too many test

    cases -> Test plan 2. Too flaky -> Retest 3. Maintainability was too low-> Test code in YAML Solution 2 New testing framework “Bucky”
  29. 38.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 38 What’s YAML? https://yaml.org/spec/1.2/spec.html Solution 2

    New testing framework “Bucky” # YAML example american: - Boston Red Sox - Detroit Tigers - New York Yankees national: - New York Mets - Chicago Cubs - Atlanta Braves
  30. 39.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 39 Solution 2 New testing framework

    “Bucky” - case_name: favorite_1 func: Save to favorite. desc: Save Real Estate to favorite from detail page procs: - proc: Open Real Estate’s detail page. exec: operate: go url: www.homes.co.jp/chintai/b-0000000000000/ - proc: Save to favorite. exec: operate: click page: bukken_detail part: favorite - proc: Check whether target Real Estate is saved as favorite. exec: verify: assert_exist_part page: bukken_detail part: complete_favorite
  31. 40.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 40 Test code in YAML •

    Very simple ◦ No programming skills dependence ◦ No if and loop ◦ Reduce complexity -> Improved maintainability -> Stabilization test execution Solution 2 New testing framework “Bucky”
  32. 41.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 41 Solution 2 New testing framework

    “Bucky” # Open Real Estate’s detail page. self.driver.get('www.homes.co.jp/chintai/b-0000000000000/’) # Save to favorite. page = bukkenluist.Page(self.driver) page.favorite.click # Check whether target Real Estate is saved as favorite. self.verify.is_true(self.is_exist_part(page, 'complete_favorite’)) def is_exist_part(page, part_name): try: getattr(page, part_name) return True except NoSuchElementException: return False except Exception as e: raise Exception(e.message) - case_name: favorite_1 func: Save to favorite. desc: Save Real Estate to favorite from detail page procs: - proc: Open Real Estate’s detail page. exec: operate: go url: www.homes.co.jp/chintai/b-0000000000000/ - proc: Save to favorite. exec: operate: click page: bukken_detail part: favorite - proc: Check whether target Real Estate is saved as favorite. exec: verify: assert_exist_part page: bukken_detail part: complete_favorite Bucky-core Old(Python)
  33. 43.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 43 Solutions. 1. Too many test

    cases -> Test plan 2. Too flaky -> Retest 3. Maintainability was too low-> Test code in YAML Conclusion
  34. 44.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 44 Testing cost per day decreased.

    Conclusion 0 5 10 15 20 25 30 35 2 days 4 days 32h (4 people * 8h) 6h (3 people * 2h)
  35. 45.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 45 The number of releasable days

    increase!! Conclusion Monday Tuesday Wednesday Thursday Friday ◯ ◯ Monday Tuesday Wednesday Thursday Friday ◯ ◯ ◯ ◯
  36. 46.

    © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 46 Bucky have many useful functions!

    We published as OSS. Please take a look and have a try of Bucky! https://github.com/lifull-dev/bucky-core https://github.com/lifull-dev/bucky-management