Technology supporting the release of HOME'S

We accelerated release frequency
from two times a week to four times a week.

We achieved this goal by solving some problems in the automated system testing which is executed before release.

Transcript

1. Technology supporting the release of HOME'S Naoto Kishino (Software Engineer in Test at LIFULL) April 19th, 2019 at Selenium Conf Tokyo
2. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 2 Introduction Naoto Kishino (木住野 奈夫人) Software Engineer in Test Group (3 members) Test Automation/CI/CD Group vision : 自動化と高速化を進め、プロとして最高のDXを実現する Create the best DX (Developer Experience) in the world. @naotospace NA070
3. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 3 Overview We accelerated release frequency from two times a week to four times a week. We achieved this goal by solving some problems in the automated system testing which is executed before release.
4. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 4 Agenda 1.About LIFULL HOME’S 2.About development process 3.Problems in our automated system testing 4.Solutions 5.Conclusion
5. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 5 About LIFULL HOME’S Do you know LIFULL HOME’S?
6. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 6 About LIFULL HOME’S
7. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 7 About LIFULL HOME’S
8. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 8 About LIFULL HOME’S • Information service about Real Estate in Japan. • Registration number’s No.1 in Japan
9. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 9 Development process
10. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 10 About development process Team A Team B Team C Multiple teams create many new features at the same time.
11. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 11 About development process Team B Team C These new features become one release package. Team A
12. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 12 Test About development process Team B Team C Team A Release! Teams do some tests for their function. After passing all the tests, package will be release.
13. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 13 Release! Test About development process Hundreds of new features release per month. It was able to release on Tuesday and Thursday. Monday Tuesday Wednesday Thursday Friday ◯ ◯
14. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 14 About development problem Large-scale features can only released on Tuesday. Monday Tuesday Wednesday Thursday Friday X (Delay) → → → → Release Release! Too Slow….
15. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 15 About development Team Change the number of releasable day in a week Monday Tuesday Wednesday Thursday Friday ◯ ◯ Monday Tuesday Wednesday Thursday Friday ◯ ◯ ◯ ◯
16. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 16 Problems in our automated system testing
17. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 17 Problems in our automated system testing Release! • E2E(End-to-End) test using Selenium. • Executed before release. • To make sure no incident is increasing by degrades. System testing
18. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 18 High cost! We almost spent 4 person-day on per test. Bottleneck!!! Problems in our automated system testing
19. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 19 Why bottleneck? 1. Too many test cases 2. Too flaky 3. Maintainability was too low Problems in our automated system testing
20. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 20 1. Too many test cases • No test plan • What we want to assure is not clear Problems in our automated system testing
21. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 21 2. Too flaky • Test code were implemented in complex processing • Some cases include unsuitable step in automated testing Problems in our automated system testing
22. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 22 3. Maintainability was too low • Complex test code • Low readability Problems in our automated system testing
23. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 23 Solution 1 Recreate test plan
24. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 24 Solutions 1. Too many test cases -> Test plan 2. Too flaky 3. Maintainability was too low Solution 1 Recreate test plan
25. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 25 Solution 1 Recreate test plan Define the scopes about manual testing and automated testing • Manual：Specification based – New features and regression test • Automated：Risk based – Comprehensive regression test
26. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 26 Solution 1 Recreate test plan All functions in LIFULL HOME’S Manual Testing Automated Testing New Feature
27. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 27 Solution 1 Recreate test plan All functions in LIFULL HOME’S Manual Testing Automated Testing New Feature
28. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 28 Solution 1 Recreate test plan All functions in LIFULL HOME’S Manual Testing Automated Testing New Feature
29. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 29 Solution 1 Recreate test plan Prioritize test scopes • Page views and page’s values on Google Analytics – Business critical • Incidents of the production
30. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 30 Solution 1 Recreate test plan • Decreased to suitable number ◦ 1000 cases -> 600 cases • Defined the rule of adding test cases ◦ Unsuitable test case prevented.
31. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 31 Solution 2 New testing framework “Bucky”
32. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 32 Solutions. 1. Too many test cases -> Test plan 2. Too flaky -> Bucky 3. Maintainability was too low -> Bucky Solution 2 New testing framework “Bucky”
33. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 33 Solution 2 New testing framework “Bucky” $ bucky run -case case_1 Data base Save test result Execute test TestC ode TestC ode Test Case Page object Element Locater Element Location Page object Make test report Bucky-core Bucky-management
34. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 34 Solutions. 1. Too many test cases -> Test plan 2. Too flaky -> Retest 3. Maintainability was too low Solution 2 New testing framework “Bucky”
35. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 35 Improve retest function Solution 2 New testing framework “Bucky” $ bucky run -r 3 $ bucky rerun --job_id 1 -r 3
36. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 36 Solutions. 1. Too many test cases -> Test plan 2. Too flaky -> Retest 3. Maintainability was too low-> Test code in YAML Solution 2 New testing framework “Bucky”
37. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 37 Test code in YAML Solution 2 New testing framework “Bucky”
38. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 38 What’s YAML? https://yaml.org/spec/1.2/spec.html Solution 2 New testing framework “Bucky” # YAML example american: - Boston Red Sox - Detroit Tigers - New York Yankees national: - New York Mets - Chicago Cubs - Atlanta Braves
39. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 39 Solution 2 New testing framework “Bucky” - case_name: favorite_1 func: Save to favorite. desc: Save Real Estate to favorite from detail page procs: - proc: Open Real Estate’s detail page. exec: operate: go url: www.homes.co.jp/chintai/b-0000000000000/ - proc: Save to favorite. exec: operate: click page: bukken_detail part: favorite - proc: Check whether target Real Estate is saved as favorite. exec: verify: assert_exist_part page: bukken_detail part: complete_favorite
40. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 40 Test code in YAML • Very simple ◦ No programming skills dependence ◦ No if and loop ◦ Reduce complexity -> Improved maintainability -> Stabilization test execution Solution 2 New testing framework “Bucky”
41. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 41 Solution 2 New testing framework “Bucky” # Open Real Estate’s detail page. self.driver.get('www.homes.co.jp/chintai/b-0000000000000/’) # Save to favorite. page = bukkenluist.Page(self.driver) page.favorite.click # Check whether target Real Estate is saved as favorite. self.verify.is_true(self.is_exist_part(page, 'complete_favorite’)) def is_exist_part(page, part_name): try: getattr(page, part_name) return True except NoSuchElementException: return False except Exception as e: raise Exception(e.message) - case_name: favorite_1 func: Save to favorite. desc: Save Real Estate to favorite from detail page procs: - proc: Open Real Estate’s detail page. exec: operate: go url: www.homes.co.jp/chintai/b-0000000000000/ - proc: Save to favorite. exec: operate: click page: bukken_detail part: favorite - proc: Check whether target Real Estate is saved as favorite. exec: verify: assert_exist_part page: bukken_detail part: complete_favorite Bucky-core Old（Python)
42. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 42 Conclusion
43. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 43 Solutions. 1. Too many test cases -> Test plan 2. Too flaky -> Retest 3. Maintainability was too low-> Test code in YAML Conclusion
44. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 44 Testing cost per day decreased. Conclusion 0 5 10 15 20 25 30 35 2 days 4 days 32h (4 people * 8h) 6h (3 people * 2h)
45. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 45 The number of releasable days increase!! Conclusion Monday Tuesday Wednesday Thursday Friday ◯ ◯ Monday Tuesday Wednesday Thursday Friday ◯ ◯ ◯ ◯
46. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 46 Bucky have many useful functions! We published as OSS. Please take a look and have a try of Bucky! https://github.com/lifull-dev/bucky-core https://github.com/lifull-dev/bucky-management
47. © LIFULL Co.,Ltd. 本書の無断転載、複製を固く禁じます。 47 Thank you for listening. Have a good Selenium life.