Journey of CI/CD Pipeline Improvement in Yahoo! JAPAN Yahoo Japan Corporation 04/19/2019 @ Selenium Conf Tokyo Teppei Yamaguchi

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ • Challenges we have faced in adaptation of company-wide pipeline and automated tests in many people / multi-product company • ଟਓ਺ɾଟϓϩμΫτͷ͋Δاۀʹ͓͚ΔɺશࣾతͳύΠϓϥ Πϯͷ੔උ͓Αͼࣗಈςετಋೖɾීٴͷ՝୊΍ͦͷࠀ෰ͷྲྀ ΕΛ঺հ͠·͢ɻ Abstraction

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ Responsibility • Transforming the whole company into “Agile” and “Test automation” Career • Programmer • Software Engineering Researcher/Consultant • Coach Test Coach, Agile Coach Yahoo Japan Corporation About me Teppei Yamaguchi

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ • Programmers & Designers: 3,000+ • Services: 100+ (Portal, News, Ad, Shopping, Auction, Mail, Weather, …) • Service quality is the responsibility of the service department • No company-wide QA process • No QA department • Members change teams often Yahoo! JAPAN 4FSWJDF" %JWJTJPO 1SPEVDU$ %JWJTJPO #VTJOFTT%FTJHOFS 1SPHSBNNFSFUD 1MBUGPSN4FSWJDF

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ • Total number of CI/CD pipelines • Executions per day(all the pipelines in total) The Current CI/CD Pipelines of Yahoo! JAPAN 40,000+ 15,400+ [times]

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ The Current CI/CD Pipelines of Yahoo! JAPAN Github Enterprise SCM &YFDVUJPO1MBUGPSN Screwdriver.cd $*$%5PPMT Clone Deploy Publish Artifactory Enterprise OpenWhisk Kubernetes Pivotal Application Service 1BDLBHF3FQPTJUPSZ npm Enterprise Docker Registry Install Concourse &&5FTU5PPM Applicat Selenium

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ Journey of CI/CD Pipeline Improvement 2013 1st Improvement Project 2nd Improvement Project 3rd Improvement Project 2019 2014 2015 2016 2017 2018 Automated E2E Testing Support

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ 1st Improvement Project - Preparation for CI/CD implementation -

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ ▼Situation • Manual testing, deployment etc. was standard for many teams • Only a few advanced teams did CI/CD with Jenkins ▼Issue • We couldn’t provide services and products continuously on the latest OS and middleware • Spend lots of time to support older OS and middleware • Security risk 1st Improvement Project -Preparation for CI/CD implementation-

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ ▼Top-down: establish new strategy • Clarify responsible groups of each module • Latest OS ▼Bottom-up: support individuals / teams • Jenkins support 1st Improvement Project -Preparation for CI/CD implementation-

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ ▼Problems in project • High business goals, making it difficult to get resources for this work ▼Action for project problem solution • Explain individually to team’s managers, over and over again 1st Improvement Project -Preparation for CI/CD implementation-

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ ▼Result • Update to latest OS and clarify responsible groups was completed • CI implementation itself hardly progressed 1st Improvement Project -Preparation for CI/CD implementation-

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ 2nd Improvement Project - CI/CD implementation -

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ ▼Situation • Some teams used CI/CD with Jenkins • The importance of CI/CD was recognized but many teams didn’t use CI/CD ▼Issue • Cost to start using CI/CD was too high • Fixing security issues in CI/CD environment was high cost • Costs to move members between teams were high because all teams customized their own CI/CD in silos and requires relearning 2nd Improvement Project -CI/CD implementation-

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ 2nd Improvement Project -CI/CD implementation- ▼Top-down: establish new strategy • Use of centralized pipeline tools(Screwdriver.cd, Concourse, Approduce) • Write test code • Explain to business managers for reserving CI/CD implementation resources ▼Bottom-up: support individuals / teams • Provide Pipeline tool support • Provide testing support with test automation

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ ▼Problems in project • High preparation cost for automated E2E testing • Difficult to automate testing for legacy systems ▼Action for project problem solution • Started new project to support automated E2E testing • Clarified the strategy so that automating tests for legacy can be postponed 2nd Improvement Project -CI/CD implementation-

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ ▼Result • Automated build and automated deployment with pipeline tools became common • Couldn’t write test code for some legacy systems 2nd Improvement Project -CI/CD implementation-

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ Automated E2E Testing Support

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ ▼Situation • Builds & deployments were executed with the pipelines, but automated E2E tests were few ▼Issue • No E2E test environment in the pipeline and members didn’t know how to build it • Automated E2E tests of mobile app was not stable • Maintaining mobile devices required for man-powered E2E testing was too effort intensive Automated E2E Testing Support

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ ▼Bottom-up: support individuals / teams • Provide centralized E2E test environment & tool(based on Appium)
 for mobile app using the pipeline • Provide Selenium container • Test automation support • Pair test programming, Pair E2E test environment development • Support incremental implementation of automated E2E test Automated E2E Testing Support

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ ▼Result • Execute automated E2E test in the pipeline • Programmers got motivated for test automation Still improving… Automated E2E Testing Support

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ 3rd Improvement Project - CI/CD implement to legacy system -

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ ▼Situation • Many teams did CI/CD with pipeline tools • CI/CD couldn’t be done for legacy systems ▼Issue • Cost to automate testing for legacy systems was too high 3rd Improvement Project -CI/CD implement to legacy system-

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ ▼Top-down: establish new strategy • Refactoring from legacy systems to new testable systems • Explain to executives for assign CI/CD implementation resources ▼Bottom-up: support individuals / teams • Provide Pipeline tool support • Provide testing support 3rd Improvement Project -CI/CD implement to legacy system-

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ ▼Result • Ongoing 3rd Improvement Project -CI/CD implement to legacy system-

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ Conclusion

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ Journey of CI/CD Pipeline Improvement Preparation 2013 1st Improvement Project CI/CD implementation 2nd Improvement Project CI/CD implement to legacy system 3rd Improvement Project 2019 2014 2015 2016 2017 2018 Automated E2E Testing Support

Copyright (C) 2019 Yahoo Japan Corporation. All Rights Reserved. ແஅҾ༻ɾసࡌېࢭ • Implementation of company-wide CI/CD takes a long time • There are many problems that can not be discovered without trying, 
 because every teams are in various situations. • Need continuous improvement • Especially if you do something that does not go well Points

No content