Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Pull Request-Driven E2E Testing with Kubernetes

Pull Request-Driven E2E Testing with Kubernetes

LINE DevDay 2020

November 26, 2020
Tweet

More Decks by LINE DevDay 2020

Other Decks in Technology

Transcript

  1. Agenda › Session Overview › Introduction › Automation Test Team

    › Basic E2E Testing › Tasks we do › Features › Pull Reqest-Driven E2E testing › Why we tackle it › Architecture › Improvements › Summary
  2. Agenda › Session Overview › Introduction › Automation Test Team

    › Basic E2E Testing › Tasks we do › Features › Pull Reqest-Driven E2E testing › Why we tackle it › Architecture › Improvements › Summary
  3. Basic E2E Test PR - 1 Pull Request-Driven Test Session

    Overview Today I would like to talk about; Production E2E Test Server Server Server Server Data Server Server Server Server Data Server Server Server Server Data E2E Test Staging PR - 2 Server Server Server Server Data E2E Test PR - 3 Server Server Server Server Data E2E Test
  4. Session Overview Today I would like to talk about; Engineering

    E2E test Role SET* *Software Engineer in Test
  5. Agenda › Session Overview › Introduction › Automation Test Team

    › Basic E2E Testing › Tasks we do › Features › Pull Reqest-Driven E2E testing › Why we tackle it › Architecture › Improvements › Summary
  6. Introduction Automation test team Technical stacks - Selenium for web

    app - Appium for mobile app - We make own execution and reporting system Automation Test Team Overview - 6 members - Mainly E2E testing. Not only web app, but also mobile app. Image: https://linefukuoka.co.jp/ja/career/engineer/interview_set
  7. Agenda › Session Overview › Introduction › Automation Test Team

    › Basic E2E Test › Tasks we do › Features › Pull Reqest-Driven E2E test › Why we tackle it › Architecture › Improvements › Summary
  8. Basic E2E Test Tasks we do - In the case

    of LINE Creators Market project
  9. Basic E2E Test Tasks we do - In the case

    of LINE Creators Market project For Two Yeas 2 Test cases 100+ Environments 2
  10. Basic E2E Test Features › The E2E Test can detect

    a very little change which cannot be found by human › It helps to improve the quality of the application by incorporating it into the iterative Regression test
  11. Basic E2E Test Features › Stateful datasets › Low parallelism

    and long execution time Parallelism Staging Server Server Server Server Data Tester Test B Test A Developer
  12. Basic E2E Test Features › Stateful datasets › Low parallelism

    and long execution time Parallelism Execution time Staging Server Server Server Server Data Tester Test B Test A Developer Test 1 Test 2 Test 3 Test 4
  13. Basic E2E Test Features › Late feedback to the developers

    › Sometimes, it is too late that finding a bug in the staging environment › When has your PR been merged??? › Are problems always simple? Might be NO in many cases › Stateful datasets › Low parallelism and long execution time
  14. Basic E2E Test Features › Late feedback to the developers

    › Sometimes, it is too late that finding a bug in the staging environment › When has your PR been merged??? › Are problems always simple? Might be NO in many cases › Deploy services on Kubernetes each pull requests › Based on pull request codes › For E2E test › Stateful datasets › Low parallelism and long execution time
  15. SET? DX? Definitions › SET › “Originally software engineers with

    deep infrastructure and tooling expertise, SETs built the frameworks and packages required to implement automation.” › Ref. https://testing.googleblog.com/2016/03/from-qa-to-engineering-productivity.html › Not Only Scripting E2E test cases
  16. SET? DX? Definitions › SET › “Originally software engineers with

    deep infrastructure and tooling expertise, SETs built the frameworks and packages required to implement automation.” › Ref. https://testing.googleblog.com/2016/03/from-qa-to-engineering-productivity.html › Not Only Scripting E2E test cases › DX › Developer eXperience › Helps developers as a developer
  17. Agenda › Session Overview › Introduction › Automation Test Team

    › Basic E2E Test › Tasks we do › Features › Pull Reqest-Driven E2E test › Architecture › Improvements › Summary
  18. Pull Request-Driven E2E Test Same as Pull Request Lifecycle Deploy

    Test Delete Build Review Test Merge Create Pull Request
  19. Pull Request-Driven E2E Test Same as Pull Request Lifecycle Review

    Test Merge Create Pull Request - Modify codes - Make a branch - Push codes to the Github - Review from colleagues - Do the Unit tests / Integration tests with CI services - Merge code modification to the target branch
  20. Pull Request-Driven E2E Test Same as Pull Request Lifecycle -

    Build docker images - Publish docker images to the docker registry - Register the domain to the DNS server - Deploy services to the Kubernetes cluster - Run E2E tests on services deployed in Kubernetes - Delete deployed services when pull request is merged Deploy Test Delete Build
  21. Pull Request-Driven E2E Test Overview Drone / CircleCI Github Docker

    Registory Virtual Browser E2E Test Github App Kubernetes Cluster PR - 1 Server Server Server Server Data PR - 2 Server Server Server Server Data PR - 3 Server Server Server Server Data
  22. Pull Request-Driven E2E Test Overview Drone / CircleCI Kubernetes Cluster

    PR - 1 Server Server Server Server Data PR - 2 Server Server Server Server Data PR - 3 Server Server Server Server Data Deploy
  23. Pull Request-Driven E2E Test Overview Virtual Browser E2E Test Github

    App Kubernetes Cluster PR - 1 Server Server Server Server Data PR - 2 Server Server Server Server Data PR - 3 Server Server Server Server Data Test
  24. Pull Request-Driven E2E Test Overview Drone / CircleCI Kubernetes Cluster

    PR - 1 Server Server Server Server Data PR - 2 Server Server Server Server Data PR - 3 Server Server Server Server Data Delete
  25. How to Control Kubernetes Manifests › Controlling / Managing Kubernetes

    manifests is one of the important things › Inject variables dynamically › e.g. pull request number, commit hash
  26. How to Control Kubernetes Manifests › Controlling / Managing Kubernetes

    manifests is one of the important things › Inject variables dynamically › e.g. pull request number, commit hash › Kustomize? Helm? ksonnet? › In our usage, anything is fine › We chose Kustomize › Since 1.14, Kubectl supports the management of Kubernetes objects using a kustomization file
  27. How to Control the Routing Test Instance Virtual Browser Load

    Balancer Virtual Browser Test Instance User *.e2e-test.example.com NGINX Ingress Controller: https://github.com/kubernetes/ingress-nginx
  28. Kubernetes Cluster Test Instance NGINX Ingress Controller Virtual Browser Load

    Balancer Virtual Browser Test Instance User Namespace pr-1 *.e2e-test.example.com Service Service Service Pod Service Service Pod pr-1.e2e- test.example.com Namespace pr-2 Service Service Service Pod Service Service Pod pr-2.e2e- test.example.com Namespace pr-3 Service Service Service Pod Service Service Pod pr-3.e2e- test.example.com NGINX Ingress Controller: https://github.com/kubernetes/ingress-nginx How to Control the Routing
  29. Pull Request-Driven E2E Test › Stateful datasets › Issues in

    parallel execution and execution time › Ensure the independence of each pull request
  30. Pull Request-Driven E2E Test › Late feedback to the developers

    › it is too late that finding a bug in the staging environment › Running E2E tests on Pull Request has made the feedback earlier › Reduce the number of bugs that are getting into the Staging environment › Also, it can be used for reviewing pull requests as a PREVIEW › Stateful datasets › Issues in parallel execution and execution time › Ensure the independence of each pull request
  31. Use as Preview Usually On local Server Server Server Server

    Data Developer Github Access Pull Build & Launch
  32. Use as Preview Usually On local Server Server Server Server

    Data Developer Github Access Pull Build & Launch PR-Driven Test On K8s Server Server Server Server Data Developer Github Access Just Access
  33. Before PR - 1 Server Server Server Server Data Test

    A Test C Test B After PR - 1 - 1 Server Server Server Server Data PR - 1 - 2 Server Server Server Server Data PR - 1 - 3 Server Server Server Server Data Test A Test B Test C Keep Going to Improve
  34. Namespace pr-1 Kubernetes Cluster Github App Service Service Service Pod

    Service Service Pod Test Job 1 Service Service Service Pod Service Service Pod Service Service Service Pod Service Service Pod Test Job 2 Test Job 3 Deploy&Test Deploy&Test Deploy&Test Deploy Trace ID Trace ID Trace ID Keep Going to Improve
  35. Kubernetes Cluster Namespace pr-1 Load Balancer User *.e2e-test.example.com NGINX Ingress

    Controller: https://github.com/kubernetes/ingress-nginx Preview Service Service Service Pod Service Service Pod NGINX Ingress Controller pr-1.e2e- test.example.com Keep Going to Improve Service Pod Pod Test Job 1 Service Pod Pod Service Pod Pod Test Job 2 Test Job 3 Deploy&Test Deploy&Test Deploy&Test
  36. Agenda › Session Overview › Introduction › Automation Test Team

    › Basic E2E Testing › Tasks we do › Features › Pull Reqest-Driven E2E testing › Why we tackle it › Architecture › Improvements › Summary
  37. Summary Engineering E2E test Role SET › The role change

    from test automation engineer to SET › Not only scripting test cases › Being a part of service development to improve DX
  38. Summary Engineering E2E test Role SET › Improving the E2E

    test environment with Kubernetes › Deployed services for each pull requests and created an E2E test environments with fast feedback