Save 37% off PRO during our Black Friday Sale! »

Pull Request-Driven E2E Testing with Kubernetes

Pull Request-Driven E2E Testing with Kubernetes

Eebedc2ee7ff95ffb9d9102c6d4a065c?s=128

LINE DevDay 2020

November 26, 2020
Tweet

Transcript

  1. None
  2. Agenda › Session Overview › Introduction › Automation Test Team

    › Basic E2E Testing › Tasks we do › Features › Pull Reqest-Driven E2E testing › Why we tackle it › Architecture › Improvements › Summary
  3. Agenda › Session Overview › Introduction › Automation Test Team

    › Basic E2E Testing › Tasks we do › Features › Pull Reqest-Driven E2E testing › Why we tackle it › Architecture › Improvements › Summary
  4. Session Overview E2E test

  5. Basic E2E Test PR - 1 Pull Request-Driven Test Session

    Overview Today I would like to talk about; Production E2E Test Server Server Server Server Data Server Server Server Server Data Server Server Server Server Data E2E Test Staging PR - 2 Server Server Server Server Data E2E Test PR - 3 Server Server Server Server Data E2E Test
  6. Session Overview Today I would like to talk about; Engineering

    E2E test Role SET* *Software Engineer in Test
  7. Agenda › Session Overview › Introduction › Automation Test Team

    › Basic E2E Testing › Tasks we do › Features › Pull Reqest-Driven E2E testing › Why we tackle it › Architecture › Improvements › Summary
  8. Introduction Automation test team Technical stacks - Selenium for web

    app - Appium for mobile app - We make own execution and reporting system Automation Test Team Overview - 6 members - Mainly E2E testing. Not only web app, but also mobile app. Image: https://linefukuoka.co.jp/ja/career/engineer/interview_set
  9. Agenda › Session Overview › Introduction › Automation Test Team

    › Basic E2E Test › Tasks we do › Features › Pull Reqest-Driven E2E test › Why we tackle it › Architecture › Improvements › Summary
  10. Basic E2E Test Tasks we do - In the case

    of LINE Creators Market project
  11. Basic E2E Test Tasks we do - In the case

    of LINE Creators Market project For Two Yeas 2 Test cases 100+ Environments 2
  12. Basic E2E Test Features › The E2E Test can detect

    a very little change which cannot be found by human › It helps to improve the quality of the application by incorporating it into the iterative Regression test
  13. Basic E2E Test Features › Stateful datasets › Low parallelism

    and long execution time
  14. Basic E2E Test Features › Stateful datasets › Low parallelism

    and long execution time Parallelism Staging Server Server Server Server Data Tester Test B Test A Developer
  15. Basic E2E Test Features › Stateful datasets › Low parallelism

    and long execution time Parallelism Execution time Staging Server Server Server Server Data Tester Test B Test A Developer Test 1 Test 2 Test 3 Test 4
  16. Basic E2E Test Features › Late feedback to the developers

    › Sometimes, it is too late that finding a bug in the staging environment › When has your PR been merged??? › Are problems always simple? Might be NO in many cases › Stateful datasets › Low parallelism and long execution time
  17. Basic E2E Test Features › Late feedback to the developers

    › Sometimes, it is too late that finding a bug in the staging environment › When has your PR been merged??? › Are problems always simple? Might be NO in many cases › Deploy services on Kubernetes each pull requests › Based on pull request codes › For E2E test › Stateful datasets › Low parallelism and long execution time
  18. About SET Role Engineering E2E test Role SET

  19. SET? DX? Definitions › SET › “Originally software engineers with

    deep infrastructure and tooling expertise, SETs built the frameworks and packages required to implement automation.” › Ref. https://testing.googleblog.com/2016/03/from-qa-to-engineering-productivity.html › Not Only Scripting E2E test cases
  20. SET? DX? Definitions › SET › “Originally software engineers with

    deep infrastructure and tooling expertise, SETs built the frameworks and packages required to implement automation.” › Ref. https://testing.googleblog.com/2016/03/from-qa-to-engineering-productivity.html › Not Only Scripting E2E test cases › DX › Developer eXperience › Helps developers as a developer
  21. SET? DX? The target to contribute Test Automation Engineer *Developer

    eXperience Application Quality DX
  22. SET? DX? The target to contribute *Developer eXperience Application Quality

    SET DX
  23. SET? DX? The target to contribute *Developer eXperience Application Quality

    SET DX Test Automation Engineer
  24. Agenda › Session Overview › Introduction › Automation Test Team

    › Basic E2E Test › Tasks we do › Features › Pull Reqest-Driven E2E test › Architecture › Improvements › Summary
  25. Pull Request-Driven E2E Test An actual flows Engineering E2E test

    Role SET
  26. Pull Request-Driven E2E Test Same as Pull Request Lifecycle Deploy

    Test Delete Build Review Test Merge Create Pull Request
  27. Pull Request-Driven E2E Test Same as Pull Request Lifecycle Review

    Test Merge Create Pull Request - Modify codes - Make a branch - Push codes to the Github - Review from colleagues - Do the Unit tests / Integration tests with CI services - Merge code modification to the target branch
  28. Pull Request-Driven E2E Test Same as Pull Request Lifecycle -

    Build docker images - Publish docker images to the docker registry - Register the domain to the DNS server - Deploy services to the Kubernetes cluster - Run E2E tests on services deployed in Kubernetes - Delete deployed services when pull request is merged Deploy Test Delete Build
  29. Pull Request-Driven E2E Test Overview Drone / CircleCI Github Docker

    Registory Virtual Browser E2E Test Github App Kubernetes Cluster PR - 1 Server Server Server Server Data PR - 2 Server Server Server Server Data PR - 3 Server Server Server Server Data
  30. Pull Request-Driven E2E Test Overview Drone / CircleCI Github Docker

    Registory Github App Build
  31. Pull Request-Driven E2E Test Overview Drone / CircleCI Kubernetes Cluster

    PR - 1 Server Server Server Server Data PR - 2 Server Server Server Server Data PR - 3 Server Server Server Server Data Deploy
  32. Pull Request-Driven E2E Test Overview Virtual Browser E2E Test Github

    App Kubernetes Cluster PR - 1 Server Server Server Server Data PR - 2 Server Server Server Server Data PR - 3 Server Server Server Server Data Test
  33. Pull Request-Driven E2E Test Overview Drone / CircleCI Kubernetes Cluster

    PR - 1 Server Server Server Server Data PR - 2 Server Server Server Server Data PR - 3 Server Server Server Server Data Delete
  34. Routing Manifests Pull Request-Driven E2E Test

  35. How to Control Kubernetes Manifests › Controlling / Managing Kubernetes

    manifests is one of the important things › Inject variables dynamically › e.g. pull request number, commit hash
  36. How to Control Kubernetes Manifests › Controlling / Managing Kubernetes

    manifests is one of the important things › Inject variables dynamically › e.g. pull request number, commit hash › Kustomize? Helm? ksonnet? › In our usage, anything is fine › We chose Kustomize › Since 1.14, Kubectl supports the management of Kubernetes objects using a kustomization file
  37. Inject Kustomize: https://kustomize.io/ Kustomize file Kubernetes manifests How to Control

    Kubernetes Manifests
  38. Inject Kustomize: https://kustomize.io/ Kustomize file Kubernetes manifests How to Control

    Kubernetes Manifests
  39. How to Control the Routing Test Instance Virtual Browser Load

    Balancer Virtual Browser Test Instance User *.e2e-test.example.com NGINX Ingress Controller: https://github.com/kubernetes/ingress-nginx
  40. Kubernetes Cluster Test Instance NGINX Ingress Controller Virtual Browser Load

    Balancer Virtual Browser Test Instance User Namespace pr-1 *.e2e-test.example.com Service Service Service Pod Service Service Pod pr-1.e2e- test.example.com Namespace pr-2 Service Service Service Pod Service Service Pod pr-2.e2e- test.example.com Namespace pr-3 Service Service Service Pod Service Service Pod pr-3.e2e- test.example.com NGINX Ingress Controller: https://github.com/kubernetes/ingress-nginx How to Control the Routing
  41. Github App Label Command

  42. Pull Request-Driven E2E Test › Stateful datasets › Issues in

    parallel execution and execution time › Ensure the independence of each pull request
  43. Pull Request-Driven E2E Test › Late feedback to the developers

    › it is too late that finding a bug in the staging environment › Running E2E tests on Pull Request has made the feedback earlier › Reduce the number of bugs that are getting into the Staging environment › Also, it can be used for reviewing pull requests as a PREVIEW › Stateful datasets › Issues in parallel execution and execution time › Ensure the independence of each pull request
  44. Use as Preview Usually On local Server Server Server Server

    Data Developer Github Access Pull Build & Launch
  45. Use as Preview Usually On local Server Server Server Server

    Data Developer Github Access Pull Build & Launch PR-Driven Test On K8s Server Server Server Server Data Developer Github Access Just Access
  46. Ref. https://linecorp.com/pdf/en/LINE_STYLE_BOOK_EN.pdf

  47. Ref. https://linecorp.com/pdf/en/LINE_STYLE_BOOK_EN.pdf

  48. Keep Going to Improve Perfectly stateless Completely parallelize Build -

    Test Less than 5min
  49. Before PR - 1 Server Server Server Server Data Test

    A Test C Test B After PR - 1 - 1 Server Server Server Server Data PR - 1 - 2 Server Server Server Server Data PR - 1 - 3 Server Server Server Server Data Test A Test B Test C Keep Going to Improve
  50. Namespace pr-1 Kubernetes Cluster Github App Service Service Service Pod

    Service Service Pod Test Job 1 Service Service Service Pod Service Service Pod Service Service Service Pod Service Service Pod Test Job 2 Test Job 3 Deploy&Test Deploy&Test Deploy&Test Deploy Trace ID Trace ID Trace ID Keep Going to Improve
  51. Kubernetes Cluster Namespace pr-1 Load Balancer User *.e2e-test.example.com NGINX Ingress

    Controller: https://github.com/kubernetes/ingress-nginx Preview Service Service Service Pod Service Service Pod NGINX Ingress Controller pr-1.e2e- test.example.com Keep Going to Improve Service Pod Pod Test Job 1 Service Pod Pod Service Pod Pod Test Job 2 Test Job 3 Deploy&Test Deploy&Test Deploy&Test
  52. Agenda › Session Overview › Introduction › Automation Test Team

    › Basic E2E Testing › Tasks we do › Features › Pull Reqest-Driven E2E testing › Why we tackle it › Architecture › Improvements › Summary
  53. Summary Engineering E2E test Role SET › The role change

    from test automation engineer to SET › Not only scripting test cases › Being a part of service development to improve DX
  54. Summary Engineering E2E test Role SET › Improving the E2E

    test environment with Kubernetes › Deployed services for each pull requests and created an E2E test environments with fast feedback
  55. Thank you