Pull Request-Driven E2E Testing with Kubernetes

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Agenda › Session Overview › Introduction › Automation Test Team › Basic E2E Testing › Tasks we do › Features › Pull Reqest-Driven E2E testing › Why we tackle it › Architecture › Improvements › Summary

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Session Overview E2E test

Slide 5

Slide 5 text

Basic E2E Test PR - 1 Pull Request-Driven Test Session Overview Today I would like to talk about; Production E2E Test Server Server Server Server Data Server Server Server Server Data Server Server Server Server Data E2E Test Staging PR - 2 Server Server Server Server Data E2E Test PR - 3 Server Server Server Server Data E2E Test

Slide 6

Slide 6 text

Session Overview Today I would like to talk about; Engineering E2E test Role SET* *Software Engineer in Test

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Introduction Automation test team Technical stacks - Selenium for web app - Appium for mobile app - We make own execution and reporting system Automation Test Team Overview - 6 members - Mainly E2E testing. Not only web app, but also mobile app. Image: https://linefukuoka.co.jp/ja/career/engineer/interview_set

Slide 9

Slide 9 text

Agenda › Session Overview › Introduction › Automation Test Team › Basic E2E Test › Tasks we do › Features › Pull Reqest-Driven E2E test › Why we tackle it › Architecture › Improvements › Summary

Slide 10

Slide 10 text

Basic E2E Test Tasks we do - In the case of LINE Creators Market project

Slide 11

Slide 11 text

Basic E2E Test Tasks we do - In the case of LINE Creators Market project For Two Yeas 2 Test cases 100+ Environments 2

Slide 12

Slide 12 text

Basic E2E Test Features › The E2E Test can detect a very little change which cannot be found by human › It helps to improve the quality of the application by incorporating it into the iterative Regression test

Slide 13

Slide 13 text

Basic E2E Test Features › Stateful datasets › Low parallelism and long execution time

Slide 14

Slide 14 text

Basic E2E Test Features › Stateful datasets › Low parallelism and long execution time Parallelism Staging Server Server Server Server Data Tester Test B Test A Developer

Slide 15

Slide 15 text

Basic E2E Test Features › Stateful datasets › Low parallelism and long execution time Parallelism Execution time Staging Server Server Server Server Data Tester Test B Test A Developer Test 1 Test 2 Test 3 Test 4

Slide 16

Slide 16 text

Basic E2E Test Features › Late feedback to the developers › Sometimes, it is too late that finding a bug in the staging environment › When has your PR been merged??? › Are problems always simple? Might be NO in many cases › Stateful datasets › Low parallelism and long execution time

Slide 17

Slide 17 text

Basic E2E Test Features › Late feedback to the developers › Sometimes, it is too late that finding a bug in the staging environment › When has your PR been merged??? › Are problems always simple? Might be NO in many cases › Deploy services on Kubernetes each pull requests › Based on pull request codes › For E2E test › Stateful datasets › Low parallelism and long execution time

Slide 18

Slide 18 text

About SET Role Engineering E2E test Role SET

Slide 19

Slide 19 text

SET? DX? Definitions › SET › “Originally software engineers with deep infrastructure and tooling expertise, SETs built the frameworks and packages required to implement automation.” › Ref. https://testing.googleblog.com/2016/03/from-qa-to-engineering-productivity.html › Not Only Scripting E2E test cases

Slide 20

Slide 20 text

Slide 21

Slide 21 text

SET? DX? The target to contribute Test Automation Engineer *Developer eXperience Application Quality DX

Slide 22

Slide 22 text

SET? DX? The target to contribute *Developer eXperience Application Quality SET DX

Slide 23

Slide 23 text

SET? DX? The target to contribute *Developer eXperience Application Quality SET DX Test Automation Engineer

Slide 24

Slide 24 text

Agenda › Session Overview › Introduction › Automation Test Team › Basic E2E Test › Tasks we do › Features › Pull Reqest-Driven E2E test › Architecture › Improvements › Summary

Slide 25

Slide 25 text

Pull Request-Driven E2E Test An actual flows Engineering E2E test Role SET

Slide 26

Slide 26 text

Pull Request-Driven E2E Test Same as Pull Request Lifecycle Deploy Test Delete Build Review Test Merge Create Pull Request

Slide 27

Slide 27 text

Pull Request-Driven E2E Test Same as Pull Request Lifecycle Review Test Merge Create Pull Request - Modify codes - Make a branch - Push codes to the Github - Review from colleagues - Do the Unit tests / Integration tests with CI services - Merge code modification to the target branch

Slide 28

Slide 28 text

Pull Request-Driven E2E Test Same as Pull Request Lifecycle - Build docker images - Publish docker images to the docker registry - Register the domain to the DNS server - Deploy services to the Kubernetes cluster - Run E2E tests on services deployed in Kubernetes - Delete deployed services when pull request is merged Deploy Test Delete Build

Slide 29

Slide 29 text

Pull Request-Driven E2E Test Overview Drone / CircleCI Github Docker Registory Virtual Browser E2E Test Github App Kubernetes Cluster PR - 1 Server Server Server Server Data PR - 2 Server Server Server Server Data PR - 3 Server Server Server Server Data

Slide 30

Slide 30 text

Pull Request-Driven E2E Test Overview Drone / CircleCI Github Docker Registory Github App Build

Slide 31

Slide 31 text

Pull Request-Driven E2E Test Overview Drone / CircleCI Kubernetes Cluster PR - 1 Server Server Server Server Data PR - 2 Server Server Server Server Data PR - 3 Server Server Server Server Data Deploy

Slide 32

Slide 32 text

Pull Request-Driven E2E Test Overview Virtual Browser E2E Test Github App Kubernetes Cluster PR - 1 Server Server Server Server Data PR - 2 Server Server Server Server Data PR - 3 Server Server Server Server Data Test

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Routing Manifests Pull Request-Driven E2E Test

Slide 35

Slide 35 text

How to Control Kubernetes Manifests › Controlling / Managing Kubernetes manifests is one of the important things › Inject variables dynamically › e.g. pull request number, commit hash

Slide 36

Slide 36 text

How to Control Kubernetes Manifests › Controlling / Managing Kubernetes manifests is one of the important things › Inject variables dynamically › e.g. pull request number, commit hash › Kustomize? Helm? ksonnet? › In our usage, anything is fine › We chose Kustomize › Since 1.14, Kubectl supports the management of Kubernetes objects using a kustomization file

Slide 37

Slide 37 text

Inject Kustomize: https://kustomize.io/ Kustomize file Kubernetes manifests How to Control Kubernetes Manifests

Slide 38

Slide 38 text

Inject Kustomize: https://kustomize.io/ Kustomize file Kubernetes manifests How to Control Kubernetes Manifests

Slide 39

Slide 39 text

How to Control the Routing Test Instance Virtual Browser Load Balancer Virtual Browser Test Instance User *.e2e-test.example.com NGINX Ingress Controller: https://github.com/kubernetes/ingress-nginx

Slide 40

Slide 40 text

Kubernetes Cluster Test Instance NGINX Ingress Controller Virtual Browser Load Balancer Virtual Browser Test Instance User Namespace pr-1 *.e2e-test.example.com Service Service Service Pod Service Service Pod pr-1.e2e- test.example.com Namespace pr-2 Service Service Service Pod Service Service Pod pr-2.e2e- test.example.com Namespace pr-3 Service Service Service Pod Service Service Pod pr-3.e2e- test.example.com NGINX Ingress Controller: https://github.com/kubernetes/ingress-nginx How to Control the Routing

Slide 41

Slide 41 text

Github App Label Command

Slide 42

Slide 42 text

Pull Request-Driven E2E Test › Stateful datasets › Issues in parallel execution and execution time › Ensure the independence of each pull request

Slide 43

Slide 43 text

Pull Request-Driven E2E Test › Late feedback to the developers › it is too late that finding a bug in the staging environment › Running E2E tests on Pull Request has made the feedback earlier › Reduce the number of bugs that are getting into the Staging environment › Also, it can be used for reviewing pull requests as a PREVIEW › Stateful datasets › Issues in parallel execution and execution time › Ensure the independence of each pull request

Slide 44

Slide 44 text

Use as Preview Usually On local Server Server Server Server Data Developer Github Access Pull Build & Launch

Slide 45

Slide 45 text

Use as Preview Usually On local Server Server Server Server Data Developer Github Access Pull Build & Launch PR-Driven Test On K8s Server Server Server Server Data Developer Github Access Just Access

Slide 46

Slide 46 text

Ref. https://linecorp.com/pdf/en/LINE_STYLE_BOOK_EN.pdf

Slide 47

Slide 47 text

Ref. https://linecorp.com/pdf/en/LINE_STYLE_BOOK_EN.pdf

Slide 48

Slide 48 text

Keep Going to Improve Perfectly stateless Completely parallelize Build - Test Less than 5min

Slide 49

Slide 49 text

Before PR - 1 Server Server Server Server Data Test A Test C Test B After PR - 1 - 1 Server Server Server Server Data PR - 1 - 2 Server Server Server Server Data PR - 1 - 3 Server Server Server Server Data Test A Test B Test C Keep Going to Improve

Slide 50

Slide 50 text

Namespace pr-1 Kubernetes Cluster Github App Service Service Service Pod Service Service Pod Test Job 1 Service Service Service Pod Service Service Pod Service Service Service Pod Service Service Pod Test Job 2 Test Job 3 Deploy&Test Deploy&Test Deploy&Test Deploy Trace ID Trace ID Trace ID Keep Going to Improve

Slide 51

Slide 51 text

Kubernetes Cluster Namespace pr-1 Load Balancer User *.e2e-test.example.com NGINX Ingress Controller: https://github.com/kubernetes/ingress-nginx Preview Service Service Service Pod Service Service Pod NGINX Ingress Controller pr-1.e2e- test.example.com Keep Going to Improve Service Pod Pod Test Job 1 Service Pod Pod Service Pod Pod Test Job 2 Test Job 3 Deploy&Test Deploy&Test Deploy&Test

Slide 52

Slide 52 text

Slide 53

Slide 53 text

Summary Engineering E2E test Role SET › The role change from test automation engineer to SET › Not only scripting test cases › Being a part of service development to improve DX

Slide 54

Slide 54 text

Summary Engineering E2E test Role SET › Improving the E2E test environment with Kubernetes › Deployed services for each pull requests and created an E2E test environments with fast feedback

Slide 55

Slide 55 text

Thank you