Slide 1
Slide 1 text
End to End Testing for
Kubernetes Cluster
施柏賢
Pohsien Shih
CNTUG Meetup #20
Slide 2
Slide 2 text
Who am I
施柏賢 Pohsien Shih
● Service Engineer
● CNTUG社群打雜工
Blog: https://pohsienshih.github.io/
Github: https://github.com/pohsienshih
Email: [email protected]
2
Slide 3
Slide 3 text
Outline
● Software Testing
● Kubernetes Testing
● Cluster E2E Testing
● Conformance Testing
● Node E2E Testing
● Conclusion
● References
3
Slide 4
Slide 4 text
Why do I need to test the Kubernetes cluster?
4
Slide 5
Slide 5 text
1. Make sure your cluster is available.
2. Deploy a production ready Kubernetes cluster.
5
The Reasons you should test your cluster.
Slide 6
Slide 6 text
1Software Testing
● Types of Testing
6
Slide 7
Slide 7 text
Types of Testing
● Unit Testing
- Test an individual software component or module.
● Integration Testing
- Test all integrated modules to verify the combined
functionality after integration.
● End to End (E2E) Testing
- Test end-to-end behavior of the system.
Software Testing 7
Slide 8
Slide 8 text
2Kubernetes Testing
● Special Interest Group (SIG)
● SIG-Testing
● test-infra
8
Slide 9
Slide 9 text
Special Interest Group
A Special Interest Group (SIG) is a community within a larger organization with a
shared interest in advancing a specific area of knowledge, learning or technology
where members cooperate to affect or to produce solutions within their particular
field, and may communicate, meet, and organize conferences.
Ref: wikipedia - Special Interest Group
9
Kubernetes Testing
Slide 10
Slide 10 text
Special Interest Group (cont.)
Kubernetes SIGs
● Kubernetes Community Governance Model
● Kubernetes SIGs and Working Groups
Subprojects
Every part of the Kubernetes code and documentation must be owned by some
subproject. These subprojects will be delivered to different SIGs.
● Subprojects of each SIGs
10
Kubernetes Testing
Slide 11
Slide 11 text
Special Interest Group (cont.)
SIG-TESTING
The SIG which is responsible for Kubernetes testing.
Ref: https://github.com/kubernetes/community/tree/master/sig-testing#testing-special-interest-group
11
Kubernetes Testing
Slide 12
Slide 12 text
SIG-Testing
Subprojects:
● prow
● test-infra
● testing-commons
● boskos
12
● gopherage
● gubernator
● repo-infra
● kind
Kubernetes Testing
Slide 13
Slide 13 text
SIG-Testing (cont.)
13
test-infra
Providing tools and configuration files for the testing and
automation needs of the Kubernetes project.
Ref: https://github.com/kubernetes/test-infra
Kubernetes Testing
Slide 14
Slide 14 text
3Cluster E2E Testing
● Introduction
● Kubetest
● Ginkgo & Gomega
● Test Lists
● Writing Test
14
Slide 15
Slide 15 text
E2E Testing
Cluster E2E testing ensures consistent and reliable behavior of
Kubernetes API and catches hard-to-test bugs before user do.
test-infra provides an official tool - Kubetest, which is the interface for
launching and running e2e testing.
Ref:
● E2E testing
● Kubetest
15
Cluster E2E Testing
Slide 16
Slide 16 text
Kubetest
16
Cluster E2E Testing
Slide 17
Slide 17 text
Kubetest Usage
17
Cluster E2E Testing
$ kubetest --build --provider --deployment
--up --test --test_args="--ginkgo.skip(focus)=xxx" --dump --down
Flag:
--build Build binaries (e2e.test, e2e_node.test) for testing.
--provider Specify an alternative provider (gce, local, gke, aks, etc.) for E2E testing, default value is gce.
--deployment Deployment strategies of Kubernetes cluster. (gce, local, gke, aks, etc.)
--up Turn up a new cluster.
--test Run E2E testing.
--test_args Test matching for ginkgo.
--down Shutdown and delete the cluster
--dump Export the result. ( junit xml format )
Slide 18
Slide 18 text
Kubetest Usage (cont.)
18
Cluster E2E Testing
Execute E2E testing on local cluster.
# Build binaries for testing
$ kubetest --build
# Turn up new local cluster
$ kubetest --deployment local --up
# Run all tests
$ kubetest --provider local --deployment local --test
# Delete the cluster
$ kubetest --deployment local --down
# Exexute with one line
$ kubetest --build --provider local --deployment local --up --test --down
Slide 19
Slide 19 text
Kubetest Usage (cont.)
19
Cluster E2E Testing
Notice:
1. Kubetest must run from Kubernetes directory.
2. Kubetest will build the Kubernetes binary, and use binary to create a
new cluster for testing.
Slide 20
Slide 20 text
Kubetest
Wait...
20
Cluster E2E Testing
Slide 21
Slide 21 text
Kubetest (cont.)
1. What is the workflow of kubetest?
2. Where are the test lists?
3. How to write my own test?
4. How to execute test on my cluster?
21
Cluster E2E Testing
Slide 22
Slide 22 text
Kubetest Workflow
What is difference (or relationship)?
● kubetest
● kubernetest/test
● kubernetes/hack/e2e*
22
Cluster E2E Testing
Slide 23
Slide 23 text
Kubetest Workflow (cont.)
23
Cluster E2E Testing
Slide 24
Slide 24 text
Kubetest Workflow (cont.)
24
Cluster E2E Testing
Slide 25
Slide 25 text
Kubetest Workflow
25
kubernetes/hack
/e2e.go
test-infra/kubetest
kubernetes/test/e2e
/e2e_test.go
Start E2E testing.
kubernetes/hack
/ginkgo-e2e.sh
kubernetes/test/e2e/f
ramework/framework
.go
Slide 26
Slide 26 text
Kubetest Workflow
26
kubernetes/hack
/e2e.go
test-infra/kubetest
kubernetes/test/e2e
/e2e_test.go
Start E2E testing.
kubernetes/hack
/ginkgo-e2e.sh
kubernetes/test/e2e/f
ramework/framework
.go
Slide 27
Slide 27 text
So, what is kubernetes/hack/e2e.go ?
27
Cluster E2E Testing
Slide 28
Slide 28 text
hack/e2e.go
28
Cluster E2E Testing
hack/e2e.go program is a wrapper around updating kubetest before
calling it.
Slide 29
Slide 29 text
hack/e2e.go
29
Cluster E2E Testing
Slide 30
Slide 30 text
Kubetest Workflow
30
kubernetes/hack
/e2e.go
test-infra/kubetest
kubernetes/test/e2e
/e2e_test.go
Start E2E testing.
kubernetes/hack
/ginkgo-e2e.sh
kubernetes/test/e2e/f
ramework/framework
.go
Slide 31
Slide 31 text
What is ginkgo ?
31
Cluster E2E Testing
Slide 32
Slide 32 text
Ginkgo & Gomega
The e2e tests in kubernetes are built atop of Ginkgo and Gomega. Ginkgo
is a is Behavior-Driven Development (BDD) style Go testing framework,
and the Gomega is a matcher library paired with Ginkgo.
Ref:
● Ginkgo
● Gomega
32
Cluster E2E Testing
Slide 33
Slide 33 text
Ginko & Gomega
33
Simple example.
https://github.com/pohsienshih/ginkgo-gomega-example
Cluster E2E Testing
Slide 34
Slide 34 text
Ginko & Gomega Example
Ginkgo: A BDD-style Go testing framework.
Gomega: A best matcher library paired with Ginkgo.
Installation:
34
$ go get github.com/onsi/ginkgo/ginkgo
$ go get github.com/onsi/gomega/...
# Install Ginkgo CLI
$ go install github.com/onsi/ginkgo/ginkgo
Cluster E2E Testing
Slide 35
Slide 35 text
Ginko & Gomega Example (cont.)
Main Program
myproject.go
35
package myproject
func Greeting(name string) string{
return "Hello " + name + "."
}
func Calc(number int) int{
return number * 10
}
Cluster E2E Testing
Slide 36
Slide 36 text
Ginko & Gomega Example (cont.)
36
$ cd myproject
$ ls
myproject.go
$ ginkgo bootstrap
$ ls
myproject.go myproject_suite_test.go
Bootstrap a Ginkgo test suite
Test suite will search and exeute all testing specs.
Cluster E2E Testing
Slide 37
Slide 37 text
Ginko & Gomega Example (cont.)
37
myproject_suite_test.go
package myproject_test
import (
"testing"
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
)
func TestMyproject(t *testing.T) {
RegisterFailHandler(Fail)
RunSpecs(t, "Myproject Suite")
}
Cluster E2E Testing
Slide 38
Slide 38 text
Ginko & Gomega Example (cont.)
38
$ ginkgo generate mytest
$ ls
myproject.go myproject_suite_test.go mytest_test.go
Create spec
mytest_test.go
Cluster E2E Testing
package myproject_test
import (
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
. "github.com/myproject"
)
var _ = Describe("Mytest", func() {
})
Slide 39
Slide 39 text
Ginko & Gomega Example (cont.)
39
mytest_test.go (Example 1 )
Cluster E2E Testing
…
var _ = Describe("Mytest", func() {
var name string= "test"
var number int= 99
Describe("Test Greeting function", func() {
Context("Giva a name", func() {
It("Should greeting", func() {
Expect(Greeting(name)).To(Equal("Hello "+name+"."))
})
})
})
Describe("Test Calc function", func() {
Context("Give a number", func() {
It("Should get the correct result", func() {
Expect(Calc(number)).To(Equal(number*10))
})
})
})
})
Spec
Slide 40
Slide 40 text
Ginko & Gomega Example (cont.)
40
mytest_test.go (Example 2)
Use BeforeEach to ensure that the initial value of the variable is the same for all tests.
Cluster E2E Testing
…
var _ = Describe("Mytest", func() {
var number int
BeforeEach(func(){
number = 99
})
Describe("Test Calc function again", func() {
Context("Give a number 99", func() {
It("Should return 990", func() {
Expect(Calc(number)).To(Equal(990))
number = Calc(number2)
})
It("Should return 990, too", func() {
Expect(Calc(number)).To(Equal(990))
})
})
})
})
Slide 41
Slide 41 text
Ginko & Gomega Example (cont.)
41
Launch testing
Cluster E2E Testing
$ ginkgo
# Show all specs
$ ginkgo -v
Running Suite: Myproject Suite
==============================
Random Seed: 1568041760
Will run 4 of 4 specs
•••
Ran 4 of 4 Specs in 0.001 seconds
SUCCESS! -- 4 Passed | 0 Failed | 0 Pending | 0 Skipped
PASS
Ginkgo ran 1 suite in 3.471827325s
Test Suite Passed
Slide 42
Slide 42 text
Kubetest Workflow
42
kubernetes/hack
/e2e.go
test-infra/kubetest
kubernetes/test/e2e
/e2e_test.go
Start E2E testing.
kubernetes/hack
/ginkgo-e2e.sh
kubernetes/test/e2e/f
ramework/framework
.go
Slide 43
Slide 43 text
Ok, now I know the kubetest workflow.
But what/where is the test lists?
43
Cluster E2E Testing
Slide 44
Slide 44 text
Test Lists
kubernetes/test/e2e/e2e_test.go
44
Cluster E2E Testing
Slide 45
Slide 45 text
Test Lists
kubernetes/test/e2e
/e2e.go
45
Execute all specs
Cluster E2E Testing
Slide 46
Slide 46 text
Cluster E2E Testing
Test Lists
kubernetes/test/e2e/e2e_test.go
46
Slide 47
Slide 47 text
Test Lists (cont.)
kubernetes/test/e2e/ui
/dashboard.go
47
Cluster E2E Testing
Slide 48
Slide 48 text
Kinds of tests
48
Cluster E2E Testing
● [Slow]
● [Serial]
● [Disruptive]
● [Internet]
● [Feature:.+]
● [Conformance]
● [LinuxOnly]
● [Privileged]
● [Alpha]
● ...
The label will be located at SIGDescribe or ginkgo.It
Slide 49
Slide 49 text
Kinds of tests (cont.)
49
Cluster E2E Testing
Execute specific kind of test.
* skip/focus can match Describe or ginkgo.It.
# Only execute tests with LinuxOnly label.
$ kubetest --test --test_args="--ginkgo. focus=\[LinuxOnly\]" --provider
local --deployment local
# Skip the tests with LinuxOnly label.
$ kubetest --test --test_args="--ginkgo. skip=\[LinuxOnly\]" --provider
local --deployment local
Slide 50
Slide 50 text
Kubetest Workflow Review
50
kubernetes/hack
/e2e.go
test-infra/kubetest
kubernetes/test/e2e
/e2e_test.go
Start E2E testing.
kubernetes/hack
/ginkgo-e2e.sh
kubernetes/test/e2e/f
ramework/framework
.go
Slide 51
Slide 51 text
Kubetest Workflow Review (cont.)
51
kubernetes/hack/
e2e.go
kubernetes/hack/
ginkgo-e2e.sh
test-infra/kubetest
kubernetes/test/e2e/
e2e.go
kubernetes/_output/
bin/e2e.test
kubernetes/test/e2e/framework
/framework.go
Begin E2E testing.
Find the kubetest,
and update to
latest version.
Deploy the Kubernetes cluster, build
kubernetes/test/e2e/e2e_test.go to
e2e.test file and execute
ginkgo-e2e.sh.
Running Ginkgo CLI to execute
e2e.test. (e2e.test will load all
testing list.)
Ginkgo will first execute
TestE2E() from e2e_test.go, then
TestE2E() will called
runE2Etest() which is in e2e.go.
runE2Etest() will execute
framework.go to begin e2e
testing.
Cluster E2E Testing
Slide 52
Slide 52 text
Kubetest Execution
52
Cluster E2E Testing
Slide 53
Slide 53 text
Writing Test
53
Write your own tests.
Cluster E2E Testing
Slide 54
Slide 54 text
Cluster E2E Testing
Writing Test
Specifications:
1. Debuggability
- Provide as detailed as possible reasons for the failure in its output.
2. Ability to run in non-dedicated test clusters
- Large numbers of tests can be executed in parallel against the same
cluster to reduce delay and improve resource utilization.
3. Speed of execution
- Reduce the execution time as possible of the test.
4. Resilience to relatively rare, temporary infrastructure glitches or delays
- Provide the resilient of the testing.
Ref: Writing Good E2E Tests
54
Slide 55
Slide 55 text
Writing Test (cont.)
55
Cluster E2E Testing
Add new test:
1. Create a folder for your testing in kubernetes/test/e2e.
2. Place the testing files into the folder.
3. Add the BUILD file into the folder.
○ kubernets/test/e2e//BUILD
4. Import your package in
○ kubernets/test/e2e/e2e_test.go
○ kubernets/test/e2e/BUILD
5. Rebuild the e2e.test
○ kubetest --build
* Don’t forget to mark the test with specific label( [Slow], [Serial], etc.) . You can consult #kinds-of-tests to
determine how your test should be marked.
Slide 56
Slide 56 text
Writing Test (cont.)
56
Cluster E2E Testing
https://github.com/pohsienshih/kubernetes-e2e-practice
Slide 57
Slide 57 text
57
Okay, now I know the E2E testing.
But how can I run the E2E testing on my already deployed
cluster rather than create a new one?
Slide 58
Slide 58 text
4Conformance Testing
● Introduction
● Execution
● Writing Test
● Heptio Sonobuoy
● Other Tools
58
Slide 59
Slide 59 text
Conformance Testing
The Kubernetes Conformance test suite is a subset of e2e tests that SIG
architecture has approved to define the core set of interoperable features that all
conformant Kubernetes clusters must support.
You can use conformance testing to test your already deployed cluster.
Ref: Conformance Testing in Kubernetes
59
Conformance Testing
Slide 60
Slide 60 text
Conformance Testing
60
Ref: Conformance Test Requirements
Slide 61
Slide 61 text
Execution
61
# Checking your kubernetes server version
$ kubectl get version
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3" ,
GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2",
GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z",
GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
# Change Kubernetes repository to the specific version
$ cd $GOPATH/src/k8s.io/kubernetes
$ git checkout v1.15.3
# Build e2e.test and kubectl
$ make WHAT="test/e2e/e2e.test vendor/github.com/onsi/ginkgo/ginkgo
cmd/kubectl"
Conformance Testing
Slide 62
Slide 62 text
Execution (cont.)
62
# Setup for conformance tests
$ export KUBECONFIG=/path/to/kubeconfig
$ export KUBERNETES_CONFORMANCE_TEST=y
# Run testing
$ kubetest --provider=skeleton --test
--test_args="--ginkgo.focus=\[Conformance\]"
*The testing time is depend on your environment. (it takes 4~6 hours in my homelab)
* If you want to execute testing by using different client/server version, please use
“--check-version-skew=false” flag.
Conformance Testing
Slide 63
Slide 63 text
Execution (cont.)
63
Conformance Testing
Slide 64
Slide 64 text
Test Lists
64
Conformance Testing
The tests with [Conformance] label.
(kubernetes/test/conformance/testdata/conformance.txt)
Ref:
● Kubernetes Conformance Test Suite - v1.9
● Conformance.txt
Slide 65
Slide 65 text
Test Lists (cont.)
65
Conformance Testing
Add [Conformance] label.
kubernetes/test/e2e/common/pods.go
Slide 66
Slide 66 text
Test Lists (cont.)
66
Conformance Testing
kubernetes/test/e2e/framework/framework.go
Slide 67
Slide 67 text
Writing Test
67
Write your own test.
Conformance Testing
Slide 68
Slide 68 text
Writing Test (cont.)
68
Conformance Testing
The process to add new conformance tests:
1. Write your tests.
2. Make sure your tests are proven to meet the conformance test
requirements.
3. Promote your tests.
Ref: Promoting Tests To Conformance
Slide 69
Slide 69 text
Writing Test (cont.)
69
Conformance Testing
Windows & Linux Considerations
Ref: Windows/Linux Considerations
Slide 70
Slide 70 text
Write Your Tests
70
Conformance Testing
Test Format
1. Use framework.ConformanceIt() instead of framework.It()
2. adds a comment test comment metadata before the ConformanceIt()
/*
Release : v1.15.3
Testname: Kubelet: log output
Description: By default the stdout and stderr from the process being
executed in a pod MUST be sent to the pod's logs.
*/
framework.ConformanceIt ("it should print the output to logs", func() {
...
})
Slide 71
Slide 71 text
Write Your Tests (cont.)
71
Conformance Testing
1. Create a folder for your test in kubernets/test/e2e.
2. Place the test files into the folder.
3. Add the BUILD file into the folder.
○ kubernets/test/e2e//BUILD
4. Import your package in
○ kubernets/test/e2e/e2e_test.go
○ kubernets/test/e2e/BUILD
5. go run test/conformance/walk.go test/e2e >
test/conformance/testdata/conformance.txt
6. Rebuild the e2e.test
○ kubetest --build
* Don’t forget to configure the test with specific label( [Slow], [Serial], etc.).
Slide 72
Slide 72 text
Conformance Test Requirements
72
Conformance Testing
Make sure your tests are proven to meet the conformance test requirements.
Slide 73
Slide 73 text
Promote Your Tests
73
Conformance Testing
Open a PR:
1. Title: "Promote xxx e2e test to Conformance"
2. Includes information and metadata in the description
● /area conformance
● @kubernetes/sig-architecture-pr-reviews @kubernetes/sig-xxx-pr-reviews
@kubernetes/cncf-conformance-wg
● Any necessary information (e.g. explain why a test cannot run on Windows)
3. Add the PR to SIG Architecture's Conformance Test Review board in the To Triage column.
4. Schedule the additional Windows tests
● /test pull-kubernetes-e2e-aks-engine-azure-windows
Slide 74
Slide 74 text
Promote Your Tests (cont.)
74
Conformance Testing
Ref: https://github.com/kubernetes/kubernetes/pull/82198
Slide 75
Slide 75 text
Heptio Sonobuoy
75
Conformance Testing
Sonobuoy is another conformance testing tool developed by VMware Tanzu
(Heptio). It provides easier way to verify your Kubernetes Cluster.
Ref: Sonobuoy
Slide 76
Slide 76 text
Heptio Sonobuoy (cont.)
76
Conformance Testing
Test lists:
1. Same as kubetest conformance testing.
2. Be placed in the Sonobuoy docker image.
* Sonobuoy supports 3 Kubernetes minor versions: the current release and 2
minor versions before.
Slide 77
Slide 77 text
Certified Kubernetes
77
Conformance Testing
Sonobuoy is also the standard conformance tool of CNCF which is used to verify
the Kubernetes deployment tools of the enterprise.
How to get the certification:
https://github.com/cncf/k8s-conformance#certified-kubernetes
Slide 78
Slide 78 text
Execution
78
Conformance Testing
$ export KUBECONFIG=/path/to/kubeconfig
$ go get -u -v github.com/heptio/sonobuoy
# Start testing and wait until they are finished run:
$ sonobuoy run --wait
# Get the result
$ results=$(sonobuoy retrieve)
$ sonobuoy e2e $results
# Delete the objects deployed by sonobuoy
$ sonobuoy delete
Slide 79
Slide 79 text
Result
79
Conformance Testing
● hosts
● meta
● plugins
● podlogs
● resources
● serverversion.json
● servergroup.json
Slide 80
Slide 80 text
Comparation to Kubetest
80
Conformance Testing
Sonobuoy
● Easier to setup
● Doesn’t guarantee to run the latest set of conformance tests.
Kubetest
● Require a bit more work to setup.
● Ensure to run the latest set of conformance tests.
Slide 81
Slide 81 text
5Node E2E Testing
● Introduction
● Execution
81
Slide 82
Slide 82 text
Node E2E Testing
Node E2E tests are component tests meant for testing the Kubelet code on a
custom host environment.
1. Only support Linux host.
2. Only support local and GCE deployment.
Ref: Node E2E Tests
82
Node E2E Testing
Slide 83
Slide 83 text
Execution
83
Node E2E Testing
$ kubetest --build
$ kubetest --test --node-tests
$ go get -u github.com/onsi/ginkgo/ginkgo
$ go get -u github.com/onsi/gomega/...
$ cp $GOPATH/bin/ginkgo /usr/local/bin
# Install etcd
$ sudo PATH=$PATH hack/install-etcd.sh
$ export PATH="/root/go/src/k8s.io/kubernetes/third_party/etcd:${PATH}"
# Run test(local)
$ make test-e2e-node
# Run test(remote)
$ make test-e2e-node REMOTE=true
Execution 1: Use kubetest (only support GCE)
Execution 2: Use make
Slide 84
Slide 84 text
Execution (cont.)
84
NodeE2E Testing
Slide 85
Slide 85 text
Workflow (kubetest)
85
Node E2E Testing 85
kubetest --build
test-infra/kubetest/
e2e.go
kubetest --test
--node-tests
Begin E2E testing.
Build
e2e_node.test.
Execute nodeTest() from
test-infra/kubetest/e2e.go.
kubernetestest/e2e
_node/runner/remo
te/run_remote.go
Execute run_remote.go
Slide 86
Slide 86 text
Workflow (make)
86
Node E2E Testing 86
make
test-e2e-node
kubernetes/hack/
make-rules/
test-e2e-node.sh
kubernetestest/e2e_
node/runner/local/ru
n_local.go
Begin E2E testing.
Build
e2e_node.test.
kubernetestest/e2e_
node/runner/remote/
run_remote.go
Execute
runner_remote.go
or runner_local.go.
Slide 87
Slide 87 text
Test Lists
87
Node E2E Testing
kubernetes/test/e2e_node/
Slide 88
Slide 88 text
6Conclusion
● Conclusion
● References
88
Slide 89
Slide 89 text
Conclusion
89
E2E testing在企業導入Kubernetes流程中是相當重要的一步，無論是使用什麼工具
部署，建議都使用E2E Testing來確認K8s Cluster是否正常，這樣當發生任何問題時，
比較可以排除掉Cluster本身的問題，減少問題定位的成本。
另外也可以定期的執行E2E Testing / Conformance Testing或是直接將測試整合進
CI/CD流程裡(不過最好根據自己的環境狀況來評估)，確保cluster的狀態都是正常
的。
Slide 90
Slide 90 text
References
Software Testing
● Types Of Software Testing: Different Testing Types With Details
Kubernetes Testing
● WIKIPEDIA - Special Interest Group
● Kubernetes SIGs and Working Groups
● Kubernetes Community Governance Model
● Subprojects of each SIGs.
● Sig-Testing
● test-infra
90
Slide 91
Slide 91 text
References
E2E Testing
● Kubernetes E2E Testing
● Kubetest
● Ginkgo
● Gomega
● End-To-End Testing in Kubernetes
● Writing good e2e tests for Kubernetes
Node E2E Testing
● Node-End-To-End Tests
91
Slide 92
Slide 92 text
References
Conformance Testing
● Conformance Testing in Kubernetes
● Kubernetes Conformance Test Suite - v1.9
● Conformance.txt
● Sonobuoy
● CNCF - Certified Kubernetes
92
Slide 93
Slide 93 text
93
Thank you for listening!