CNTUG Meetup #20 - End to End Testing for Kubernetes Cluster

現今 Kubernetes Cluster 部署方式越來越多種，部署門檻也越來越低，但是要如何確認自己佈出來的Cluster真的是可用、運作正常的呢？

此議程會帶大家探討 Kubernetes 的 E2E testing，介紹如何使用 Kubernetes
community 官方釋出的工具 kubetest 來對 Kubernetes Cluster 做測試，驗證各種功能是否正常，此外也會解釋 kubetest 的運作流程以及如何客製化自己的測試。

Outline:
* Kubernetes Testing
* kubetest
* E2E Testing
* Conformance Testing
* Node E2E Testing

# Cloud Native Taiwan User Group Meetup #20

Transcript

1. End to End Testing for Kubernetes Cluster 施柏賢 Pohsien Shih CNTUG Meetup #20
2. Who am I 施柏賢 Pohsien Shih • Service Engineer • CNTUG社群打雜工 Blog: https://pohsienshih.github.io/ Github: https://github.com/pohsienshih Email: pohsien324@gmail.com 2
3. Outline • Software Testing • Kubernetes Testing • Cluster E2E Testing • Conformance Testing • Node E2E Testing • Conclusion • References 3
4. Why do I need to test the Kubernetes cluster? 4
5. 1. Make sure your cluster is available. 2. Deploy a production ready Kubernetes cluster. 5 The Reasons you should test your cluster.
6. 1Software Testing • Types of Testing 6
7. Types of Testing • Unit Testing - Test an individual software component or module. • Integration Testing - Test all integrated modules to verify the combined functionality after integration. • End to End (E2E) Testing - Test end-to-end behavior of the system. Software Testing 7
8. 2Kubernetes Testing • Special Interest Group (SIG) • SIG-Testing • test-infra 8
9. Special Interest Group A Special Interest Group (SIG) is a community within a larger organization with a shared interest in advancing a specific area of knowledge, learning or technology where members cooperate to affect or to produce solutions within their particular field, and may communicate, meet, and organize conferences. Ref: wikipedia - Special Interest Group 9 Kubernetes Testing
10. Special Interest Group (cont.) Kubernetes SIGs • Kubernetes Community Governance Model • Kubernetes SIGs and Working Groups Subprojects Every part of the Kubernetes code and documentation must be owned by some subproject. These subprojects will be delivered to different SIGs. • Subprojects of each SIGs 10 Kubernetes Testing
11. Special Interest Group (cont.) SIG-TESTING The SIG which is responsible for Kubernetes testing. Ref: https://github.com/kubernetes/community/tree/master/sig-testing#testing-special-interest-group 11 Kubernetes Testing
12. SIG-Testing Subprojects: • prow • test-infra • testing-commons • boskos 12 • gopherage • gubernator • repo-infra • kind Kubernetes Testing
13. SIG-Testing (cont.) 13 test-infra Providing tools and configuration files for the testing and automation needs of the Kubernetes project. Ref: https://github.com/kubernetes/test-infra Kubernetes Testing
14. 3Cluster E2E Testing • Introduction • Kubetest • Ginkgo & Gomega • Test Lists • Writing Test 14
15. E2E Testing Cluster E2E testing ensures consistent and reliable behavior of Kubernetes API and catches hard-to-test bugs before user do. test-infra provides an official tool - Kubetest, which is the interface for launching and running e2e testing. Ref: • E2E testing • Kubetest 15 Cluster E2E Testing
16. Kubetest 16 Cluster E2E Testing
17. Kubetest Usage 17 Cluster E2E Testing $ kubetest --build --provider <yourprovider> --deployment <yourdeployer> --up --test --test_args="--ginkgo.skip(focus)=xxx" --dump <folder> --down Flag: --build Build binaries (e2e.test, e2e_node.test) for testing. --provider Specify an alternative provider (gce, local, gke, aks, etc.) for E2E testing, default value is gce. --deployment Deployment strategies of Kubernetes cluster. (gce, local, gke, aks, etc.) --up Turn up a new cluster. --test Run E2E testing. --test_args Test matching for ginkgo. --down Shutdown and delete the cluster --dump Export the result. ( junit xml format )
18. Kubetest Usage (cont.) 18 Cluster E2E Testing Execute E2E testing on local cluster. # Build binaries for testing $ kubetest --build # Turn up new local cluster $ kubetest --deployment local --up # Run all tests $ kubetest --provider local --deployment local --test # Delete the cluster $ kubetest --deployment local --down # Exexute with one line $ kubetest --build --provider local --deployment local --up --test --down
19. Kubetest Usage (cont.) 19 Cluster E2E Testing Notice: 1. Kubetest must run from Kubernetes directory. 2. Kubetest will build the Kubernetes binary, and use binary to create a new cluster for testing.
20. Kubetest Wait... 20 Cluster E2E Testing
21. Kubetest (cont.) 1. What is the workflow of kubetest? 2. Where are the test lists? 3. How to write my own test? 4. How to execute test on my cluster? 21 Cluster E2E Testing
22. Kubetest Workflow What is difference (or relationship)? • kubetest • kubernetest/test • kubernetes/hack/e2e* 22 Cluster E2E Testing
23. Kubetest Workflow (cont.) 23 Cluster E2E Testing
24. Kubetest Workflow (cont.) 24 Cluster E2E Testing
25. Kubetest Workflow 25 kubernetes/hack /e2e.go test-infra/kubetest kubernetes/test/e2e /e2e_test.go Start E2E testing. kubernetes/hack /ginkgo-e2e.sh kubernetes/test/e2e/f ramework/framework .go
26. Kubetest Workflow 26 kubernetes/hack /e2e.go test-infra/kubetest kubernetes/test/e2e /e2e_test.go Start E2E testing. kubernetes/hack /ginkgo-e2e.sh kubernetes/test/e2e/f ramework/framework .go
27. So, what is kubernetes/hack/e2e.go ? 27 Cluster E2E Testing
28. hack/e2e.go 28 Cluster E2E Testing hack/e2e.go program is a wrapper around updating kubetest before calling it.
29. hack/e2e.go 29 Cluster E2E Testing
30. Kubetest Workflow 30 kubernetes/hack /e2e.go test-infra/kubetest kubernetes/test/e2e /e2e_test.go Start E2E testing. kubernetes/hack /ginkgo-e2e.sh kubernetes/test/e2e/f ramework/framework .go
31. What is ginkgo ? 31 Cluster E2E Testing
32. Ginkgo & Gomega The e2e tests in kubernetes are built atop of Ginkgo and Gomega. Ginkgo is a is Behavior-Driven Development (BDD) style Go testing framework, and the Gomega is a matcher library paired with Ginkgo. Ref: • Ginkgo • Gomega 32 Cluster E2E Testing
33. Ginko & Gomega 33 Simple example. https://github.com/pohsienshih/ginkgo-gomega-example Cluster E2E Testing
34. Ginko & Gomega Example Ginkgo: A BDD-style Go testing framework. Gomega: A best matcher library paired with Ginkgo. Installation: 34 $ go get github.com/onsi/ginkgo/ginkgo $ go get github.com/onsi/gomega/... # Install Ginkgo CLI $ go install github.com/onsi/ginkgo/ginkgo Cluster E2E Testing
35. Ginko & Gomega Example (cont.) Main Program myproject.go 35 package myproject func Greeting(name string) string{ return "Hello " + name + "." } func Calc(number int) int{ return number * 10 } Cluster E2E Testing
36. Ginko & Gomega Example (cont.) 36 $ cd myproject $ ls myproject.go $ ginkgo bootstrap $ ls myproject.go myproject_suite_test.go Bootstrap a Ginkgo test suite Test suite will search and exeute all testing specs. Cluster E2E Testing
37. Ginko & Gomega Example (cont.) 37 myproject_suite_test.go package myproject_test import ( "testing" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) func TestMyproject(t *testing.T) { RegisterFailHandler(Fail) RunSpecs(t, "Myproject Suite") } Cluster E2E Testing
38. Ginko & Gomega Example (cont.) 38 $ ginkgo generate mytest $ ls myproject.go myproject_suite_test.go mytest_test.go Create spec mytest_test.go Cluster E2E Testing package myproject_test import ( . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" . "github.com/myproject" ) var _ = Describe("Mytest", func() { })
39. Ginko & Gomega Example (cont.) 39 mytest_test.go (Example 1 ) Cluster E2E Testing … var _ = Describe("Mytest", func() { var name string= "test" var number int= 99 Describe("Test Greeting function", func() { Context("Giva a name", func() { It("Should greeting", func() { Expect(Greeting(name)).To(Equal("Hello "+name+".")) }) }) }) Describe("Test Calc function", func() { Context("Give a number", func() { It("Should get the correct result", func() { Expect(Calc(number)).To(Equal(number*10)) }) }) }) }) Spec
40. Ginko & Gomega Example (cont.) 40 mytest_test.go (Example 2) Use BeforeEach to ensure that the initial value of the variable is the same for all tests. Cluster E2E Testing … var _ = Describe("Mytest", func() { var number int BeforeEach(func(){ number = 99 }) Describe("Test Calc function again", func() { Context("Give a number 99", func() { It("Should return 990", func() { Expect(Calc(number)).To(Equal(990)) number = Calc(number2) }) It("Should return 990, too", func() { Expect(Calc(number)).To(Equal(990)) }) }) }) })
41. Ginko & Gomega Example (cont.) 41 Launch testing Cluster E2E Testing $ ginkgo # Show all specs $ ginkgo -v Running Suite: Myproject Suite ============================== Random Seed: 1568041760 Will run 4 of 4 specs ••• Ran 4 of 4 Specs in 0.001 seconds SUCCESS! -- 4 Passed | 0 Failed | 0 Pending | 0 Skipped PASS Ginkgo ran 1 suite in 3.471827325s Test Suite Passed
42. Kubetest Workflow 42 kubernetes/hack /e2e.go test-infra/kubetest kubernetes/test/e2e /e2e_test.go Start E2E testing. kubernetes/hack /ginkgo-e2e.sh kubernetes/test/e2e/f ramework/framework .go
43. Ok, now I know the kubetest workflow. But what/where is the test lists? 43 Cluster E2E Testing
44. Test Lists kubernetes/test/e2e/e2e_test.go 44 Cluster E2E Testing
45. Test Lists kubernetes/test/e2e /e2e.go 45 Execute all specs Cluster E2E Testing
46. Cluster E2E Testing Test Lists kubernetes/test/e2e/e2e_test.go 46
47. Test Lists (cont.) kubernetes/test/e2e/ui /dashboard.go 47 Cluster E2E Testing
48. Kinds of tests 48 Cluster E2E Testing • [Slow] • [Serial] • [Disruptive] • [Internet] • [Feature:.+] • [Conformance] • [LinuxOnly] • [Privileged] • [Alpha] • ... The label will be located at SIGDescribe or ginkgo.It
49. Kinds of tests (cont.) 49 Cluster E2E Testing Execute specific kind of test. * skip/focus can match Describe or ginkgo.It. # Only execute tests with LinuxOnly label. $ kubetest --test --test_args="--ginkgo. focus=\[LinuxOnly\]" --provider local --deployment local # Skip the tests with LinuxOnly label. $ kubetest --test --test_args="--ginkgo. skip=\[LinuxOnly\]" --provider local --deployment local
50. Kubetest Workflow Review 50 kubernetes/hack /e2e.go test-infra/kubetest kubernetes/test/e2e /e2e_test.go Start E2E testing. kubernetes/hack /ginkgo-e2e.sh kubernetes/test/e2e/f ramework/framework .go
51. Kubetest Workflow Review (cont.) 51 kubernetes/hack/ e2e.go kubernetes/hack/ ginkgo-e2e.sh test-infra/kubetest kubernetes/test/e2e/ e2e.go kubernetes/_output/ bin/e2e.test kubernetes/test/e2e/framework /framework.go Begin E2E testing. Find the kubetest, and update to latest version. Deploy the Kubernetes cluster, build kubernetes/test/e2e/e2e_test.go to e2e.test file and execute ginkgo-e2e.sh. Running Ginkgo CLI to execute e2e.test. (e2e.test will load all testing list.) Ginkgo will first execute TestE2E() from e2e_test.go, then TestE2E() will called runE2Etest() which is in e2e.go. runE2Etest() will execute framework.go to begin e2e testing. Cluster E2E Testing
52. Kubetest Execution 52 Cluster E2E Testing
53. Writing Test 53 Write your own tests. Cluster E2E Testing
54. Cluster E2E Testing Writing Test Specifications: 1. Debuggability - Provide as detailed as possible reasons for the failure in its output. 2. Ability to run in non-dedicated test clusters - Large numbers of tests can be executed in parallel against the same cluster to reduce delay and improve resource utilization. 3. Speed of execution - Reduce the execution time as possible of the test. 4. Resilience to relatively rare, temporary infrastructure glitches or delays - Provide the resilient of the testing. Ref: Writing Good E2E Tests 54
55. Writing Test (cont.) 55 Cluster E2E Testing Add new test: 1. Create a folder for your testing in kubernetes/test/e2e. 2. Place the testing files into the folder. 3. Add the BUILD file into the folder. ◦ kubernets/test/e2e/<yourfolder>/BUILD 4. Import your package in ◦ kubernets/test/e2e/e2e_test.go ◦ kubernets/test/e2e/BUILD 5. Rebuild the e2e.test ◦ kubetest --build * Don’t forget to mark the test with specific label( [Slow], [Serial], etc.) . You can consult #kinds-of-tests to determine how your test should be marked.
56. Writing Test (cont.) 56 Cluster E2E Testing https://github.com/pohsienshih/kubernetes-e2e-practice
57. 57 Okay, now I know the E2E testing. But how can I run the E2E testing on my already deployed cluster rather than create a new one?
58. 4Conformance Testing • Introduction • Execution • Writing Test • Heptio Sonobuoy • Other Tools 58
59. Conformance Testing The Kubernetes Conformance test suite is a subset of e2e tests that SIG architecture has approved to define the core set of interoperable features that all conformant Kubernetes clusters must support. You can use conformance testing to test your already deployed cluster. Ref: Conformance Testing in Kubernetes 59 Conformance Testing
60. Conformance Testing 60 Ref: Conformance Test Requirements
61. Execution 61 # Checking your kubernetes server version $ kubectl get version Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3" , GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} # Change Kubernetes repository to the specific version $ cd $GOPATH/src/k8s.io/kubernetes $ git checkout v1.15.3 # Build e2e.test and kubectl $ make WHAT="test/e2e/e2e.test vendor/github.com/onsi/ginkgo/ginkgo cmd/kubectl" Conformance Testing
62. Execution (cont.) 62 # Setup for conformance tests $ export KUBECONFIG=/path/to/kubeconfig $ export KUBERNETES_CONFORMANCE_TEST=y # Run testing $ kubetest --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\]" *The testing time is depend on your environment. (it takes 4~6 hours in my homelab) * If you want to execute testing by using different client/server version, please use “--check-version-skew=false” flag. Conformance Testing
63. Execution (cont.) 63 Conformance Testing
64. Test Lists 64 Conformance Testing The tests with [Conformance] label. (kubernetes/test/conformance/testdata/conformance.txt) Ref: • Kubernetes Conformance Test Suite - v1.9 • Conformance.txt
65. Test Lists (cont.) 65 Conformance Testing Add [Conformance] label. kubernetes/test/e2e/common/pods.go
66. Test Lists (cont.) 66 Conformance Testing kubernetes/test/e2e/framework/framework.go
67. Writing Test 67 Write your own test. Conformance Testing
68. Writing Test (cont.) 68 Conformance Testing The process to add new conformance tests: 1. Write your tests. 2. Make sure your tests are proven to meet the conformance test requirements. 3. Promote your tests. Ref: Promoting Tests To Conformance
69. Writing Test (cont.) 69 Conformance Testing Windows & Linux Considerations Ref: Windows/Linux Considerations
70. Write Your Tests 70 Conformance Testing Test Format 1. Use framework.ConformanceIt() instead of framework.It() 2. adds a comment test comment metadata before the ConformanceIt() /* Release : v1.15.3 Testname: Kubelet: log output Description: By default the stdout and stderr from the process being executed in a pod MUST be sent to the pod's logs. */ framework.ConformanceIt ("it should print the output to logs", func() { ... })
71. Write Your Tests (cont.) 71 Conformance Testing 1. Create a folder for your test in kubernets/test/e2e. 2. Place the test files into the folder. 3. Add the BUILD file into the folder. ◦ kubernets/test/e2e/<yourfolder>/BUILD 4. Import your package in ◦ kubernets/test/e2e/e2e_test.go ◦ kubernets/test/e2e/BUILD 5. go run test/conformance/walk.go test/e2e > test/conformance/testdata/conformance.txt 6. Rebuild the e2e.test ◦ kubetest --build * Don’t forget to configure the test with specific label( [Slow], [Serial], etc.).
72. Conformance Test Requirements 72 Conformance Testing Make sure your tests are proven to meet the conformance test requirements.
73. Promote Your Tests 73 Conformance Testing Open a PR: 1. Title: "Promote xxx e2e test to Conformance" 2. Includes information and metadata in the description • /area conformance • @kubernetes/sig-architecture-pr-reviews @kubernetes/sig-xxx-pr-reviews @kubernetes/cncf-conformance-wg • Any necessary information (e.g. explain why a test cannot run on Windows) 3. Add the PR to SIG Architecture's Conformance Test Review board in the To Triage column. 4. Schedule the additional Windows tests • /test pull-kubernetes-e2e-aks-engine-azure-windows
74. Promote Your Tests (cont.) 74 Conformance Testing Ref: https://github.com/kubernetes/kubernetes/pull/82198
75. Heptio Sonobuoy 75 Conformance Testing Sonobuoy is another conformance testing tool developed by VMware Tanzu (Heptio). It provides easier way to verify your Kubernetes Cluster. Ref: Sonobuoy
76. Heptio Sonobuoy (cont.) 76 Conformance Testing Test lists: 1. Same as kubetest conformance testing. 2. Be placed in the Sonobuoy docker image. * Sonobuoy supports 3 Kubernetes minor versions: the current release and 2 minor versions before.
77. Certified Kubernetes 77 Conformance Testing Sonobuoy is also the standard conformance tool of CNCF which is used to verify the Kubernetes deployment tools of the enterprise. How to get the certification: https://github.com/cncf/k8s-conformance#certified-kubernetes
78. Execution 78 Conformance Testing $ export KUBECONFIG=/path/to/kubeconfig $ go get -u -v github.com/heptio/sonobuoy # Start testing and wait until they are finished run: $ sonobuoy run --wait # Get the result $ results=$(sonobuoy retrieve) $ sonobuoy e2e $results # Delete the objects deployed by sonobuoy $ sonobuoy delete
79. Result 79 Conformance Testing • hosts • meta • plugins • podlogs • resources • serverversion.json • servergroup.json
80. Comparation to Kubetest 80 Conformance Testing Sonobuoy • Easier to setup • Doesn’t guarantee to run the latest set of conformance tests. Kubetest • Require a bit more work to setup. • Ensure to run the latest set of conformance tests.
81. 5Node E2E Testing • Introduction • Execution 81
82. Node E2E Testing Node E2E tests are component tests meant for testing the Kubelet code on a custom host environment. 1. Only support Linux host. 2. Only support local and GCE deployment. Ref: Node E2E Tests 82 Node E2E Testing
83. Execution 83 Node E2E Testing $ kubetest --build $ kubetest --test --node-tests $ go get -u github.com/onsi/ginkgo/ginkgo $ go get -u github.com/onsi/gomega/... $ cp $GOPATH/bin/ginkgo /usr/local/bin # Install etcd $ sudo PATH=$PATH hack/install-etcd.sh $ export PATH="/root/go/src/k8s.io/kubernetes/third_party/etcd:${PATH}" # Run test(local) $ make test-e2e-node # Run test(remote) $ make test-e2e-node REMOTE=true Execution 1: Use kubetest (only support GCE) Execution 2: Use make
84. Execution (cont.) 84 NodeE2E Testing
85. Workflow (kubetest) 85 Node E2E Testing 85 kubetest --build test-infra/kubetest/ e2e.go kubetest --test --node-tests Begin E2E testing. Build e2e_node.test. Execute nodeTest() from test-infra/kubetest/e2e.go. kubernetestest/e2e _node/runner/remo te/run_remote.go Execute run_remote.go
86. Workflow (make) 86 Node E2E Testing 86 make test-e2e-node kubernetes/hack/ make-rules/ test-e2e-node.sh kubernetestest/e2e_ node/runner/local/ru n_local.go Begin E2E testing. Build e2e_node.test. kubernetestest/e2e_ node/runner/remote/ run_remote.go Execute runner_remote.go or runner_local.go.
87. Test Lists 87 Node E2E Testing kubernetes/test/e2e_node/
88. 6Conclusion • Conclusion • References 88
89. Conclusion 89 E2E testing在企業導入Kubernetes流程中是相當重要的一步，無論是使用什麼工具 部署，建議都使用E2E Testing來確認K8s Cluster是否正常，這樣當發生任何問題時， 比較可以排除掉Cluster本身的問題，減少問題定位的成本。 另外也可以定期的執行E2E Testing / Conformance Testing或是直接將測試整合進 CI/CD流程裡(不過最好根據自己的環境狀況來評估)，確保cluster的狀態都是正常 的。
90. References Software Testing • Types Of Software Testing: Different Testing Types With Details Kubernetes Testing • WIKIPEDIA - Special Interest Group • Kubernetes SIGs and Working Groups • Kubernetes Community Governance Model • Subprojects of each SIGs. • Sig-Testing • test-infra 90
91. References E2E Testing • Kubernetes E2E Testing • Kubetest • Ginkgo • Gomega • End-To-End Testing in Kubernetes • Writing good e2e tests for Kubernetes Node E2E Testing • Node-End-To-End Tests 91
92. References Conformance Testing • Conformance Testing in Kubernetes • Kubernetes Conformance Test Suite - v1.9 • Conformance.txt • Sonobuoy • CNCF - Certified Kubernetes 92
93. 93 Thank you for listening!