你部署的cluster真的可用嗎? 淺談 Kubernetes Cluster 測試之道

大多數的人在部署完 Kubernetes cluster 之後, 都只透過部署一些簡單的 deployment 或是 service 來確認 cluster 是否是正常的, 但是這樣是不太足夠的,
有沒有甚麼方法可以完整測試 cluster, 並找出淺在的問題呢?

此議程將介紹 Kubernetes E2E testing測試方式, 並帶大家探討如何使用 Kubernetes
community 官方釋出的工具 kubetest 來對 Kubernetes cluster 做測試，驗證各種功能是否正常，此外也會解釋 kubetest 的運作流程以及如何客製化自己的測試。

This session will discuss the concept of Kubernetes E2E testing, and use kubetest (The e2e tool developed by Kubernetes community) to verify the Kubernetes cluster. We will also discuss the detail of kubetest workflow and demonstrate how to customize your own testing.

* sig-testing Introduction
* Kubernetes E2E Testing
* Kubetest
* How to Write Your First Test
* Conformance Testing

# Cloud Native Forum Taiwan 2019

Transcript

1. 你部署的cluster真的可用嗎? 淺談 Kubernetes Cluster 測試之道 End to End Testing for Kubernetes Cluster 施柏賢 Pohsien Shih
2. @ Pohsien Who am I 施柏賢 Pohsien Shih • System Engineer • A member of Cloud Native Taiwan User Group (CNTUG) • Cloud Native x Kubernetes x Container Blog: https://pohsienshih.github.io/ Github: https://github.com/pohsienshih Email: pohsien324@gmail.com 2
3. @ Pohsien Outline • Software Testing • Kubernetes Testing • Cluster E2E Testing • Conformance Testing • Conclusion • References 3
4. @ Pohsien Why do I need to test the Kubernetes cluster? 4
5. @ Pohsien 1. Make sure your cluster is available. 2. Deploy a production ready Kubernetes cluster. 5 The Reasons You Should Test Your Cluster
6. @ Pohsien 1Software Testing • Types of Testing 6
7. @ Pohsien Types of Testing • Unit Testing - Test an individual software component or module. • Integration Testing - Test all integrated modules to verify the combined functionality after integration. • End to End (E2E) Testing - Test end-to-end behavior of the system. Software Testing 7
8. @ Pohsien 2Kubernetes Testing • Special Interest Group (SIG) • SIG-Testing • test-infra 8
9. @ Pohsien Special Interest Group A Special Interest Group (SIG) is a community within a larger organization with a shared interest in advancing a specific area of knowledge, learning or technology where members cooperate to affect or to produce solutions within their particular field, and may communicate, meet, and organize conferences. Ref: wikipedia - Special Interest Group 9 Kubernetes Testing
10. @ Pohsien Special Interest Group (cont.) Kubernetes SIGs • Kubernetes Community Governance Model • Kubernetes SIGs and Working Groups Subprojects Every part of the Kubernetes code and documentation must be owned by some subproject. These subprojects will be delivered to different SIGs. • Subprojects of each SIGs 10 Kubernetes Testing
11. @ Pohsien Special Interest Group (cont.) SIG-TESTING The SIG which is responsible for Kubernetes testing. Ref: https://github.com/kubernetes/community/tree/master/sig-testing#testing-special-interest-group 11 Kubernetes Testing
12. @ Pohsien SIG-Testing Subprojects: • prow • test-infra • testing-commons • boskos 12 • gopherage • gubernator • repo-infra • kind Kubernetes Testing
13. @ Pohsien SIG-Testing (cont.) 13 test-infra Providing tools and configuration files for the testing and automation needs of the Kubernetes project. Ref: https://github.com/kubernetes/test-infra Kubernetes Testing
14. @ Pohsien 3Cluster E2E Testing • Introduction • Kubetest • Ginkgo & Gomega • Test Lists • Customize Testing 14
15. @ Pohsien E2E Testing Cluster E2E testing ensures consistent and reliable behavior of Kubernetes API and catches hard-to-test bugs before user do. test-infra provides an official tool - Kubetest, which is the interface for launching and running e2e testing. Ref: • E2E testing • Kubetest 15 Cluster E2E Testing
16. @ Pohsien Kubetest 16 Cluster E2E Testing
17. @ Pohsien Kubetest Usage 17 $ kubetest --build --provider <yourprovider> --deployment <yourdeployer> --up --test --test_args="--ginkgo.skip(focus)=xxx" --dump <folder> --down Cluster E2E Testing Flag: --build Build binaries (e2e.test, e2e_node.test) for testing. --provider Specify an alternative provider (gce, local, gke, aks, etc.) for E2E testing, default value is gce. --deployment Deployment strategies of Kubernetes cluster. (gce, local, gke, aks, etc.) --up Turn up a new cluster. --test Run E2E testing. --test_args Test matching for ginkgo. --down Shutdown and delete the cluster --dump Export the result. ( junit xml format )
18. @ Pohsien Kubetest Usage (cont.) 18 Execute E2E testing on local cluster. Cluster E2E Testing # Build binaries for testing $ kubetest --build # Turn up new local cluster $ kubetest --deployment local --up # Run all tests $ kubetest --provider local --deployment local --test # Delete the cluster $ kubetest --deployment local --down # Exexute with one line $ kubetest --build --provider local --deployment local --up --test --down
19. @ Pohsien Kubetest Usage (cont.) 19 Notice: Kubetest is used to verify the whole of Kubernetes source code, it will build the Kubernetes binary and create a new cluster for testing. Therefore, Kubetest must run from Kubernetes directory. Cluster E2E Testing
20. @ Pohsien Kubetest Wait... 20 Cluster E2E Testing
21. @ Pohsien Kubetest (cont.) 1. What is the workflow of kubetest? 2. Where are the test lists? 3. How to write my own test? 4. How to execute test on my cluster? 21 Cluster E2E Testing
22. @ Pohsien Kubetest Workflow What is difference (or relationship)? • kubetest • kubernetest/test • kubernetes/hack/e2e* 22 Cluster E2E Testing
23. @ Pohsien Kubetest Workflow (cont.) 23 Cluster E2E Testing
24. @ Pohsien Kubetest Workflow (cont.) 24 Cluster E2E Testing
25. @ Pohsien Kubetest Workflow 25 kubernetes/hack /e2e.go test-infra/kubetest kubernetes/test/e2e /e2e_test.go Start E2E testing. kubernetes/hack /ginkgo-e2e.sh kubernetes/test/e2e/f ramework/framework .go
26. @ Pohsien Kubetest Workflow 26 kubernetes/hack /e2e.go test-infra/kubetest kubernetes/test/e2e /e2e_test.go Start E2E testing. kubernetes/hack /ginkgo-e2e.sh kubernetes/test/e2e/f ramework/framework .go
27. @ Pohsien So, what is kubernetes/hack/e2e.go ? 27 Cluster E2E Testing
28. @ Pohsien hack/e2e.go 28 hack/e2e.go program is a wrapper around updating kubetest before calling it. Cluster E2E Testing
29. @ Pohsien hack/e2e.go 29 Cluster E2E Testing
30. @ Pohsien Kubetest Workflow 30 kubernetes/hack /e2e.go test-infra/kubetest kubernetes/test/e2e /e2e_test.go Start E2E testing. kubernetes/hack /ginkgo-e2e.sh kubernetes/test/e2e/f ramework/framework .go
31. @ Pohsien What is ginkgo ? 31 Cluster E2E Testing
32. @ Pohsien The e2e tests in kubernetes are built atop of Ginkgo and Gomega. Ginkgo is a is Behavior-Driven Development (BDD) style Go testing framework, and the Gomega is a matcher library paired with Ginkgo. Ref: • Ginkgo • Gomega Ginkgo & Gomega 32 Cluster E2E Testing
33. @ Pohsien Ginko & Gomega 33 Simple example. https://github.com/pohsienshih/ginkgo-gomega-example Cluster E2E Testing
34. @ Pohsien Ginko & Gomega Example Ginkgo: A BDD-style Go testing framework. Gomega: A best matcher library paired with Ginkgo. 1. Main Program 2. Test Suite 3. Specs (Test Spec1 , Test Spec2 ...) 34 Cluster E2E Testing
35. @ Pohsien Ginko & Gomega Example (cont.) Main Program myproject.go 35 package myproject func Greeting(name string) string{ return "Hello " + name + "." } func Calc(number int) int{ return number * 10 } Cluster E2E Testing
36. @ Pohsien Cluster E2E Testing Ginko & Gomega Example (cont.) 36 Test Suite myproject_suite_test.go package myproject_test import ( "testing" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) func TestMyproject(t *testing.T) { RegisterFailHandler(Fail) RunSpecs(t, "Myproject Suite") }
37. @ Pohsien Ginko & Gomega Example (cont.) 37 Spec: mytest_test.go Cluster E2E Testing … var _ = Describe("Mytest", func() { var name string= "test" var number int= 99 Describe("Test Greeting function", func() { Context("Giva a name", func() { It("Should greeting", func() { Expect(Greeting(name)).To(Equal("Hello "+name+".")) }) }) }) Describe("Test Calc function", func() { Context("Give a number", func() { It("Should get the correct result", func() { Expect(Calc(number)).To(Equal(number*10)) }) }) }) }) Spec
38. @ Pohsien Ginko & Gomega Example (cont.) 38 Launch testing Cluster E2E Testing $ ginkgo # Show all specs $ ginkgo -v Running Suite: Myproject Suite ============================== Random Seed: 1568041760 Will run 4 of 4 specs ••• Ran 4 of 4 Specs in 0.001 seconds SUCCESS! -- 4 Passed | 0 Failed | 0 Pending | 0 Skipped PASS Ginkgo ran 1 suite in 3.471827325s Test Suite Passed
39. @ Pohsien Kubetest Workflow 39 kubernetes/hack /e2e.go test-infra/kubetest kubernetes/test/e2e /e2e_test.go Start E2E testing. kubernetes/hack /ginkgo-e2e.sh kubernetes/test/e2e/f ramework/framework .go
40. @ Pohsien Ok, now I know the kubetest workflow. But what/where is the test lists? 40 Cluster E2E Testing
41. @ Pohsien Test Lists kubernetes/test/e2e/e2e_test.go 41 Cluster E2E Testing
42. @ Pohsien Test Lists kubernetes/test/e2e /e2e.go 42 Execute all specs Cluster E2E Testing
43. @ Pohsien Cluster E2E Testing Test Lists kubernetes/test/e2e/e2e_test.go 43
44. @ Pohsien Test Lists (cont.) kubernetes/test/e2e/ui /dashboard.go 44 Cluster E2E Testing
45. @ Pohsien Kinds of Tests 45 • [Slow] • [Serial] • [Disruptive] • [Internet] • [Feature:.+] • [Conformance] • [LinuxOnly] • [Privileged] • [Alpha] • ... Ref: E2E Testing Kinds of Tests The label will be located at SIGDescribe or ginkgo.It Cluster E2E Testing
46. @ Pohsien Kinds of Tests (cont.) 46 Execute specific kind of test. * skip/focus can match the string from Describe or ginkgo.It. # Only execute tests wwith LinuxOnly label. $ kubetest --test --test_args="--ginkgo. focus=\[LinuxOnly\]" --provider local --deployment local # Skip the tests with LinuxOnly label. $ kubetest --test --test_args="--ginkgo. skip=\[LinuxOnly\]" --provider local --deployment local Cluster E2E Testing
47. @ Pohsien Kubetest Workflow Review 47 kubernetes/hack /e2e.go test-infra/kubetest kubernetes/test/e2e /e2e_test.go Start E2E testing. kubernetes/hack /ginkgo-e2e.sh kubernetes/test/e2e/f ramework/framework .go
48. @ Pohsien Kubetest Workflow Review (cont.) 48 kubernetes/hack/ e2e.go kubernetes/hack/ ginkgo-e2e.sh test-infra/kubetest kubernetes/test/e2e/ e2e.go kubernetes/_output/ bin/e2e.test kubernetes/test/e2e/framework /framework.go Begin E2E testing. Find the kubetest, and update to latest version. Deploy the Kubernetes cluster, build kubernetes/test/e2e/e2e_test.go to e2e.test file and execute ginkgo-e2e.sh. Running Ginkgo CLI to execute e2e.test. (e2e.test will load all testing list.) Ginkgo will first execute TestE2E() from e2e_test.go, then TestE2E() will called runE2Etest() which is in e2e.go. runE2Etest() will execute framework.go to begin e2e testing. Cluster E2E Testing
49. @ Pohsien Kubetest Execution 49 Cluster E2E Testing
50. @ Pohsien Junit Output 50 Cluster E2E Testing • junit_01.xml • junit_runner.xml • metadata.json
51. @ Pohsien Customize Testing 51 Write your own tests. Cluster E2E Testing
52. @ Pohsien Write Your Test Specifications: 1. Debuggability - Provide as detailed as possible reasons for the failure in its output. 2. Ability to run in non-dedicated test clusters - Large numbers of tests can be executed in parallel against the same cluster to reduce delay and improve resource utilization. 3. Speed of execution - Reduce the execution time as possible of the test. 4. Resilience to relatively rare, temporary infrastructure glitches or delays - Provide the resilient of the testing. Ref: Writing Good E2E Tests 52 Cluster E2E Testing
53. @ Pohsien Write Your Test (cont.) 53 Add new test: 1. Create a folder for your testing in kubernets/test/e2e. 2. Place the testing files into the folder. 3. Add the BUILD file into the folder. ◦ kubernets/test/e2e/<yourfolder>/BUILD 4. Import your package in ◦ kubernets/test/e2e/e2e_test.go ◦ kubernets/test/e2e/BUILD 5. Rebuild the e2e.test ◦ kubetest --build * Don’t forget to mark the test with specific labels( [Slow], [Serial], etc.) . You can consult #kinds-of-tests to determine how your test should be marked. Cluster E2E Testing
54. @ Pohsien Cluster E2E Testing Write Your Test (cont.) 54 https://github.com/pohsienshih/kubernetes-e2e-practice
55. @ Pohsien 55 Okay, now I know the E2E testing. But how can I run the E2E testing on my already deployed cluster rather than create a new one?
56. @ Pohsien 4Conformance Testing • Introduction • Execution • Customize Testing • Heptio Sonobuoy • Other Tools 56
57. @ Pohsien Conformance Testing The Kubernetes Conformance test suite is a subset of e2e tests that SIG architecture has approved to define the core set of interoperable features that all conformant Kubernetes clusters must support. You can use conformance testing to test your already deployed cluster. Ref: Conformance Testing in Kubernetes 57 Conformance Testing
58. @ Pohsien Conformance Testing 58 Ref: Conformance Test Requirements
59. @ Pohsien Execution 59 # Checking your kubernetes server version $ kubectl get version Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3" , GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} # Change Kubernetes repository to the specific version $ cd $GOPATH/src/k8s.io/kubernetes $ git checkout v1.15.3 # Build e2e.test and kubectl $ make WHAT="test/e2e/e2e.test vendor/github.com/onsi/ginkgo/ginkgo cmd/kubectl" Conformance Testing
60. @ Pohsien Execution (cont.) 60 # Setup for conformance tests $ export KUBECONFIG=/path/to/kubeconfig $ export KUBERNETES_CONFORMANCE_TEST=y # Run testing $ kubetest --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\]" *The testing time is depend on your environment. (it takes 4~6 hours in my homelab) * If you want to execute testing by using different client/server version, please use “--check-version-skew=false” flag. Conformance Testing
61. @ Pohsien Execution (cont.) 61 Conformance Testing
62. @ Pohsien Test Lists 62 The tests with [Conformance] label. (kubernetes/test/conformance/testdata/conformance.txt) Ref: • Kubernetes Conformance Test Suite - v1.9 • Conformance.txt Conformance Testing
63. @ Pohsien Test Lists (cont.) 63 Add [Conformance] label. kubernetes/test/e2e/common/pods.go Conformance Testing
64. @ Pohsien Test Lists (cont.) 64 kubernetes/test/e2e/framework/framework.go Conformance Testing
65. @ Pohsien Customize Testing 65 Write your own test. Conformance Testing
66. @ Pohsien Customize Testing (cont.) 66 The process to add new conformance tests: 1. Write your test. 2. Make sure your tests are proven to meet the conformance test requirements. 3. Promote your test. Ref: Promoting Tests To Conformance Conformance Testing
67. @ Pohsien Customize Testing (cont.) 67 Windows & Linux Considerations Ref: Windows/Linux Considerations Conformance Testing
68. @ Pohsien Write Your Test 68 Test Format 1. Use framework.ConformanceIt() instead of framework.It() 2. Add a comment test comment metadata before the ConformanceIt() /* Release : v1.15.3 Testname: Kubelet: log output Description: By default the stdout and stderr from the process being executed in a pod MUST be sent to the pod's logs. */ framework.ConformanceIt ("it should print the output to logs", func() { ... }) Conformance Testing
69. @ Pohsien Write Your Test (cont.) 69 1. Create a folder for your test in kubernets/test/e2e. 2. Place the test files into the folder. 3. Add the BUILD file into the folder. ◦ kubernets/test/e2e/<yourfolder>/BUILD 4. Import your package in ◦ kubernets/test/e2e/e2e_test.go ◦ kubernets/test/e2e/BUILD 5. go run test/conformance/walk.go test/e2e > test/conformance/testdata/conformance.txt 6. Rebuild the e2e.test ◦ kubetest --build * Don’t forget to configure the test with specific label( [Slow], [Serial], etc.).
70. @ Pohsien Conformance Test Requirements 70 Conformance Testing Make sure your tests are proven to meet the conformance test requirements.
71. @ Pohsien Promote Your Test 71 Open a PR: 1. Title: "Promote xxx e2e test to Conformance" 2. Includes information and metadata in the description • /area conformance • @kubernetes/sig-architecture-pr-reviews @kubernetes/sig-xxx-pr-reviews @kubernetes/cncf-conformance-wg • Any necessary information (e.g. explain why a test cannot run on Windows) 3. Add the PR to SIG Architecture's Conformance Test Review board in the To Triage column. 4. Schedule the additional Windows tests • /test pull-kubernetes-e2e-aks-engine-azure-windows
72. @ Pohsien Promote Your Test (cont.) 72 Ref: https://github.com/kubernetes/kubernetes/pull/82198 Conformance Testing
73. @ Pohsien Heptio Sonobuoy 73 Sonobuoy is another conformance testing tool developed by VMware Tanzu (Heptio). It provides easiler way to verify your Kubernetes Cluster. Ref: Sonobuoy Conformance Testing
74. @ Pohsien Heptio Sonobuoy (cont.) 74 Test lists: 1. Same as kubetest conformance testing. 2. Be placed in the Sonobuoy docker image. * Sonobuoy supports 3 Kubernetes minor versions: the current release and 2 minor versions before. Conformance Testing
75. @ Pohsien Certified Kubernetes 75 Sonobuoy is also the standard conformance tool of CNCF which is used to verify the Kubernetes deployment tools of the enterprise. How to get the certification: https://github.com/cncf/k8s-conformance#certified-kubernetes Conformance Testing
76. @ Pohsien Certified Kubernetes (cont.) 76 Conformance Testing https://www.cncf.io/certification/software-conformance/
77. @ Pohsien Execution 77 $ export KUBECONFIG=/path/to/kubeconfig $ go get -u -v github.com/heptio/sonobuoy # Start testing and wait until they are finished run: $ sonobuoy run --wait # Get the result $ results=$(sonobuoy retrieve) $ sonobuoy e2e $results # Delete the objects deployed by sonobuoy $ sonobuoy delete Conformance Testing
78. @ Pohsien Result 78 • hosts • meta • plugins • podlogs • resources • serverversion.json • servergroup.json Conformance Testing
79. @ Pohsien Comparation to Kubetest 79 Sonobuoy • Easier to setup • Doesn’t guarantee to run the latest set of conformance tests. Kubetest • Require a bit more work to setup. • Ensure to run the latest set of conformance tests. Conformance Testing
80. @ Pohsien 5Conclusion • Conclusion • References 80
81. @ Pohsien Conclusion 81 E2E testing在企業導入Kubernetes流程中是相當重要的一步，無論是使用什麼工具 部署，建議都使用E2E Testing來確認K8s Cluster是否正常，這樣當發生任何問題時， 比較可以排除掉Cluster本身的問題，減少問題定位的成本。 另外也可以定期的執行E2E Testing / Conformance Testing或是直接將測試整合進 CI/CD流程裡(不過最好根據自己的環境狀況來評估)，確保cluster的狀態都是正常 的。
82. @ Pohsien References Software Testing • Types Of Software Testing: Different Testing Types With Details Kubernetes Testing • WIKIPEDIA - Special Interest Group • Kubernetes SIGs and Working Groups • Kubernetes Community Governance Model • Subprojects of each SIGs. • Sig-Testing • test-infra 82
83. @ Pohsien References E2E Testing • Kubernetes E2E Testing • Kubetest • Ginkgo • Gomega • End-To-End Testing in Kubernetes • Writing good e2e tests for Kubernetes Node E2E Testing • Node-End-To-End Tests 83
84. @ Pohsien References Conformance Testing • Conformance Testing in Kubernetes • Kubernetes Conformance Test Suite - v1.9 • Conformance.txt • Sonobuoy • CNCF - Certified Kubernetes 84
85. @ Pohsien 85 Thank you for listening!
86. @ Pohsien Cloud Native Taiwan User Group 86 Meetup Facebook 十月份 Meetup 10/18 (五) 19:20 天瓏書局 Coding Space