Upgrade to Pro — share decks privately, control downloads, hide ads and more …

CNTUG Meetup #20 - End to End Testing for Kubernetes Cluster

Pohsien Shih
September 28, 2019

CNTUG Meetup #20 - End to End Testing for Kubernetes Cluster

現今 Kubernetes Cluster 部署方式越來越多種,部署門檻也越來越低,但是要如何確認自己佈出來的Cluster真的是可用、運作正常的呢?

此議程會帶大家探討 Kubernetes 的 E2E testing,介紹如何使用 Kubernetes
community 官方釋出的工具 kubetest 來對 Kubernetes Cluster 做測試,驗證各種功能是否正常,此外也會解釋 kubetest 的運作流程以及如何客製化自己的測試。

* Kubernetes Testing
* kubetest
* E2E Testing
* Conformance Testing
* Node E2E Testing

# Cloud Native Taiwan User Group Meetup #20

Pohsien Shih

September 28, 2019

More Decks by Pohsien Shih

Other Decks in Technology


  1. End to End Testing for Kubernetes Cluster 施柏賢 Pohsien Shih

    CNTUG Meetup #20
  2. Who am I 施柏賢 Pohsien Shih • Service Engineer •

    CNTUG社群打雜工 Blog: https://pohsienshih.github.io/ Github: https://github.com/pohsienshih Email: pohsien324@gmail.com 2
  3. Outline • Software Testing • Kubernetes Testing • Cluster E2E

    Testing • Conformance Testing • Node E2E Testing • Conclusion • References 3
  4. Why do I need to test the Kubernetes cluster? 4

  5. 1. Make sure your cluster is available. 2. Deploy a

    production ready Kubernetes cluster. 5 The Reasons you should test your cluster.
  6. 1Software Testing • Types of Testing 6

  7. Types of Testing • Unit Testing - Test an individual

    software component or module. • Integration Testing - Test all integrated modules to verify the combined functionality after integration. • End to End (E2E) Testing - Test end-to-end behavior of the system. Software Testing 7
  8. 2Kubernetes Testing • Special Interest Group (SIG) • SIG-Testing •

    test-infra 8
  9. Special Interest Group A Special Interest Group (SIG) is a

    community within a larger organization with a shared interest in advancing a specific area of knowledge, learning or technology where members cooperate to affect or to produce solutions within their particular field, and may communicate, meet, and organize conferences. Ref: wikipedia - Special Interest Group 9 Kubernetes Testing
  10. Special Interest Group (cont.) Kubernetes SIGs • Kubernetes Community Governance

    Model • Kubernetes SIGs and Working Groups Subprojects Every part of the Kubernetes code and documentation must be owned by some subproject. These subprojects will be delivered to different SIGs. • Subprojects of each SIGs 10 Kubernetes Testing
  11. Special Interest Group (cont.) SIG-TESTING The SIG which is responsible

    for Kubernetes testing. Ref: https://github.com/kubernetes/community/tree/master/sig-testing#testing-special-interest-group 11 Kubernetes Testing
  12. SIG-Testing Subprojects: • prow • test-infra • testing-commons • boskos

    12 • gopherage • gubernator • repo-infra • kind Kubernetes Testing
  13. SIG-Testing (cont.) 13 test-infra Providing tools and configuration files for

    the testing and automation needs of the Kubernetes project. Ref: https://github.com/kubernetes/test-infra Kubernetes Testing
  14. 3Cluster E2E Testing • Introduction • Kubetest • Ginkgo &

    Gomega • Test Lists • Writing Test 14
  15. E2E Testing Cluster E2E testing ensures consistent and reliable behavior

    of Kubernetes API and catches hard-to-test bugs before user do. test-infra provides an official tool - Kubetest, which is the interface for launching and running e2e testing. Ref: • E2E testing • Kubetest 15 Cluster E2E Testing
  16. Kubetest 16 Cluster E2E Testing

  17. Kubetest Usage 17 Cluster E2E Testing $ kubetest --build --provider

    <yourprovider> --deployment <yourdeployer> --up --test --test_args="--ginkgo.skip(focus)=xxx" --dump <folder> --down Flag: --build Build binaries (e2e.test, e2e_node.test) for testing. --provider Specify an alternative provider (gce, local, gke, aks, etc.) for E2E testing, default value is gce. --deployment Deployment strategies of Kubernetes cluster. (gce, local, gke, aks, etc.) --up Turn up a new cluster. --test Run E2E testing. --test_args Test matching for ginkgo. --down Shutdown and delete the cluster --dump Export the result. ( junit xml format )
  18. Kubetest Usage (cont.) 18 Cluster E2E Testing Execute E2E testing

    on local cluster. # Build binaries for testing $ kubetest --build # Turn up new local cluster $ kubetest --deployment local --up # Run all tests $ kubetest --provider local --deployment local --test # Delete the cluster $ kubetest --deployment local --down # Exexute with one line $ kubetest --build --provider local --deployment local --up --test --down
  19. Kubetest Usage (cont.) 19 Cluster E2E Testing Notice: 1. Kubetest

    must run from Kubernetes directory. 2. Kubetest will build the Kubernetes binary, and use binary to create a new cluster for testing.
  20. Kubetest Wait... 20 Cluster E2E Testing

  21. Kubetest (cont.) 1. What is the workflow of kubetest? 2.

    Where are the test lists? 3. How to write my own test? 4. How to execute test on my cluster? 21 Cluster E2E Testing
  22. Kubetest Workflow What is difference (or relationship)? • kubetest •

    kubernetest/test • kubernetes/hack/e2e* 22 Cluster E2E Testing
  23. Kubetest Workflow (cont.) 23 Cluster E2E Testing

  24. Kubetest Workflow (cont.) 24 Cluster E2E Testing

  25. Kubetest Workflow 25 kubernetes/hack /e2e.go test-infra/kubetest kubernetes/test/e2e /e2e_test.go Start E2E

    testing. kubernetes/hack /ginkgo-e2e.sh kubernetes/test/e2e/f ramework/framework .go
  26. Kubetest Workflow 26 kubernetes/hack /e2e.go test-infra/kubetest kubernetes/test/e2e /e2e_test.go Start E2E

    testing. kubernetes/hack /ginkgo-e2e.sh kubernetes/test/e2e/f ramework/framework .go
  27. So, what is kubernetes/hack/e2e.go ? 27 Cluster E2E Testing

  28. hack/e2e.go 28 Cluster E2E Testing hack/e2e.go program is a wrapper

    around updating kubetest before calling it.
  29. hack/e2e.go 29 Cluster E2E Testing

  30. Kubetest Workflow 30 kubernetes/hack /e2e.go test-infra/kubetest kubernetes/test/e2e /e2e_test.go Start E2E

    testing. kubernetes/hack /ginkgo-e2e.sh kubernetes/test/e2e/f ramework/framework .go
  31. What is ginkgo ? 31 Cluster E2E Testing

  32. Ginkgo & Gomega The e2e tests in kubernetes are built

    atop of Ginkgo and Gomega. Ginkgo is a is Behavior-Driven Development (BDD) style Go testing framework, and the Gomega is a matcher library paired with Ginkgo. Ref: • Ginkgo • Gomega 32 Cluster E2E Testing
  33. Ginko & Gomega 33 Simple example. https://github.com/pohsienshih/ginkgo-gomega-example Cluster E2E Testing

  34. Ginko & Gomega Example Ginkgo: A BDD-style Go testing framework.

    Gomega: A best matcher library paired with Ginkgo. Installation: 34 $ go get github.com/onsi/ginkgo/ginkgo $ go get github.com/onsi/gomega/... # Install Ginkgo CLI $ go install github.com/onsi/ginkgo/ginkgo Cluster E2E Testing
  35. Ginko & Gomega Example (cont.) Main Program myproject.go 35 package

    myproject func Greeting(name string) string{ return "Hello " + name + "." } func Calc(number int) int{ return number * 10 } Cluster E2E Testing
  36. Ginko & Gomega Example (cont.) 36 $ cd myproject $

    ls myproject.go $ ginkgo bootstrap $ ls myproject.go myproject_suite_test.go Bootstrap a Ginkgo test suite Test suite will search and exeute all testing specs. Cluster E2E Testing
  37. Ginko & Gomega Example (cont.) 37 myproject_suite_test.go package myproject_test import

    ( "testing" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) func TestMyproject(t *testing.T) { RegisterFailHandler(Fail) RunSpecs(t, "Myproject Suite") } Cluster E2E Testing
  38. Ginko & Gomega Example (cont.) 38 $ ginkgo generate mytest

    $ ls myproject.go myproject_suite_test.go mytest_test.go Create spec mytest_test.go Cluster E2E Testing package myproject_test import ( . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" . "github.com/myproject" ) var _ = Describe("Mytest", func() { })
  39. Ginko & Gomega Example (cont.) 39 mytest_test.go (Example 1 )

    Cluster E2E Testing … var _ = Describe("Mytest", func() { var name string= "test" var number int= 99 Describe("Test Greeting function", func() { Context("Giva a name", func() { It("Should greeting", func() { Expect(Greeting(name)).To(Equal("Hello "+name+".")) }) }) }) Describe("Test Calc function", func() { Context("Give a number", func() { It("Should get the correct result", func() { Expect(Calc(number)).To(Equal(number*10)) }) }) }) }) Spec
  40. Ginko & Gomega Example (cont.) 40 mytest_test.go (Example 2) Use

    BeforeEach to ensure that the initial value of the variable is the same for all tests. Cluster E2E Testing … var _ = Describe("Mytest", func() { var number int BeforeEach(func(){ number = 99 }) Describe("Test Calc function again", func() { Context("Give a number 99", func() { It("Should return 990", func() { Expect(Calc(number)).To(Equal(990)) number = Calc(number2) }) It("Should return 990, too", func() { Expect(Calc(number)).To(Equal(990)) }) }) }) })
  41. Ginko & Gomega Example (cont.) 41 Launch testing Cluster E2E

    Testing $ ginkgo # Show all specs $ ginkgo -v Running Suite: Myproject Suite ============================== Random Seed: 1568041760 Will run 4 of 4 specs ••• Ran 4 of 4 Specs in 0.001 seconds SUCCESS! -- 4 Passed | 0 Failed | 0 Pending | 0 Skipped PASS Ginkgo ran 1 suite in 3.471827325s Test Suite Passed
  42. Kubetest Workflow 42 kubernetes/hack /e2e.go test-infra/kubetest kubernetes/test/e2e /e2e_test.go Start E2E

    testing. kubernetes/hack /ginkgo-e2e.sh kubernetes/test/e2e/f ramework/framework .go
  43. Ok, now I know the kubetest workflow. But what/where is

    the test lists? 43 Cluster E2E Testing
  44. Test Lists kubernetes/test/e2e/e2e_test.go 44 Cluster E2E Testing

  45. Test Lists kubernetes/test/e2e /e2e.go 45 Execute all specs Cluster E2E

  46. Cluster E2E Testing Test Lists kubernetes/test/e2e/e2e_test.go 46

  47. Test Lists (cont.) kubernetes/test/e2e/ui /dashboard.go 47 Cluster E2E Testing

  48. Kinds of tests 48 Cluster E2E Testing • [Slow] •

    [Serial] • [Disruptive] • [Internet] • [Feature:.+] • [Conformance] • [LinuxOnly] • [Privileged] • [Alpha] • ... The label will be located at SIGDescribe or ginkgo.It
  49. Kinds of tests (cont.) 49 Cluster E2E Testing Execute specific

    kind of test. * skip/focus can match Describe or ginkgo.It. # Only execute tests with LinuxOnly label. $ kubetest --test --test_args="--ginkgo. focus=\[LinuxOnly\]" --provider local --deployment local # Skip the tests with LinuxOnly label. $ kubetest --test --test_args="--ginkgo. skip=\[LinuxOnly\]" --provider local --deployment local
  50. Kubetest Workflow Review 50 kubernetes/hack /e2e.go test-infra/kubetest kubernetes/test/e2e /e2e_test.go Start

    E2E testing. kubernetes/hack /ginkgo-e2e.sh kubernetes/test/e2e/f ramework/framework .go
  51. Kubetest Workflow Review (cont.) 51 kubernetes/hack/ e2e.go kubernetes/hack/ ginkgo-e2e.sh test-infra/kubetest

    kubernetes/test/e2e/ e2e.go kubernetes/_output/ bin/e2e.test kubernetes/test/e2e/framework /framework.go Begin E2E testing. Find the kubetest, and update to latest version. Deploy the Kubernetes cluster, build kubernetes/test/e2e/e2e_test.go to e2e.test file and execute ginkgo-e2e.sh. Running Ginkgo CLI to execute e2e.test. (e2e.test will load all testing list.) Ginkgo will first execute TestE2E() from e2e_test.go, then TestE2E() will called runE2Etest() which is in e2e.go. runE2Etest() will execute framework.go to begin e2e testing. Cluster E2E Testing
  52. Kubetest Execution 52 Cluster E2E Testing

  53. Writing Test 53 Write your own tests. Cluster E2E Testing

  54. Cluster E2E Testing Writing Test Specifications: 1. Debuggability - Provide

    as detailed as possible reasons for the failure in its output. 2. Ability to run in non-dedicated test clusters - Large numbers of tests can be executed in parallel against the same cluster to reduce delay and improve resource utilization. 3. Speed of execution - Reduce the execution time as possible of the test. 4. Resilience to relatively rare, temporary infrastructure glitches or delays - Provide the resilient of the testing. Ref: Writing Good E2E Tests 54
  55. Writing Test (cont.) 55 Cluster E2E Testing Add new test:

    1. Create a folder for your testing in kubernetes/test/e2e. 2. Place the testing files into the folder. 3. Add the BUILD file into the folder. ◦ kubernets/test/e2e/<yourfolder>/BUILD 4. Import your package in ◦ kubernets/test/e2e/e2e_test.go ◦ kubernets/test/e2e/BUILD 5. Rebuild the e2e.test ◦ kubetest --build * Don’t forget to mark the test with specific label( [Slow], [Serial], etc.) . You can consult #kinds-of-tests to determine how your test should be marked.
  56. Writing Test (cont.) 56 Cluster E2E Testing https://github.com/pohsienshih/kubernetes-e2e-practice

  57. 57 Okay, now I know the E2E testing. But how

    can I run the E2E testing on my already deployed cluster rather than create a new one?
  58. 4Conformance Testing • Introduction • Execution • Writing Test •

    Heptio Sonobuoy • Other Tools 58
  59. Conformance Testing The Kubernetes Conformance test suite is a subset

    of e2e tests that SIG architecture has approved to define the core set of interoperable features that all conformant Kubernetes clusters must support. You can use conformance testing to test your already deployed cluster. Ref: Conformance Testing in Kubernetes 59 Conformance Testing
  60. Conformance Testing 60 Ref: Conformance Test Requirements

  61. Execution 61 # Checking your kubernetes server version $ kubectl

    get version Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3" , GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:50Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} # Change Kubernetes repository to the specific version $ cd $GOPATH/src/k8s.io/kubernetes $ git checkout v1.15.3 # Build e2e.test and kubectl $ make WHAT="test/e2e/e2e.test vendor/github.com/onsi/ginkgo/ginkgo cmd/kubectl" Conformance Testing
  62. Execution (cont.) 62 # Setup for conformance tests $ export

    KUBECONFIG=/path/to/kubeconfig $ export KUBERNETES_CONFORMANCE_TEST=y # Run testing $ kubetest --provider=skeleton --test --test_args="--ginkgo.focus=\[Conformance\]" *The testing time is depend on your environment. (it takes 4~6 hours in my homelab) * If you want to execute testing by using different client/server version, please use “--check-version-skew=false” flag. Conformance Testing
  63. Execution (cont.) 63 Conformance Testing

  64. Test Lists 64 Conformance Testing The tests with [Conformance] label.

    (kubernetes/test/conformance/testdata/conformance.txt) Ref: • Kubernetes Conformance Test Suite - v1.9 • Conformance.txt
  65. Test Lists (cont.) 65 Conformance Testing Add [Conformance] label. kubernetes/test/e2e/common/pods.go

  66. Test Lists (cont.) 66 Conformance Testing kubernetes/test/e2e/framework/framework.go

  67. Writing Test 67 Write your own test. Conformance Testing

  68. Writing Test (cont.) 68 Conformance Testing The process to add

    new conformance tests: 1. Write your tests. 2. Make sure your tests are proven to meet the conformance test requirements. 3. Promote your tests. Ref: Promoting Tests To Conformance
  69. Writing Test (cont.) 69 Conformance Testing Windows & Linux Considerations

    Ref: Windows/Linux Considerations
  70. Write Your Tests 70 Conformance Testing Test Format 1. Use

    framework.ConformanceIt() instead of framework.It() 2. adds a comment test comment metadata before the ConformanceIt() /* Release : v1.15.3 Testname: Kubelet: log output Description: By default the stdout and stderr from the process being executed in a pod MUST be sent to the pod's logs. */ framework.ConformanceIt ("it should print the output to logs", func() { ... })
  71. Write Your Tests (cont.) 71 Conformance Testing 1. Create a

    folder for your test in kubernets/test/e2e. 2. Place the test files into the folder. 3. Add the BUILD file into the folder. ◦ kubernets/test/e2e/<yourfolder>/BUILD 4. Import your package in ◦ kubernets/test/e2e/e2e_test.go ◦ kubernets/test/e2e/BUILD 5. go run test/conformance/walk.go test/e2e > test/conformance/testdata/conformance.txt 6. Rebuild the e2e.test ◦ kubetest --build * Don’t forget to configure the test with specific label( [Slow], [Serial], etc.).
  72. Conformance Test Requirements 72 Conformance Testing Make sure your tests

    are proven to meet the conformance test requirements.
  73. Promote Your Tests 73 Conformance Testing Open a PR: 1.

    Title: "Promote xxx e2e test to Conformance" 2. Includes information and metadata in the description • /area conformance • @kubernetes/sig-architecture-pr-reviews @kubernetes/sig-xxx-pr-reviews @kubernetes/cncf-conformance-wg • Any necessary information (e.g. explain why a test cannot run on Windows) 3. Add the PR to SIG Architecture's Conformance Test Review board in the To Triage column. 4. Schedule the additional Windows tests • /test pull-kubernetes-e2e-aks-engine-azure-windows
  74. Promote Your Tests (cont.) 74 Conformance Testing Ref: https://github.com/kubernetes/kubernetes/pull/82198

  75. Heptio Sonobuoy 75 Conformance Testing Sonobuoy is another conformance testing

    tool developed by VMware Tanzu (Heptio). It provides easier way to verify your Kubernetes Cluster. Ref: Sonobuoy
  76. Heptio Sonobuoy (cont.) 76 Conformance Testing Test lists: 1. Same

    as kubetest conformance testing. 2. Be placed in the Sonobuoy docker image. * Sonobuoy supports 3 Kubernetes minor versions: the current release and 2 minor versions before.
  77. Certified Kubernetes 77 Conformance Testing Sonobuoy is also the standard

    conformance tool of CNCF which is used to verify the Kubernetes deployment tools of the enterprise. How to get the certification: https://github.com/cncf/k8s-conformance#certified-kubernetes
  78. Execution 78 Conformance Testing $ export KUBECONFIG=/path/to/kubeconfig $ go get

    -u -v github.com/heptio/sonobuoy # Start testing and wait until they are finished run: $ sonobuoy run --wait # Get the result $ results=$(sonobuoy retrieve) $ sonobuoy e2e $results # Delete the objects deployed by sonobuoy $ sonobuoy delete
  79. Result 79 Conformance Testing • hosts • meta • plugins

    • podlogs • resources • serverversion.json • servergroup.json
  80. Comparation to Kubetest 80 Conformance Testing Sonobuoy • Easier to

    setup • Doesn’t guarantee to run the latest set of conformance tests. Kubetest • Require a bit more work to setup. • Ensure to run the latest set of conformance tests.
  81. 5Node E2E Testing • Introduction • Execution 81

  82. Node E2E Testing Node E2E tests are component tests meant

    for testing the Kubelet code on a custom host environment. 1. Only support Linux host. 2. Only support local and GCE deployment. Ref: Node E2E Tests 82 Node E2E Testing
  83. Execution 83 Node E2E Testing $ kubetest --build $ kubetest

    --test --node-tests $ go get -u github.com/onsi/ginkgo/ginkgo $ go get -u github.com/onsi/gomega/... $ cp $GOPATH/bin/ginkgo /usr/local/bin # Install etcd $ sudo PATH=$PATH hack/install-etcd.sh $ export PATH="/root/go/src/k8s.io/kubernetes/third_party/etcd:${PATH}" # Run test(local) $ make test-e2e-node # Run test(remote) $ make test-e2e-node REMOTE=true Execution 1: Use kubetest (only support GCE) Execution 2: Use make
  84. Execution (cont.) 84 NodeE2E Testing

  85. Workflow (kubetest) 85 Node E2E Testing 85 kubetest --build test-infra/kubetest/

    e2e.go kubetest --test --node-tests Begin E2E testing. Build e2e_node.test. Execute nodeTest() from test-infra/kubetest/e2e.go. kubernetestest/e2e _node/runner/remo te/run_remote.go Execute run_remote.go
  86. Workflow (make) 86 Node E2E Testing 86 make test-e2e-node kubernetes/hack/

    make-rules/ test-e2e-node.sh kubernetestest/e2e_ node/runner/local/ru n_local.go Begin E2E testing. Build e2e_node.test. kubernetestest/e2e_ node/runner/remote/ run_remote.go Execute runner_remote.go or runner_local.go.
  87. Test Lists 87 Node E2E Testing kubernetes/test/e2e_node/

  88. 6Conclusion • Conclusion • References 88

  89. Conclusion 89 E2E testing在企業導入Kubernetes流程中是相當重要的一步,無論是使用什麼工具 部署,建議都使用E2E Testing來確認K8s Cluster是否正常,這樣當發生任何問題時, 比較可以排除掉Cluster本身的問題,減少問題定位的成本。 另外也可以定期的執行E2E Testing

    / Conformance Testing或是直接將測試整合進 CI/CD流程裡(不過最好根據自己的環境狀況來評估),確保cluster的狀態都是正常 的。
  90. References Software Testing • Types Of Software Testing: Different Testing

    Types With Details Kubernetes Testing • WIKIPEDIA - Special Interest Group • Kubernetes SIGs and Working Groups • Kubernetes Community Governance Model • Subprojects of each SIGs. • Sig-Testing • test-infra 90
  91. References E2E Testing • Kubernetes E2E Testing • Kubetest •

    Ginkgo • Gomega • End-To-End Testing in Kubernetes • Writing good e2e tests for Kubernetes Node E2E Testing • Node-End-To-End Tests 91
  92. References Conformance Testing • Conformance Testing in Kubernetes • Kubernetes

    Conformance Test Suite - v1.9 • Conformance.txt • Sonobuoy • CNCF - Certified Kubernetes 92
  93. 93 Thank you for listening!