Extending Kubernetes with Go

Transcript

1. Machine Learning for Large Scale Code Analysis Extending Kubernetes With Go Alternative title: Kubernetes Operators with cute Gophers
2. “Kubernetes is just a framework for containers” 2
3. 3 What is an Operator? As seen by CoreOS An Operator is a method of packaging, deploying and managing a Kubernetes application. A Kubernetes application is an application that is both deployed on Kubernetes and managed using the Kubernetes APIs and kubectl tooling. You can think of Operators as the runtime that manages this type of application on Kubernetes.
4. And we all know one… Even though it isn’t officially an operator 4
5. 5 Deployments Seen as an example of operators* “A Deployment controller provides declarative updates for Pods and ReplicaSets.” Deployment ReplicaSet Pod Pod Pod
6. 6 A Kubernetes Controller An overview • Lives in a Pod • Uses in cluster API access • Watched resources, takes action • Stores all data inside Kubernetes • Only 1 instance! (redundancy? Use locks)
7. 7 Operators vs Controllers A hot discussion topic “An operator is a specific type of controller” • Implements CRD • App focused (prometheus, etcd, …) • Term coined by CoreOS More context: https://github.com/kubeflow/tf-operator/issues/300
8. Building a simple controller 8
9. The Toolchain kubernetes/client-go 9
10. The Toolchain • 2nd class citizens • Does not include utils In other languages 10
11. Getting started with client-go 11
12. Connecting package main import ( "k8s.io/client-go/kubernetes" "k8s.io/client-go/rest" ) func main() { config, err := rest.InClusterConfig() clientset, err := kubernetes.NewForConfig(config) } In cluster 12
13. 13 k8s.io/client-go/rest • Underlying REST library • Most important: contains a Config object • Responsible for talking to the APIs • Can be used directly, but you should not Fun fact: still contains very Java like patterns like Factory from the old days. See https://archive.fosdem.org/2019/schedule/event/kubernetesclusterfuck/
14. 14 k8s.io/client-go/kubernetes • Does 1 thing: ClientSets! • Representing objects in code apiVersion: apps/v1 kind: Deployment metadata: name: [...] clientset.AppsV1().Deployments("namespace").Get([...]) clientset.AppsV1().Deployments("namespace").List([...]) clientset.AppsV1().Deployments("namespace").Create([...]) clientset.AppsV1().Deployments("namespace").Delete([...])
15. 15 rest.InClusterConfig() How to see you’re in a cluster • KUBERNETES_SERVICE_HOST && KUBERNETES_SERVICE_PORT • Mounted service tokens
16. Connecting package main import ( "k8s.io/client-go/kubernetes" "k8s.io/client-go/rest" ) func main() { config, err := rest.InClusterConfig() clientset, err := kubernetes.NewForConfig(config) } In cluster 16 And how to test locally???
17. Connecting import ( "k8s.io/client-go/kubernetes" "k8s.io/client-go/tools/clientcmd" ) func main() { config, err := clientcmd.NewNonInteractiveDeferredLoadingClientConfig( clientcmd.NewDefaultClientConfigLoadingRules(), &clientcmd.ConfigOverrides{ CurrentContext: "minikube", }, ).ClientConfig() clientset, err := kubernetes.NewForConfig(config) } Using a config file 17
18. 18 k8s.io/client-go/tools/clientcmd The tool for kubeconfig “Package clientcmd provides one stop shopping for building a working client from a fixed config, from a .kubeconfig file, from command line flags, or from any merged combination.”
19. Watching objects func main() { config, _ := rest.InClusterConfig() // Where is err? I never said this was production code clientset, _ := kubernetes.NewForConfig(config) watcher, err := clientset.CoreV1().Pods("").Watch(metav1.ListOptions{}) for { event := <-watcher.ResultChan() if event.Type == watch.Added { fmt.Printf("Found a new pod: %s\n", (event.Object.(*corev1.Pod).GetName())) } } } Let’s do something useful? 19
20. Watching objects watcher, err := clientset. CoreV1().Pods("").Watch(metav1.ListOptions{}) for { event := <-watcher. ResultChan () if event.Type == watch.Added { fmt. Printf("Found a new pod: %s \n", (event.Object.(*corev1.Pod). GetName())) } } Can you guess the output? 20 All namespaces A B C
21. Watching objects watcher, err := clientset. CoreV1().Pods("").Watch(metav1.ListOptions{}) for { event := <-watcher. ResultChan () if event.Type == watch.Added { fmt. Printf("Found a new pod: %s \n", (event.Object.(*corev1.Pod). GetName())) } } Can you guess the output? 21 All namespaces A B C
22. 22 .Watch() Simple resource watcher • Gives all existing on first call • “Low” level • Error prone (eg. errors on proxy timeouts) • Will repeat all data every x hours ◦ Inefficient with complex code
23. Cache Magic! 23
24. 24 Cache In Kubernetes client-go • Provided in k8s.io/client-go/tools/cache • 2 “Stores”: cache and FIFO queue • Very complex and hard to implement
25. 25 Informers Cache enhanced watchers • Implements a cache • Deals with watcher errors • Does re-sync to not make mistakes • Uses functions for event handling
26. Informers func main() { [...] podInformer := coreinformers.NewPodInformer(clientSet,”namespace”, time.Minute, cache.Indexers{}) podInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) {}, // ability to handle events here UpdateFunc: func(oldObj interface{}, newObj interface{}) {}, DeleteFunc: func(obj interface{}) {}, }) stop := make(chan struct{}) defer close(stop) go podInformer.Run(stop) // runs informer in thread with ability to stop it } Cache enhanced watchers 26
27. 27 Writing Objects All the options! • Create() • Patch() • Update() • UpdateStatus()
28. 28 Writing Objects All the options! • Create() • Patch() • Update() • UpdateStatus()
29. Writing Objects import ( core_v1 "k8s.io/api/core/v1" meta_v1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) func () setUpService(){ service, err = client.CoreV1().Services(pod.GetNamespace()).Create(&core_v1.Service{ ObjectMeta: meta_v1.ObjectMeta{ Name: "Test", Annotations: map[string]string{"heritage": "srcd-operator"}, }, Spec: core_v1.ServiceSpec{ Type: core_v1.ServiceTypeClusterIP, Ports: []core_v1.ServicePort{core_v1.ServicePort{Port: 80}}, }, }) } Simple core v1 service example 29
30. 30 k8s.io/api Has Go structs for all specs
31. 31 client-go takeaways • Offers simple APIs to do CRUD • Offers feature complete tools • One stop for all Kubernetes interactions
32. Operator Framework 32
33. Operator Framework CoreOS’ answer for Kubernetes deployments (now community maintained) 33
34. 34 Operator SDK Go edition • Specifically for “operators” • Does the heavy lifting for you • Manages CRDs • Creates Deployment • Framework handles watchers and actions
35. 35 Operator SDK Sample AppService // newPodForCR returns a busybox pod with the same name/namespace as the cr func newPodForCR (cr *testv1alpha1.AppService) *corev1.Pod { labels := map[string]string{ "app": cr.Name, } return &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: cr.Name + "-pod", Namespace: cr.Namespace, Labels: labels, }, Spec: corev1.PodSpec{ Containers: []corev1.Container{ { Name: "busybox", Image: "busybox", Command: [] string{"sleep", "3600"}, }, }, }, }
36. 36 So much more… That’s for a later talk
37. Coding Challenge 37
38. 38 Boring Wozniak Steve Wozniak is not boring // GetRandomName generates a random name from the list of adjectives and surnames in this package func GetRandomName(retry int) string { begin: name := fmt.Sprintf("%s_%s", left[rand.Intn(len(left))], right[rand.Intn(len(right))]) if name == "boring_wozniak" /* Steve Wozniak is not boring */ { goto begin } if retry > 0 { name = fmt.Sprintf("%s%d", name, rand.Intn(10)) } return name }
39. 39 Boring Wozniak Steve Wozniak is not boring Write a controller that makes sure all boring wozniak deployments get replaced. Instructions & Manifests at: https://github.com/meyskens/boring-wozniak-controller
40. Machine Learning for Large Scale Code Analysis sourced.tech blog.sourced.tech github.com/src-d Thank you! Gopher by Renee French Gopher artwork by Ashley McNamara, Egon Elbre and Miguel Molina