Staying Informed with Kubernetes Informers

Kubernetes state is changing all the time. Pods are being created. Deployments are adding more replicas. Load balancers are being created from services. All of these things can happen without anyone noticing. But sometimes we need to notice, however, for when we need to react to such events. What if we need to push the change to an audit log? When if we want to inform a Slack room about a new deployment? In Kubernetes, this is possible with the informers that are baked into the API and Go client. In this talk we’ll learn how informers work, and how to receive updates when resources change using a simple Go application.

Transcript

1. KUBERNETES INFORMERS STAYING INFORMED WITH
2. WHAT ARE WE TALKING ABOUT? ▸ How Kubernetes organizes objects and how that relates to the libraries that access the API like client-go ▸ Receiving updates about Kubernetes objects ▸ This includes, pods, deployments, anything! ▸ We’ll build a simple audit application to learn them as well
3. WHO AM I? ROBERT ROSS BUT PEOPLE CALL ME BOBBY TABLES @BOBBYTABLES
4. NAMELY DIGITALOCEAN THUNDERBOLT LABS 5 OTHERS THAT WE’LL SKIP PREVIOUSLY WORKED AT
5. I work at An incident response tool
6. KUBERNETES OBJECTS LET’S LEARN ABOUT
7. KUBERNETES OBJECTS ▸ Usually represented as YAML or JSON (but usually YAML) ▸ Describe the desired state of your Kubernetes cluster apiVersion: apps/v1 kind: Deployment metadata: name: rails namespace: laddertruck labels: owner: infra spec: replicas: 20 strategy: type: RollingUpdate
8. SOME OBJECT EXAMPLES ▸ Basically everything in Kubernetes is represented with an object ▸ Deployments ▸ Pods ▸ Volumes ▸ Even IP Endpoints
9. K8S OBJECTS AND THE API ▸ Kubernetes has 2 top level keys that define the object type ▸ The API Version ▸ The Kind ▸ These are used when interacting with the Kubernetes API to construct the path for interacting with the object type
10. apiVersion: apps/v1 kind: Deployment metadata: name: rails namespace: laddertruck labels: owner: infra spec: replicas: 20 strategy: type: RollingUpdate TAKE THIS MANIFEST FOR EXAMPLE
11. apiVersion: apps/v1 kind: Deployment metadata: name: rails namespace: laddertruck labels: owner: infra spec: replicas: 20 strategy: type: RollingUpdate http://localhost:8080/apis/apps/v1/deployments
12. ProTip™ List all available resource types your Kubernetes API has registered $ kubectl api-resources NAME SHORTNAMES NAMESPACED KIND bindings true Binding componentstatuses cs false ComponentStatus configmaps cm true ConfigMap
13. ProTip™ Or if you’re feeling adventurous… $ kubectl proxy --port=8080 & $ curl localhost:8080/apis | jq . { "kind": "APIGroupList", "apiVersion": "v1", "groups": [ { "name": "apiregistration.k8s.io", "versions": [ { "groupVersion": "apiregistration.k8s.io/v1", "version": "v1" }, { "groupVersion": "apiregistration.k8s.io/v1beta1", "version": "v1beta1" } ]… Start an HTTP proxy to the API server locally and then you can 
 look at the raw JSON responses from the API
14. apiVersion: apps/v1 kind: Deployment metadata: name: rails namespace: laddertruck labels: owner: infra spec: replicas: 20 strategy: type: RollingUpdate localhost:8080/apis/apps/v1/namespaces/laddertruck/deployments/rails RETRIEVING A SINGLE RESOURCE REQUIRES THE NAMESPACE IT LIVES UNDER
15. THIS WORKS FOR CUSTOM RESOURCES TOO (LIKE ISTIO CRDS) $ kubectl get crd signalfxs.config.istio.io 2019-06-17T19:14:08Z
16. RELATION TO CLIENT-GO ▸ Understanding these resource conventions is important for using all of the client libraries, namely client-go. ▸ Using the same conventions, we can easily make something that retrieves a list of Pods from the API https://github.com/ashleymcnamara/gophers
17. THE CLIENT-GO STRUCTURE MIMICS THE K8S API ▸ If you understand the relationship of Kubernetes objects to their API endpoints, you can understand the client-go package and so many other libraries
18. A TINY EXAMPLE package main import ( "flag" "fmt" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/kubernetes" "k8s.io/client-go/tools/clientcmd" ) func main() { clientset, err := kubernetes.NewForConfig(config) if err != nil { panic(err.Error()) } pods, err := clientset.CoreV1().Pods("").List(metav1.ListOptions{}) if err != nil { panic(err.Error()) } fmt.Printf("%+v", pods) }
19. A TINY EXAMPLE package main import ( "flag" "fmt" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/kubernetes" "k8s.io/client-go/tools/clientcmd" ) func main() { clientset, err := kubernetes.NewForConfig(config) if err != nil { panic(err.Error()) } pods, err := clientset.CoreV1().Pods("").List(metav1.ListOptions{}) if err != nil { panic(err.Error()) } fmt.Printf("%+v", pods) } http://localhost:8080/api/v1/pods
20. func main() { clientset, err := kubernetes.NewForConfig(config) if err != nil { panic(err.Error()) } pods, err := clientset.CoreV1().Pods("").List(metav1.ListOptions{}) if err != nil { panic(err.Error()) } fmt.Printf("%+v", pods) } http://localhost:8080/api/v1/pods apiVersion: v1 kind: Pod metadata: name: rails namespace: laddertruck labels: owner: infra
21. THIS IS TRUE FOR TYPES AS WELL ▸ The Kubernetes team also publishes the Go packages the same way the API structure works ▸ For example, if you have a “batch/v1” api version, and a “Job” kind, the go import looks like: import "k8s.io/api/batch/v1"
22. BUT THIS IS ABOUT INFORMERS
23. INFORMERS ARE A CRITICAL 
 PIECE OF KUBERNETES ▸ All of the run loops of Kubernetes (pod scheduling, load balancer creation, etc) use Informers ▸ Handle retrieving updates from the Kubernetes API and caching/ indexing them ▸ They’re a framework(?) built into the go client that allows building real time update applications informer.AddEventHandler(cache.Reso AddFunc: func(obj interface{}) // "k8s.io/apimachinery/pkg // interface that allows us mObj := obj.(v1.Object) log.Printf("New Pod Added t }, })
24. BUT HOW?
25. STEP 1
26. RETRIEVE THE LATEST LIST OF RESOURCES ▸ Typically you’ll retrieve the current state of the world of Kubernetes first, for example all pods { "kind": "PodList", "apiVersion": "v1", "metadata": { "selfLink": "/api/v1/namespaces/istio-system/pods", "resourceVersion": "17666" }, "items": [ { "metadata": { "name": "grafana-67c69bb567-c6xcp", "namespace": "istio-system", "resourceVersion": "17024", "creationTimestamp": "2019-06-17T19:14:41Z", "labels": { "app": "grafana", "chart": "grafana", "heritage": "Tiller", curl http://localhost:8080/api/v1/namespaces/istio-system/pods Grab the last resource version 
 in the list
27. STEP 2
28. START A WATCH ON THE SAME PATH ▸ Next, you want to start another request to the same API endpoint ▸ This request needs to have a “watch” parameter and “resourceVersion” ▸ The Kubernetes API will maintain an open connection and send chunked responses anytime a resource updates with a resourceVersion after the given one
29. START A WATCH ON THE SAME PATH curl http://localhost:8080/api/v1/namespaces/istio-system/pods?watch=1&resourceVersion=17909 { "type": "DELETED", "object": { "kind": "Pod", "apiVersion": "v1", "metadata": { "name": "grafana-67c69bb567-c6xcp", "generateName": "grafana-67c69bb567-", "namespace": "istio-system", "resourceVersion": "17933", Resource version is after 
 the given one in the initial request
30. DO IT LIVE
31. WHAT JUST HAPPENED ▸ We hit the Kubernetes API and retrieved a list of pods ▸ We grabbed the highest resourceVersion returned in that list ▸ Then we initiated another request to the same endpoint with a watch parameter to stream the results
32. USING CLIENT-GO
33. CLIENT-GO INFORMERS KUBERNETES API SHARED INFORMER INDEX CONTROLLER QUEUE Push object keys to queue
 when informer sends them
 to the controller Store all objects from the
 watch to an index List and Watch resources from API Pop object keys and do the work
 we want to do Initialize an
 informer for our controller Retrieve the object from
 the index when processing
34. LET’S AUDIT WITH THE GO CLIENT
35. DEMO LET’S BUILD A SIMPLE INFORMER ▸ We’re going to walk through a simple informer now that retrieves updates for Deployments ▸ We’re going to write the updates to a file as JSON ▸ Then drink a beer ▸ Let’s do it!
36. NICE.
37. THIS IS HOW FIREHYDRANT AUDITS ▸ FireHydrant built our Kubernetes integration with these same ideas ▸ Informers are great because we can diff changes that are happening to your Kubernetes cluster too ▸ Without them we’d have no practical way to audit Kubernetes changes into our change log
38. WHICH MEANS WE GET A LIST OF CHANGE EVENTS
39. WINDDOWN
40. INFORMERS ARE POWERFUL ▸ We’ve covered how Kubernetes objects are organized and how the API is structured ▸ We also covered how to retrieve updates from the Kubernetes API ▸ Finally, we used all of these pieces to build a simple audit tool for deployments
41. THANKS! Bobby bobby@firehydrant.io Checkout firehydrant.io for a
 “no big deal” incident response tool!