Customizing and scaling Kubernetes for News

At Bonnier News we autoscale our Kubernetes cluster to prepare for the unknown, support our devs and increase efficiency of our cluster resources. By using custom controllers we can also autoscale using spot nodes for specific applications.

Transcript

1. …
4. Customizing and scaling Kubernetes for News Magnus Bengtsson magnus.bengtsson@bonniernews.se @cldmnky
5. Agenda • Why I’m here • About controllers, Custom resource definitions and Admission web hooks • How we use autoscaling and controllers in our clusters
9. Containers
10. Where we are today
12. What’s a controller in k8s?
13. ”In Kubernetes, a controller is a control loop that watches the shared state of the cluster through the apiserver and makes changes attempting to move the current state towards the desired state. Examples of controllers that ship with Kubernetes today are the replication controller, endpoints controller, namespace controller, and serviceaccounts controller.” tldr;
14. Controllers # Observe changes to services, including the clusterIP and invoke a script for each $ oc observe services -a '{ .spec.clusterIP }' -- register_dns.sh
15. Controllers and owners • Kubectl create -f deployment.yaml • A deployment controller watches deployment api objects. • New -> create a replicaset object with a owner reference ((write back status) • The replicaset controller will create pods and set their owner references. • The kubelet will create containers. • Controllers doesn’t always need to create objects within kubernetes,. • Do something based on the changed state • … • But they always watch api objects for changes
16. Yaml! apiVersion: extensions/v1beta1 kind: Deployment spec: replicas: 2 revisionHistoryLimit: 5 strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 1 type: RollingUpdate status: availableReplicas: 2 conditions: - lastTransitionTime: 2019-01-24T16:06:42Z lastUpdateTime: 2019-01-24T16:06:42Z message: Deployment has minimum availability. reason: MinimumReplicasAvailable status: "True" type: Available observedGeneration: 9 readyReplicas: 2 replicas: 2 updatedReplicas: 2 apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: name: videoplayer-989ff9f7c ownerReferences: - apiVersion: extensions/v1beta1 blockOwnerDeletion: true controller: true kind: Deployment name: videoplayer spec: replicas: 2 selector: matchLabels: app: videoplayer status: availableReplicas: 2 fullyLabeledReplicas: 2 observedGeneration: 2 readyReplicas: 2 replicas: 2 - apiVersion: v1 kind: Pod name: videoplayer-989ff9f7c-qm8sp ownerReferences: - apiVersion: extensions/v1beta1 blockOwnerDeletion: true controller: true kind: ReplicaSet name: videoplayer-989ff9f7c uid: 98663d56-61fb-11e9-9d75-0a3e78805272 kubectl apply -f deployment.yaml
17. Custom Resource Definitions • Your own api object inside kubernetes • Listen to events for your api object • Reconcile it! apiVersion: crew.testproject.org/v1 kind: FirstMate metadata: labels: controller-tools.k8s.io: "1.0" name: firstmate-sample spec: # Add fields here foo: bar
18. Admission webhooks Before the object is persisted to Etcd and an event is created • Mutating • Validating
19. Write your own in 2(?) minutes $ kubebuilder init --domain bonniernews.io --license apache2 --owner ”Bonnier News”’ $ kubebuilder create api --group core --version v1 --kind Events # Implement the controller’s Reconcile function, code, code, code! $ make deploy • Kubebuilder • Kubebuilder is an SDK for rapidly building and publishing Kubernetes APIs in Go
20. Autoscaling
21. Autoscaling machinery • Cluster-Autoscaler • AWS ASG’s and launch configs. • Node provisioning • Spot node controllers (ohoy-controller) • Placement control (scheduler predicates) • Resource calculations
22. Node provisioning • A way to provision nodes • You should probably use GKE, EKS or kops • We use: • Cloud-init • Saltstack
23. Cluster-autoscaler https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler • Adjusts the size of the cluster • Scale up if there are insufficient resources • Scale down if there are underutilized nodes • Simulates the scheduler • Predicates (taints. tolerations, affinity) • Resources • VolumeZonePredicate predicate HPA & VPA friendly!
24. Container resources • Set requests: spec.containers[].resources.requests.cpu spec.containers[].resources.requests.memory
25. AWS ASG (! ) "k8s.io/cluster-autoscaler/node-template/label/node-type.ohoy.io" = "spot" "k8s.io/cluster-autoscaler/node-template/taint/node-type" = "spot:NoSchedule" "k8s.io/cluster-autoscaler/enabled" = "true" "kubernetes.io/cluster/${var.cluster_name}" = "true" • One ASG per AZ and instance type • K8S metadata as AWS tags on ASG’s (The cluster-autoscaler reads them) • Taints and labels
26. Spot node tweaks…
27. Affinity rules (spot nodes) spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: node-type.ohoy.io operator: In values: - spot • Node affinity - attract pods to nodes
28. Taints (spot nodes) taints: - effect: NoSchedule key: node-type value: spot Node taints: • Taint nodes to disable scheduling • NoSchedule • NoExecute
29. Tolerations (spot nodes) tolerations: - effect: NoSchedule key: node-type operator: Equal value: spot Pod tolerates: • Tolerations, override taints
30. ohoy-controller _,-._ ; ___ : ,------------------. ,--' (. .) '--.__ | | _; ||| \ | ARRRRRRRRRRRR, | '._,-----''';=.____," | OHOY MATEY! | /// < o> |##| | | (o \`--' //`-----------------' ///\ >>>> _\ <<<< // --._>>>>>>>><<<<<<<< / ___() >>>[||||]<<<< `--'>>>>>>>><<<<<<< >>>>>>><<<<<< >>>>><<<<< >>ctr<<
31. ohoy-controller reconcile for l, v := range r.config.Spec.SpotNodeLabels { nodeType, ok := node.Labels[l] if ok && nodeType == v { // it's a spotnode if !nodeHasTaint(node, r.config.Spec.SpotNodeTaintKey) { node.Spec.Taints = append(node.Spec.Taints, corev1.Taint{ Key: r.config.Spec.SpotNodeTaintKey, Value: r.config.Spec.SpotNodeTaintValue, Effect: corev1.TaintEffectNoSchedule, }) } err = r.Update(context.TODO(), node) if err != nil { return reconcile.Result{}, err } } }
32. Autoscaling flow Pod Schedulable? No CA Tolerations?
 Affinity rules? ASG New Node Provision Node Node Spot Node? Yes Ohoy Controller
34. Summary
35. Use affinity, taints and tolerations
36. Use resources (be friendly to your scheduler)
37. Write controllers to make your cluster work FOR YOU!
38. Questions?