Lambda-style Kubernetes orchestration with Google Cloud’s Metacontroller

Slides for my talk at the Kubernetes/Cloud-Native Meetup in Munich.

Transcript

1. PUBLIC Antoine Cotten, SAP January 24, 2018 Lambda-style Kubernetes orchestration with Google Cloud’s Metacontroller.
2. 2 PUBLIC © 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Chen Goldberg GKE engineering team lead Anthony Yeh Software engineer on GKE team
4. 4 PUBLIC © 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Kubernetes controllers overview § Technical concepts and patterns Google Cloud’s Metacontroller § Why Lambdas? § Getting started § Demo: indexed jobs § Code dissection Agenda
5. Kubernetes controllers overview
6. 6 PUBLIC © 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Technical concepts api-server core extensions custom … Informer Controller Reflector EventHandler Indexer Watch()
7. 7 PUBLIC © 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Technical concepts api-server core extensions custom … Informer Controller Reflector EventHandler Indexer List()
8. 8 PUBLIC © 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Technical concepts api-server core extensions custom … Informer Controller Reflector EventHandler Indexer workqueue Workers
9. 9 PUBLIC © 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Technical concepts api-server core extensions custom … Informer Controller Reflector EventHandler Indexer workqueue Informer Workers
10. Google Cloud’s Metacontroller
11. 11 PUBLIC © 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ
12. 12 PUBLIC © 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ “Go is not good enough.” A curated list of articles complaining that Go isn't good enough github.com/ksimka/go-is-not-good
13. 13 PUBLIC © 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Metacontroller
14. 14 PUBLIC © 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ The next big orchestration thing will come from you.
15. 15 PUBLIC © 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Getting started Controller registration CustomResourceDefinition group: metacontroller.k8s.io version: v1alpha1 names: kind: CompositeController plural: compositecontrollers scope: Cluster /apiextensions CustomResourceDefinition group: apps.hybris.com version: v1 names: kind: Ecommerce plural: ecommerces scope: Namespaced
16. 16 PUBLIC © 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Getting started Controller registration api-server metacontroller.k8s.io apps.hybris.com CompositeController Ecommerce apiVersion: metacontroller.k8s.io/v1alpha1 kind: CompositeController metadata: name: ecommerce-ctrl spec: parentResource: apiVersion: apps.hybris.com/v1 resource: ecommerces childResources: - apiVersion: v1 resources: ["pods"] clientConfig: service: name: ecommerce-ctrl namespace: default hooks: sync: path: /sync generateSelector: true MetaController ecommerce-ctrl
17. 17 PUBLIC © 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Getting started Resource registration apiVersion: apps.hybris.com/v1 kind: Ecommerce metadata: name: shop-1 spec: search-backend: solr ingress-class: apache template: spec: containers: - name: hybris image: hybris-commerce:6.6.0.0 command: ["catalina.sh", "run"] http://ecommerce-controller.default/sync ecommerce-controller MetaController api-server metacontroller.k8s.io apps.hybris.com CompositeController Ecommerce shop-1 ecommerce-ctrl
18. 18 PUBLIC © 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Getting started Control loop http://ecommerce-controller.default/sync ecommerce-controller MetaController List() api-server metacontroller.k8s.io apps.hybris.com CompositeController Ecommerce shop-1 ecommerce-ctrl
19. 19 PUBLIC © 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Getting started Control loop http://ecommerce-controller.default/sync ecommerce-controller MetaController { "parent": { "apiVersion": "apps.hybris.com/v1", "kind": "Ecommerce", ... }, "children": { "Pod.v1": {} } } api-server metacontroller.k8s.io apps.hybris.com CompositeController Ecommerce shop-1 ecommerce-ctrl
20. 20 PUBLIC © 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Getting started Control loop http://ecommerce-controller.default/sync ecommerce-controller MetaController { "children": [ { "kind": "Pod", "metadata": { "name": "hybris" }, "spec": { "containers": ... } }, { "kind": "Pod", "metadata": { "name": "solr" }, "spec": { "containers": ... } }, { another pod } ], "status": { "conditions": [{ "type": "Available", "status": "False" }] } } api-server metacontroller.k8s.io apps.hybris.com CompositeController Ecommerce shop-1 ecommerce-ctrl
21. 21 PUBLIC © 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Getting started Control loop http://ecommerce-controller.default/sync ecommerce-controller MetaController Create() Update() Delete() api-server metacontroller.k8s.io apps.hybris.com CompositeController Ecommerce shop-1 ecommerce-ctrl core
22. Demo: indexed jobs
23. 23 PUBLIC © 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Code dissection Main loop func main() { config := rest.InClusterConfig() for { resyncAll(config) time.Sleep(5 * time.Second) } } func resyncAll(config *rest.Config) { // Discover all supported resources. dc := discovery.NewDiscoveryClientForConfig(config) resources := dc.ServerResources() // Generate clientset for all resources (k8s.io/client-go/dynamic) clientset := newDynamicClientset(config, newResourceMap(resources)) // Sync all CompositeController objects. syncAllCompositeControllers(clientset) // Sync all InitializerController objects. syncAllInitializerControllers(clientset) }
24. 24 PUBLIC © 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Code dissection Sync Composite controllers func syncAllCompositeControllers(clientset *dynamicClientset) { // Retrieve all CompositeController resources using generated client. ccClient := clientset.Resource(v1alpha1.SchemeGroupVersion.String(), "compositecontrollers", "") obj := ccClient.List(metav1.ListOptions{}) // Assert runtime.Object (k8s.io/apimachinery/pkg/.../unstructured) ccList := obj.(*unstructured.UnstructuredList) for i := range ccList.Items { // Serialize/deserialize object to/from Json (hack) data := json.Marshal(&ccList.Items[i]) cc := &v1alpha1.CompositeController{} json.Unmarshal(data, cc) // Sync resources managed by this controller syncCompositeController(clientset, cc) } }
25. 25 PUBLIC © 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Code dissection Sync parent resource func syncCompositeController(clientset *dynamicClientset, cc *v1alpha1.CompositeController) { // Sync all objects of the parent type, in all namespaces. parentClient := clientset.Resource(cc.Spec.ParentResource.APIVersion, cc.Spec.ParentResource.Resource, "") obj := parentClient.List(metav1.ListOptions{}) parentList := obj.(*unstructured.UnstructuredList) for i := range parentList.Items { parent := &list.Items[i] syncParentResource(clientset, cc, parentClient.APIResource(), parent) } }
26. 26 PUBLIC © 2017 SAP SE or an SAP affiliate company. All rights reserved. ǀ Code dissection Sync child resources func syncParentResource(clientset *dynamicClientset, cc *v1alpha1.CompositeController, parentResource *APIResource, parent *unstructured.Unstructured) { // Claim all matching child resources, including orphan/adopt as necessary. labelSelector = metav1.AddLabelToSelector(labelSelector, "controller-uid", string(parent.GetUID())) selector := metav1.LabelSelectorAsSelector(labelSelector) children := claimChildren(clientset, cc, parentResource, parent, selector) // Call the sync hook for this parent. syncRequest := &syncHookRequest{ Parent: parent, Children: children, } syncResult := callSyncHook(cc, syncRequest) // Reconcile children. if parent.GetDeletionTimestamp() == nil { manageChildren(clientset, cc, parent, children, makeChildMap(syncResult.Children)) } // Update parent status. updateParentStatus(clientset, cc, parentResource, parent, syncResult.Status) }
27. Thank you. Contact information: antoine.cotten@sap.com @AntoineCotten