Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Extending Kubernetes with Operators/Controllers

Extending Kubernetes with Operators/Controllers

Within the Kubernetes community, there has always been a desire to continue to add new features and functionality into the apiserver; you see this every day from the version of each primitive you use. This desire grew so much that the community introduced custom aggregate apiservers these allow you to build custom apis and run them on the master with coordination from the exposed apiserver, but they require you to run your own storage layer or convince your cluster admin to let you use the primary etcd cluster. The community then created ThirdPartyResources; these gave you the ability to create custom resources that were registered with the apiserver and were able to be run what CoreOS coined as the Operator pattern. TPRs, while great, eventually were rebuilt and moved into apiextensions and called CRDs. In this talk, we will build an Operator using CRDs, the k8s code-generation libraries and informer libraries which are generated.

CRDs and Operators are excellent for anything from managing some custom resource like public keys in an SSH authorized_key file or to help you to create easily deployable custom-off-the-shelf software that any cluster administrator or developer could deploy into their environment without having to conform to a predefined spec.

Christopher Hein

April 17, 2019
Tweet

More Decks by Christopher Hein

Other Decks in Programming

Transcript

  1. Extending Kubernetes with CRDs and
    Controllers/Operators
    Chris Hein | Developer Advocate | AWS | CNCF Ambassador
    [email protected]
    @christopherhein
    christopherhein

    View full-size slide

  2. What is Kubernetes?

    View full-size slide

  3. Open source container
    management platform
    Helps you run
    containers at scale
    Gives you primitives
    for building
    modern applications

    View full-size slide

  4. Open source container
    management platform
    Helps you run
    containers at scale
    Gives you primitives
    for building
    modern applications

    View full-size slide

  5. What does all of that mean
    {api}

    View full-size slide

  6. Let’s see it
    $ kubectl get pods -v=7
    GET https://apiserver/api/v1/namespaces/default/pods?limit=500
    Request Headers:
    Accept: application/json;as=Table;v=v1beta1;g=meta.k8s.io,
    application/json
    User-Agent: kubectl/v1.12.1 (linux/amd64) kubernetes/4ed3216
    Response Status: 200 OK in 145 milliseconds
    Response Headers:
    Content-Type: application/json
    Content-Length: 1909
    Date: Wed, 28 Nov 2018 00:23:05 GMT
    Audit-Id: 7b949a88-f3d2-429d-9b19-889c01f2c634
    Response Body: ...

    View full-size slide

  7. API Layer
    {
    "paths": [
    "/api",
    "/api/v1",
    "/apis",
    "/apis/",
    "/apis/admissionregistration.k8s.io",
    "/apis/admissionregistration.k8s.io/v1beta1",
    ...
    ]
    }

    View full-size slide

  8. Kubernetes objects
    • Record of intent
    • Persistent entities
    • Marshal the desired state of your cluster
    • Examples: Pods, Services, Ingress,
    NetworkPolicies, ConfigMaps, Secrets, etc.

    View full-size slide

  9. How do they work?

    View full-size slide

  10. Control Loop
    state
    feedback
    reconcile

    View full-size slide

  11. What do we get out of this complexity
    Immutable Declarative Self-healing
    {kind}

    View full-size slide

  12. How does it connect?

    View full-size slide

  13. attachdetach, bootstrapsigner, clusterrole-
    aggregation, cronjob, csrapproving,
    csrcleaner, csrsigning, daemonset, deployment,
    disruption, endpoint, garbagecollector, job,
    namespace, horizontalpodautoscaling, nodeipam,
    nodelifecycle, persistentvolume-binder,
    persistentvolume-expander, podgc, pv-
    protection, pvc-protection, replicaset,
    replicationcontroller, resourcequota, route,
    service, serviceaccount, serviceaccount-token,
    statefulset, tokencleaner, ttl, ttl-after-
    finished bootstrapsigner, tokencleaner

    View full-size slide

  14. Control Loop
    state
    feedback
    reconcile

    View full-size slide

  15. How do you extend it?

    View full-size slide

  16. Extending Kubernetes
    • Built to be highly configurable
    • Uses the control loop pattern
    • Extensions incl. Storage, Device, Network
    plugins, Aggregate apiservers, CRDs and more
    • Awesome Operators list - https://git.io/vh6Qm

    View full-size slide

  17. CustomResourceDefinitions
    (CRDs)

    View full-size slide

  18. CustomResourceDefinitions (CRDs)
    • Dynamically creates RESTful routes in the
    apiserver
    • Defined with OR without a Pod to use them
    • Uses YAML to tell Kubernetes to create them
    • Built-in Mechanisms for
    .Status
    and sub-resources
    • Ability to validate at the apiserver

    View full-size slide

  19. CRD YAML
    Standard K8s
    primitives style
    apiVersion: apiextensions.k8s.io/v1beta1
    kind: CustomResourceDefinition
    metadata:
    name: crontabs.stable.example.com
    spec:
    group: stable.example.com
    version: v1
    scope: Namespaced
    names:
    plural: crontabs
    singular: crontab
    kind: CronTab
    shortNames:
    - ct

    View full-size slide

  20. Defined Custom Resource
    Again…
    Standard K8s
    primitives style
    apiVersion: "stable.example.com/v1"
    kind: CronTab
    metadata:
    name: my-new-cron-object
    spec:
    cronSpec: "* * * * */5"
    image: my-awesome-cron-image

    View full-size slide

  21. What do you do with CRDs?

    View full-size slide

  22. Build Operators

    View full-size slide

  23. “Operators are domain specific”
    - Kris Nova

    View full-size slide

  24. Operators
    • Coined by CoreOS
    • Standalone Pod in a Kubernetes cluster
    • Watches kube-apiserver for updates
    • Can watch any Kubernetes resource
    • Allows you to take action on new or updates to
    existing resources changes

    View full-size slide

  25. Simple use case
    • https://git.io/fjOZ3
    • Uses client-go and exposes a CRD for managing
    authorized_key files on your nodes
    • Deploy the operator + CRDs then deploy an
    AuthorizedKey Resource for each employee
    !!! DEMO USE ONLY !!!

    View full-size slide

  26. Code Generation FTW! !

    View full-size slide

  27. # apis/node/types.go
    // +genclient
    // +genclient:noStatus
    // +genclient:nonNamespaced
    // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
    type AuthorizedKey struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ObjectMeta `json:"metadata"`
    Data AuthorizedKeyData `json:"data"`
    }
    type AuthorizedKeyData struct {
    Key string `json:"key"`
    }

    View full-size slide

  28. # hack/update-codegen.sh
    scriptdir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
    projectdir="$(pwd | sed "s#$GOPATH\/src\/##g")"
    cd ${scriptdir}/vendor/k8s.io/code-generator && ./generate-groups.sh \
    all \
    ${projectdir}/generated \
    ${projectdir}/apis \
    "node.chrishein.com:v1alpha1" \

    View full-size slide

  29. # generated/...
    generated/
    !"" clientset
    # $"" versioned
    # !"" clientset.go
    # !"" doc.go
    # !"" fake
    # # !"" clientset_generated.go
    # # !"" doc.go
    # # $"" register.go
    # !"" scheme
    # # !"" doc.go
    # # $"" register.go
    # $"" typed
    ...

    View full-size slide

  30. # main.go
    cfg, _ := clientcmd.BuildConfigFromFlags(masterURL, kubeconfig)
    nodeClient, _ := clientset.NewForConfig(cfg)
    clientInformerFactory := informers.NewSharedInformerFactory(nodeClient,
    time.Minute*30)
    controller := ctrl.New(nodeClient,
    clientInformerFactory.Node().V1alpha1().AuthorizedKeys())
    clientInformerFactory.Start(stopCh)
    if err = controller.Run(1, stopCh); err != nil {
    klog.Fatalf("error running controller error=%s", err.Error())
    }

    View full-size slide

  31. # controller/controller.go#New
    authKeyInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs
    {
    AddFunc: controller.enqueueAuthKey,
    UpdateFunc: func(old, new interface{}) {
    newKey := new.(*nodev1alpha1.AuthorizedKey)
    oldKey := old.(*nodev1alpha1.AuthorizedKey)
    if newKey.ResourceVersion == oldKey.ResourceVersion {
    return
    }
    controller.enqueueAuthKey(new)
    },
    DeleteFunc: controller.deleteAuthKey,
    })

    View full-size slide

  32. # controller/controller.go#syncHander
    _, name, _ := cache.SplitMetaNamespaceKey(key)
    authKey, _ := c.authKeyLister.Get(name)
    authorizedKeyFile = authorizedkey.File{
    UID: authKey.Name,
    Key: authKey.Data.Key,
    }
    err = authorizedKeyFile.Sync(false)
    return nil

    View full-size slide

  33. Other use cases

    View full-size slide

  34. Demo
    AWS Service Operator
    https://git.io/fhWu3

    View full-size slide

  35. Others…
    databases
    storage solution out of cluster tools
    ?

    View full-size slide

  36. Thanks!
    Chris Hein | Developer Advocate | AWS | CNCF Ambassador
    [email protected]
    @christopherhein
    christopherhein

    View full-size slide