Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Rage Against the API-Machinery: Writing an Operator for Production

Kasten
November 14, 2018

Rage Against the API-Machinery: Writing an Operator for Production

An operator is a set of CustomResourceDefinitions (CRDs) that extends the Kubernetes API and a controller that handles the new API objects. Not only have the number of projects following the operator pattern exploded, but so have the number of ways to bootstrap an operator. The operator pattern is the basis for Kubernetes’ extensibility, but it is difficult to achieve the same robustness as in-tree APIs/controllers.

In this talk, the speakers will present what it takes to write a production-ready Operator based on their experience developing and running Kanister in production. They will compare popular operator kits, SDKs, and guides, presenting their trade-offs. Best practices for building, testing, and API versioning will also be covered. After the talk, the audience will feel comfortable developing a production-ready operator. Familiarity with CRDs is a suggested prerequisite.

Kasten

November 14, 2018
Tweet

More Decks by Kasten

Other Decks in Technology

Transcript

  1. Ilya Kislenko
    @depohmel [email protected]
    Tom Manville
    @tdmanv [email protected]

    View full-size slide

  2. ● Kanister: Open-source Operator
    ○ Framework for application-level data management
    ○ Example Apps: MySQL, MongoDB, PostgreSQL, ElasticSearch
    ○ Generic Infra Support: Volumes, ObjectStore
    ● K10’s API
    ○ 2 CRD-controllers
    ○ 1 Aggregated API server
    你好

    View full-size slide

  3. You’ve decided an Operator solves your problems
    ● Centralize/automate operational experience
    ● Coordinate app operations w/ Kubernetes
    What now?
    ● Bootstrapping is simple
    ● Production is hard

    View full-size slide

  4. Does your Operator feel like a native
    Kubernetes API?

    View full-size slide

  5. ● Extending the Kubernetes API
    ● Production Requirements
    ● Papercuts

    View full-size slide

  6. ● Extending the Kubernetes API
    ○ Operator Definition
    ○ Custom Resources
    ○ API Conventions
    ○ Bootstrapping Projects
    ● Production Requirements
    ● Papercuts

    View full-size slide

  7. CustomResource
    Domain

    View full-size slide

  8. CustomResource Controller
    Domain

    View full-size slide

  9. CustomResource Controller
    Domain
    Operator

    View full-size slide

  10. Human ops knowledge → Software
    Observe
    Analyze
    Act
    ● Support complex Operations
    ● Active reconciliation
    ● Extensible

    View full-size slide

  11. CustomResource (CR)
    ● Instance of CRD
    ● Also Kubernetes API Object
    ● Scope: Cluster or Namespace
    ● Arbitrary Schema
    ● Out-of-tree API
    CustomResourceDefinition (CRD)
    ● Defines new API
    ● Kubernetes API Object
    ● Scope: Cluster
    ● Predefined Schema
    ● In-tree API
    ● Adds a new endpoint to the API
    server

    View full-size slide

  12. CustomResource (CR)
    apiVersion:
    apiextensions.k8s.io/v1beta1
    kind: CustomResourceDefinition
    metadata:
    name:
    crontabs.stable.example.com
    spec:
    group: stable.example.com
    versions:
    - name: v1
    storage: true
    scope: Cluster
    names:
    plural: crontabs
    kind: CronTab
    shortNames:
    - ct
    apiVersion:
    "stable.example.com/v1"
    kind: CronTab
    metadata:
    name: my-new-cron-object
    spec:
    cronSpec: "* * * * */5"
    image: my-awesome-cron-image
    CustomResourceDefinition (CRD)

    View full-size slide

  13. https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md
    Follow best practices for:
    ● Restful
    ● Naming/Namespacing
    ● ObjectMeta
    ○ Labels
    ○ Versioning
    ● Declarative vs. Imperative
    ● Spec vs. Status

    View full-size slide

  14. ● Rook
    https://github.com/rook/operator-kit
    ● Giant Swarm
    https://github.com/giantswarm/operatorkit

    View full-size slide

  15. ● Operator SDK
    https://github.com/operator-framework/operator-sdk
    ● Kubebuilder
    https://github.com/kubernetes-sigs/kubebuilder
    ● Metacontroller
    https://github.com/GoogleCloudPlatform/metacontroller

    View full-size slide

  16. ● Extending the Kubernetes API
    ● Production-Ready Features
    ○ Clients
    ○ RBAC
    ○ Safety
    ○ Events
    ○ Testing
    ● Papercuts

    View full-size slide

  17. kubectl Go SDK
    cli := kubernetes.NewClient()
    ns := &v1.Namespace{
    ObjectMeta: metav1.ObjectMeta{
    Name: "widget-namespace",
    },
    }
    err := cli.CoreV1().Namespaces().Create(ns)
    $ cat <apiVersion: v1
    kind: Namespace
    metadata:
    name: widget-namespace
    EOF

    View full-size slide

  18. kind: ClusterRole
    apiVersion:
    rbac.authorization.k8s.io/v1
    metadata:
    name: widget-operator
    rules:
    - apiGroups:
    - widgets.company.io
    resources:
    - "*"
    verbs:
    - "*"
    ● RBAC is a double-edged sword
    ● Users != ServiceAccounts
    ● CRD != CR
    ● Follow principle of least privilege

    View full-size slide

  19. ● Changes are usually what breaks production
    ● Operators can handle complex state transitions
    ● You should trust your Operator as you trust your
    DevOps
    ● Suggestion: Use one of Workload types

    View full-size slide

  20. https://github.com/kubernetes/code-generator
    ● deepcopy-gen
    ● client-gen
    ● Informer-gen
    ● lister-gen
    // +genclient
    // +genclient:noStatus
    // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

    View full-size slide

  21. $ kubectl describe actionset backup-84w5z

    Events:
    Type Reason Age From Message
    ---- ------ ---- ---- -------
    Normal Started Action 42s Kanister Controller Executing action backup
    Normal Started Phase 42s Kanister Controller Executing phase dumpToObjectStore
    Normal Ended Phase 41s Kanister Controller Completed phase dumpToObjectStore
    Normal Update Complete 41s Kanister Controller Updated ActionSet 'backup-84w5z'
    $ kubectl get events
    LAST SEEN FIRST SEEN COUNT NAME KIND
    54m 54m 1 backup-84w5z.1565461009ce6235 ActionSet ...
    54m 54m 1 backup-84w5z.156546100ac72a84 ActionSet ...
    54m 54m 1 backup-84w5z.15654610618cb419 ActionSet ...
    54m 54m 1 backup-84w5z.15654610620c7952 ActionSet ...

    View full-size slide

  22. // Initialize Event Recorder
    broadcaster := record.NewBroadcaster()
    broadcaster.StartEventWatcher(
    func(event *core.Event) {
    _, err := client.Core().Events(event.Namespace).Create(event)
    },
    )
    source := core.EventSource{Component: "Widget Controller"}
    recorder := broadcaster.NewRecorder(scheme.Scheme, source)
    // Record Event
    recorder.Event(obj, corev1.EventTypeNormal, "Started", "Started work on Widget!")

    View full-size slide

  23. stdlib
    $ go test .
    import "testing"
    func TestFail(t *testing.T) {
    t.Fail()
    }
    import "github.com/onsi/ginkgo"
    var _ = ginkgo. Describe("Some behavior" ,
    func() {
    ginkgo.Context("My test", func() {
    ginkgo. It("may fail", func() {
    ginkgo. Fail("This test failed" )
    })
    })
    })
    ginkgo

    View full-size slide

  24. In-Cluster
    Out-of-Cluster
    config, err := rest.InClusterConfig()
    config, err := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(
    clientcmd.NewDefaultClientConfigLoadingRules(),
    &clientcmd.ConfigOverrides{},
    ).ClientConfig()

    View full-size slide

  25. // Always return a new Widget with the request name.
    reaction := func(action testing.Action) (bool, runtime.Object, error) {
    get, _ := action.(testing.GetAction)
    ret := &v1.Widget{
    ObjectMeta: metav1.ObjectMeta{
    Name: get.GetName(),
    },
    }
    return true, w, nil
    }
    // Create fake Clientset
    cli := fake.NewSimpleClientset()
    cli.PrependReactor("get", "widgets", reaction)

    View full-size slide

  26. ● Extending the Kubernetes API
    ● Production-Ready Features
    ● Papercuts
    ○ Validation
    ○ CRD Lifecycle
    ○ Object Versioning
    ○ Code Generation

    View full-size slide

  27. ● You are responsible for validating content
    ● OpenAPI v3.0 validation
    ○ Basic schema validation
    ● Validating Admission Webhook
    Admission webhooks are essentially part of the cluster control-plane.
    You should write and deploy them with great caution.

    View full-size slide

  28. Create CRD in the controller
    ● Self-contained dependency
    ● Controller requires additional permissions
    Create CRD during deployment
    ● Custom logic is difficult
    ● Controller will fail w/o CRD

    View full-size slide

  29. Kubernetes v1.11.0
    ● Multiple versions supported
    ● “Nop” Conversion
    Future Improvements
    ● Mutating webhooks for conversion
    ● Separate validation for different versions

    View full-size slide

  30. ● Follow API conventions
    ● Bootstrap with a go-based project
    ● Generate your code
    ● Setup RBAC for your CRs
    ● Create Kubernetes Events
    ● Invest in tests
    ● Beware of gaps between CRDs and core APIs

    View full-size slide

  31. www.kanister.io
    https://github.com/kanisterio/kanister
    https://twitter.com/tdmanv
    https://twitter.com/depohmel
    https://kasten.io

    View full-size slide

  32. Go Gophers
    ● https://github.com/ashleymcnamara/gophers/blob/master/LICENSE
    Ginkgo
    ● https://github.com/onsi/ginkgo/blob/master/LICENSE
    Koala art:
    ● https://www.instagram.com/anaitsart/

    View full-size slide