Intro to Kubernetes operators

Transcript

1. An intro to Kubernetes operators Lili Cosic Software Engineer at Red Hat @LiliCosic
2. Intro to Kubernetes
3. An intro to Kubernetes operators - Lili Cosic Kubernetes What is Kubernetes? • Open source platform for managing containerized workloads and services • Containers, containers, containers • Name originates from Greek - helmsman or pilot • Google open-sourced in 2014 • Based on Borg - Google’ internal project • K_ _ _ _ _ _ _ _S -> k8s
4. An intro to Kubernetes operators - Lili Cosic Kubernetes What is great about Kubernetes? • Scalability of workloads • Separation of workloads • Native stable resources (Deployments, Pods) • Extending the API - Custom Resources
5. An intro to Kubernetes operators - Lili Cosic Kubernetes What is Kubernetes not? • Not a PaaS • Not limited to the types of apps supported • Not opinionated on: ◦ Deploying ◦ Run CI/CD ◦ Logging, monitoring or alerting
6. Pod - smallest deployable unit; a grouping of one or more containers
7. An intro to Kubernetes operators - Lili Cosic Credit: Aqua Security
8. DEMO TIME Simple Kubernetes deployment
9. Kubernetes controllers
10. An intro to Kubernetes operators - Lili Cosic What is a controller? • Deployment has deployment controller • Reconciles given state • Pattern of controllers ◦ Non terminating loop that regulates the state of the system ◦ A control loop watches the shared state of the cluster via the API server and makes changes to move from the current to the desired state
11. Custom Resources
12. An intro to Kubernetes operators - Lili Cosic Custom Resources • CRD - CustomResourceDefinition • Kubernetes serves and handles your custom resource Extends the Kubernetes API
13. An intro to Kubernetes operators - Lili Cosic CRD - custom resource definition yaml manifest apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: crontabs.stable.example.com spec: group: stable.example.com versions: - name: v1 served: true storage: true scope: Namespaced names: plural: crontabs singular: crontab kind: CronTab shortNames: - ct
14. Operators, operators, operators
16. An intro to Kubernetes operators - Lili Cosic Operator is a Kubernetes controller - specific to operating an application
17. An intro to Kubernetes operators - Lili Cosic Every operator is a controller - but not every controller is an operator!
18. An intro to Kubernetes operators - Lili Cosic What is an “operator”? • Non core Kubernetes controller • Makes use of CRDs - custom resource definition • Holds the knowledge of how an application needs to be deployed, managed and packaged • Reconcile loop • Extends the Kubernetes API to make the application a custom resource • Concept introduced by CoreOS
19. An intro to Kubernetes operators - Lili Cosic Good examples of operators Awesome operators! • github.com/coreos/prometheus-operator • github.com/coreos/etcd-operator • github.com/zalando-incubator/postgres-operator • github.com/jaegertracing/jaeger-operator • github.com/operator-framework/awesome-operators
20. An intro to Kubernetes operators - Lili Cosic When to choose creating an operator? • Application uses declarative API • Resources are scoped to a namespace or a cluster • Encapsulate business logic • Build automation that watches for updates of Kubernetes objects • Create or update resources via the Kubernetes native API • Top level support from kubectl
21. An intro to Kubernetes operators - Lili Cosic When to just use a ConfigMap or Secret? Sometimes creating an operator is not needed • Existing well known config file format (e.g. mysql.cnf ) • Config file is used to just configure an application running in a Pod • No need to reconcile to the state
22. An intro to Kubernetes operators - Lili Cosic 101 ways to build an operator • go language: ◦ client-go & co. ◦ operator-sdk ◦ kubebuilder • Other languages: ◦ Kubernetes python and java clients and others • Other tools: ◦ helm ◦ ansible
23. Building operators using Kubernetes native clients
24. An intro to Kubernetes operators - Lili Cosic Kubernetes native clients • k8s.io/client-go ◦ rest ◦ discovery ◦ dynamic • k8s.io/api • ks8.io/apimachinery
25. An intro to Kubernetes operators - Lili Cosic Kubernetes native clients Pros: Cons: • Same as upstream controllers use • Stability of Kubernetes code • Versioning based on Kubernetes releases • Ability to finetune • Large ecosystem • No abstractions/helpers • A lot of inside knowledge to optimize correctly • New major version on every Kubernetes minor version release
26. An intro to Kubernetes operators - Lili Cosic client-go talk client-go: The good, the bad and the ugly
27. An intro to Kubernetes operators - Lili Cosic client-go Credit: github.com/kubernetes/sample-controller
28. An intro to Kubernetes operators - Lili Cosic Informer pattern • ListWatcher ◦ Helps you filter resources you want to watch • ResourceEventHandler ◦ Add, Update and Delete event trigger functions
29. An intro to Kubernetes operators - Lili Cosic Workqueue • Resource Event Handler puts an item to the workqueue • Workqueue consists of keys: ◦ <resource_namespace>/<resource_name> • Only one worker works on one item at a time • First In First Out
30. An intro to Kubernetes operators - Lili Cosic Example of an “operator” - Kubernetes native client github.com/kubernetes/sample-controller
31. kubebuilder
32. An intro to Kubernetes operators - Lili Cosic kubebuilder • github.com/kubernetes-sigs/kubebuilder • SDK for building Kubernetes APIs using CRDs • Part of kubernetes-sigs organization • Uses controller-runtime under the hood • Doesn’t strictly advertise itself as operator builder tool • Creates go language controllers/operators
33. An intro to Kubernetes operators - Lili Cosic Using kubebuilder Example of a go operator $ mkdir kubebuilder-operator && cd kubebuilder-operator $ kubebuilder init --domain k8s.io --license apache2 --owner "The JOnTheBeach Audience" $ kubebuilder create api --group ships --version v1beta1 --kind Sloop $ # Edit the logic code $ pkg/controller/sloop/sloop_controller.go
34. operator-sdk
35. An intro to Kubernetes operators - Lili Cosic operator-sdk • github.com/operator-framework/operator-sdk • Aimed at creating operators • Part of operator-framework • Operator types you can create: ◦ go ◦ helm ◦ ansible • Testing framework • Metrics • Building and running commands
36. An intro to Kubernetes operators - Lili Cosic Using operator-sdk Example of a go operator $ operator-sdk new app-operator $ cd app-operator $ # Add a new API for the custom resource AppService $ operator-sdk add api --api-version=app.example.com/v1alpha1 --kind=AppService $ # Add a new controller that watches for AppService $ operator-sdk add controller --api-version=app.example.com/v1alpha1 --kind=AppService $ # Build operator $ operator-sdk build quay.io/example/app-operator
37. operator-sdk helm
38. An intro to Kubernetes operators - Lili Cosic Using operator-sdk Example of a helm operator $ # Create a new helm operator project $ operator-sdk new nginx-operator --api-version=example.com/v1alpha1 --kind=Nginx --type=helm $ # Edit watches.yaml file to customize the operator logic
39. operator-sdk ansible
40. An intro to Kubernetes operators - Lili Cosic Using operator-sdk Example of an ansible operator $ # Create a new ansible based operator project: $ operator-sdk new memcached-operator --api-version=cache.example.com/v1alpha1 --kind=Memcached --type=ansible $ # Edit watches file to customize the logic
41. An intro to Kubernetes operators - Lili Cosic Operator framework interactive playground learn.openshift.com/operatorframework Guided tutorial on creating operators using operator-sdk
42. Deploying an operator
43. An intro to Kubernetes operators - Lili Cosic Deploying (operator) on Kubernetes Ways of deploying • YAML • Kustomize • Jsonnet • OLM • Helm
44. An intro to Kubernetes operators - Lili Cosic YAML # Register the CRD $ kubectl apply -f crd.yaml # Create any needed Role Based Access Control $ kubectl apply -f rbac.yaml # Deploy the operator $ kubectl apply -f deploy.yaml # Create an instance $ kubectl apply -f cr.yaml Create operator using yaml and kubectl CLI tool
45. An intro to Kubernetes operators - Lili Cosic YAML - CRD manifest file apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: memcacheds.cache.example.com spec: group: cache.example.com names: kind: Memcached listKind: MemcachedList plural: memcacheds singular: memcached scope: Namespaced subresources: status: {} validation: openAPIV3Schema: properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds' type: string metadata: type: object spec: properties: size: format: int32 type: integer required: - size type: object status: properties: nodes: items: type: string type: array required: - nodes type: object version: v1alpha1 versions: - name: v1alpha1 served: true storage: true
46. An intro to Kubernetes operators - Lili Cosic YAML - deployment manifest file apiVersion: apps/v1 kind: Deployment metadata: name: memcached-operator spec: replicas: 1 selector: matchLabels: name: memcached-operator template: metadata: labels: name: memcached-operator spec: serviceAccountName: memcached-operator containers: - name: memcached-operator image: memached:123 command: - memcached-operator imagePullPolicy: Always env: - name: WATCH_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: OPERATOR_NAME value: "memcached-operator"
47. An intro to Kubernetes operators - Lili Cosic YAML - CR manifest file apiVersion: cache.example.com/v1alpha1 kind: Memcached metadata: name: example-memcached spec: size: 3
48. An intro to Kubernetes operators - Lili Cosic OLM - Operator Lifecycle Manager • Upgrades • Descriptions and metadata • Dependency resolution • Multiple versions of operator to install Key features
49. DEMO: creating and deploying an operator
50. Debugging an operator
51. An intro to Kubernetes operators - Lili Cosic Debugging an operator • `operator-sdk up local` - run • Log all the things! • Get logs of an operator ◦ `kubectl logs <pod-name>` ◦ Use CR name to label any pods the operator creates • `kubectl events -n NAMESPACE` • `Spec.Paused` Helpful tools and tips
52. Kubernetes cluster stack for big data
53. An intro to Kubernetes operators - Lili Cosic More information • Kubernetes User Group Big Data • Regular User Group Meeting: Wednesdays at 18:00 UTC (biweekly) • Big Data slack channel - #ug-big-data
54. An intro to Kubernetes operators - Lili Cosic Current status of big data on Kubernetes https://github.com/kubernetes/community/blob/55da27045f90ce36aa1e876af7f9a32 06c228608/ug-big-data/resources.md
55. An intro to Kubernetes operators - Lili Cosic Spark operator Spark on Kubernetes github.com/GoogleCloudPlatform/spark-on-k8s-operator
56. twitter.com/LiliCosic QUESTIONS? github.com/lilic