Extending Kubernetes with
Custom Resources and
Operator Frameworks
Slide 2
Slide 2 text
Ian Lewis
● @IanMLewis
●
● Tokyo, Japan
● #kubernetes, #go, #python
Slide 3
Slide 3 text
The Problem
Slide 4
Slide 4 text
Problems
● Dynamic, self-healing environment
● Kubernetes provides building blocks, not complete solutions
● New API and constructs
Slide 5
Slide 5 text
Memcached
Slide 6
Slide 6 text
Problems w/ Deploying Memcached
● Memcached needs client side load balancing
● Needs some kind of service discovery
● Don't want to update application code
● Want to support replication + sharding topologies
Slide 7
Slide 7 text
Deploy memcached
● Deploy a proxy using a
Deployment
● Configure proxy using a
ConfigMap
● When backends change
create new ConfigMap and
trigger a rolling-update for
the proxy
memcached
memcached
memcached
client proxy
Google Cloud Platform
controller-manager
Pod
metadata.name: nginx-xxxx-xxxx
spec.nodeName:
ReplicaSet
metadata.name: nginx-xxxx
Deployment
metadata.name: nginx
Deployment
Controller
ReplicaSet
Controller
Scheduler
Slide 35
Slide 35 text
Google Cloud Platform
controller-manager
Pod
metadata.name: nginx-xxxx-xxxx
spec.nodeName:
ReplicaSet
metadata.name: nginx-xxxx
Deployment
metadata.name: nginx
Deployment
Controller
ReplicaSet
Controller
Scheduler
Slide 36
Slide 36 text
Google Cloud Platform
controller-manager
Pod
metadata.name: nginx-xxxx-xxxx
spec.nodeName: node1
ReplicaSet
metadata.name: nginx-xxxx
Deployment
metadata.name: nginx
Deployment
Controller
ReplicaSet
Controller
Scheduler
Slide 37
Slide 37 text
Google Cloud Platform
Pod
metadata.name: nginx-xxxx-xxxx
spec.nodeName: node1
status: Pending
node1
kubelet docker
Slide 38
Slide 38 text
Google Cloud Platform
Pod
metadata.name: nginx-xxxx-xxxx
spec.nodeName: node1
status: ContainerCreating
node1
kubelet docker
Slide 39
Slide 39 text
Google Cloud Platform
Pod
metadata.name: nginx-xxxx-xxxx
spec.nodeName: node1
status: ContainerCreating
node1
kubelet docker
Slide 40
Slide 40 text
Google Cloud Platform
Pod
metadata.name: nginx-xxxx-xxxx
spec.nodeName: node1
status: ContainerCreating
node1
kubelet docker
Docker Hub /
GCR
Slide 41
Slide 41 text
Google Cloud Platform
Pod
metadata.name: nginx-xxxx-xxxx
spec.nodeName: node1
status: ContainerCreating
node1
kubelet docker
nginx-xxxx-x
xxx
Slide 42
Slide 42 text
Google Cloud Platform
Pod
metadata.name: nginx-xxxx-xxxx
spec.nodeName: node1
status: ContainerCreating
node1
kubelet docker
nginx-xxxx-x
xxx
Slide 43
Slide 43 text
Google Cloud Platform
Pod
metadata.name: nginx-xxxx-xxxx
spec.nodeName: node1
status: Running
node1
kubelet docker
nginx-xxxx-x
xxx
Slide 44
Slide 44 text
Extending Kubernetes
Slide 45
Slide 45 text
Extending Kubernetes
● Need a place to store state - Data
● Need to do something - Logic
Slide 46
Slide 46 text
Custom Resource Definition (Data)
● Type definition for a custom
type
● Allows the same CRUD +
WATCH
● Can describe higher level
constructs
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: foo.example.com
spec:
group: example.com
version: v1
names:
kind: Foo
plural: foos
scope: Namespaced
Slide 47
Slide 47 text
Controllers (Logic)
● Typically runs in the cluster
● Uses the Kubernetes API
● One idiomatic client
○ client-go
● Many generated client libraries
○ Go
○ Python
○ Java
Slide 48
Slide 48 text
client-go
● Most featureful
● Used by Kubernetes built-in
components
● More like a controller
framework than a client library
Slide 49
Slide 49 text
Operator Frameworks
Slide 50
Slide 50 text
Operator Frameworks
● Provide a simplified controller
API
● Rely on code generation to
provide API clients for CRDs
Slide 51
Slide 51 text
Frequent Requests
● Controllers can potentially run
often
● Easy to overload the API server
● Some GETS could be X00MB of
data
Object
Slide 52
Slide 52 text
Concurrent Updates
● Overwriting object state
● The API Server isn't a database
● No transactions
Object
Slide 53
Slide 53 text
operator-sdk
● Built by former CoreOS
developers at
Red Hat
● Quick and dirty
● Provides support for one
controller per process
● Caching of watched objects w/
client-go
● Serial updates per CRD object
Slide 54
Slide 54 text
Architecture Patterns
Slide 55
Slide 55 text
Reuse Built-in Objects
● Services, Deployments,
ConfigMaps, Secrets
● Built on the logic of other
controllers
● Architect based on what you
would do manually
API
Slide 56
Slide 56 text
Use Multiple Controllers
● Multiple controllers per
process
● Reuse caches, informers etc.
● Keep controllers simple
● Each controller
manages/writes to one object
type
● All message passing is done
through the API
API
Slide 57
Slide 57 text
kubebuilder
● Built by Kubernetes developers at Google
● More robust
● Helps manage lifecycle of generated code
● Supports controllers for built-in objects
● Supports multiple controllers in single operator binary