Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Bending Kubernetes to Your Needs

Bending Kubernetes to Your Needs

Michael Hausenblas

July 30, 2018

More Decks by Michael Hausenblas

Other Decks in Technology


  1. Bending Kubernetes to Your Needs
 An overview of customization options

    in Kubernetes Michael Hausenblas @mhausenblas
 Developer Advocate, Red Hat
 2018-07-30, Cloud Matters, London
  2. Hit me up on Twitter: @mhausenblas 2 • Developer Advocate

    @ Red Hat (Go, Kubernetes, OpenShift) • Developer Advocate @ Mesosphere (Mesos, DC/OS, Kubernetes) • Chief Data Engineer @ MapR (HDFS, HBase, Drill, etc.) • Applied research (4y in Ireland, 7y in Austria) • Nowadays mainly developing tools in Go (Python, Node, Java, C++) • Kinda developer turned ops (aka appops) $ whois mhausenblas
  3. Hit me up on Twitter: @mhausenblas 10 • in-tree (upstream)

    via SIG or direct PR • maintain your own fork • built-in customization approaches Customization options in principle
  4. Hit me up on Twitter: @mhausenblas 11 • configuration files

    and flags (kubelet, kube-apiserver, etc.) • extension points • cloud providers • kubelet (plugins for network/devices/storage and container runtimes) • kubectl plugins • access extensions in the API server • custom resources/controllers • extension API servers • scheduler extensions Customization approaches I I A A A I I A I infrastructure API
  5. Hit me up on Twitter: @mhausenblas 12 Extension patterns example:

    manage a CRD example: authn/authz example: network, storage, kubectl
  6. Hit me up on Twitter: @mhausenblas 13 Cloud providers github.com/kubernetes

    • libraries (in-tree)/controller manager • interfaces for things like: • load balancers • network routes • nodes/VMs I
  7. Hit me up on Twitter: @mhausenblas 14 kubelet: network/device/storage plugins

    • Network—standard: CNI
 https://mhausenblas.info/cn-ref/ • Devices—GPUs, FPGAs, etc.
 https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/ • Storage—20+ in-tree, up-and-coming standard: CSI
 https://kubernetes.io/blog/2018/04/10/container-storage-interface-beta/ I
  8. Hit me up on Twitter: @mhausenblas 15 kubelet: container runtimes

    • Container runtime—standard: CRI (since 1.5)
 https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/ • Nowadays multiple options: • containerd • Kata containers • gVisor • hyper.sh http://cri-o.io I
  9. Hit me up on Twitter: @mhausenblas 16 kubectl plugins •

    Extend the set of commands
 https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/ • Write in any programming language (note: these are binary extensions) • Examples: context control, service catalog, user verification I ~/.kube/plugins
  10. Hit me up on Twitter: @mhausenblas 19 Interlude 2: the

    life of an API server request Above is based on Extensible Admission is Beta and Kubernetes deep dive: API Server – part 1 persisting to etcd API HTTP handler authn & authz mutating admission object schema validation validating admission mutating webhooks validating webhooks
  11. Hit me up on Twitter: @mhausenblas 20 Interlude 3: core

    resources (in-tree) A https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/
  12. Hit me up on Twitter: @mhausenblas 22 Access extensions in

    the API server • Admission controllers (in-tree, via configuration of the API server)
 https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/ • Dynamic Admission Control
 https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/ • Admission Webhooks (beta) • Initializers (alpha)
  13. Hit me up on Twitter: @mhausenblas 23 Custom resources •

    Extend “known” resources beyond core resources (pods, services, etc.)
 https://blog.openshift.com/kubernetes-deep-dive-api-server-part-3a/ • Use API server to store custom resources in etcd for you • Use CLI to interact with custom resources in the usual way:
 kubectl get | create | delete … A
  14. Hit me up on Twitter: @mhausenblas 25 Custom controller •

    Beyond the controller manager (which is in-tree!) • Custom controller • look after core resources
 https://github.com/kelseyhightower/secrets-controller • look after custom resources
 https://github.com/kubernetes/sample-controller A
  15. Hit me up on Twitter: @mhausenblas 26 Operators • operator

    =~ custom resource + controller
 https://coreos.com/blog/introducing-operator-framework • Motivation: application lifecycle management • Use one of over 30 available operators or write your own using the framework
 https://github.com/operator-framework/awesome-operators A github.com/operator-framework
  16. Hit me up on Twitter: @mhausenblas 27 Extension API servers

    • Full control but a lot of effort and responsibility
 https://kubernetes.io/docs/tasks/access-kubernetes-api/setup-extension-api-server/ • Typically more LOC than an controller/operator • Have to manage storage in etcd yourself • And beyond: the Open Service Broker API and the service catalog
 https://www.openservicebrokerapi.org/ A
  17. Hit me up on Twitter: @mhausenblas 28 Scheduler extensions •

    A scheduler selects a node to run your pods on, based on resource requirements, QoS, affinity, etc.
 https://jvns.ca/blog/2017/07/27/how-does-the-kubernetes-scheduler-work/ • You modify policies or run multiple schedulers (with pod opt-in)
 https://embano1.github.io/post/sched-reconcile/ • You can use a webhook 
 https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/scheduler_extender.md I
  18. Hit me up on Twitter: @mhausenblas 29 Other stuff you

    can customize • Monitoring, alerting, logging • Secret management (encryption at rest) • Ingress
 https://kubernetes.io/docs/concepts/services-networking/ingress/ • DNS
 https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/ • kube-proxy
  19. Hit me up on Twitter: @mhausenblas 31 • https://kubernetes.io/docs/concepts/extend-kubernetes/extend-cluster/ •

    https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/ • https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/ • https://kubernetes.io/docs/tasks/access-kubernetes-api/setup-extension-api-server/ • https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/ • https://kubernetes.io/docs/reference/access-authn-authz/webhook/ • https://kubernetes.io/docs/setup/scratch/#cloud-provider • https://kubernetes.io/blog/2018/01/extensible-admission-is-beta/ Kubernetes docs and blog posts
  20. Hit me up on Twitter: @mhausenblas 32 • Tim Hockin

    & Michael Rubin—Kubernetes Distributions and ‘Kernels'
 https://www.youtube.com/watch?v=fXBjA2hH-CQ • Stefan Schimanski: • Kubernetes as a API driven platform, Reykjavík Kubernetes Meetup
 https://www.youtube.com/watch?v=BiE7oKeEzDU • SIG API Machinery Deep Dive
 https://www.youtube.com/watch?v=XsFH7OEIIvI • James Munnelly—Extending the Kubernetes API: What the Docs Don't Tell You
  21. Hit me up on Twitter: @mhausenblas 33 • Tim Hockin—Kubernetes

 https://speakerdeck.com/thockin/kubernetes-extensibility • Jonathan Berkhahn & Carolyn Van Slyck—Kubectl Plugins 101
 https://kccnceu18.sched.com/event/DqwJ/kubectl-plugins-101-jonathan-berkhahn-ibm-carolyn-van- slyck-microsoft-intermediate-skill-level-slides-attached • Adrien Trouillaud—Kubernetes Custom Resource, Controller & Operator Development Tools
 https://admiralty.io/kubernetes-custom-resource-controller-and-operator-development-tools.html • Toader Sebastian—A complete guide to Kubernetes Operator SDK
 https://banzaicloud.com/blog/operator-sdk/ • Rob Szumski—Building an Kubernetes Operator for Prometheus and Thanos
 https://robszumski.com/building-an-operator/ Articles and slide decks
  22. Hit me up on Twitter: @mhausenblas 34 • https://github.com/kubernetes/kubectl/tree/master/pkg/pluginutils •

    https://github.com/carolynvs/kubectl-flags-plugin • https://github.com/jordanwilson230/kubectl-plugins • https://github.com/kelseyhightower/denyenv-validating-admission-webhook • https://github.com/kubernetes-sigs/controller-tools • https://github.com/kubernetes-sigs/kubebuilder • https://metacontroller.app/ • https://github.com/yaronha/kube-crd • https://github.com/operator-framework/awesome-operators • https://github.com/operator-framework Repos and tools