Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Bending Kubernetes to Your Needs

Bending Kubernetes to Your Needs


Michael Hausenblas

July 30, 2018


  1. Bending Kubernetes to Your Needs
 An overview of customization options

    in Kubernetes Michael Hausenblas @mhausenblas
 Developer Advocate, Red Hat
 2018-07-30, Cloud Matters, London
  2. Hit me up on Twitter: @mhausenblas 2 • Developer Advocate

    @ Red Hat (Go, Kubernetes, OpenShift) • Developer Advocate @ Mesosphere (Mesos, DC/OS, Kubernetes) • Chief Data Engineer @ MapR (HDFS, HBase, Drill, etc.) • Applied research (4y in Ireland, 7y in Austria) • Nowadays mainly developing tools in Go (Python, Node, Java, C++) • Kinda developer turned ops (aka appops) $ whois mhausenblas
  3. Hit me up on Twitter: @mhausenblas 3 admin SRE developer

    infosec architect PM PHB
  4. A little analogy that might help …

  5. Hit me up on Twitter: @mhausenblas 5 kernel distribution

  6. A quick Kubernetes 101

  7. Hit me up on Twitter: @mhausenblas 7 Moving parts—physical view

  8. Hit me up on Twitter: @mhausenblas 8 Moving parts—logical view

  9. How can I customize Kubernetes?

  10. Hit me up on Twitter: @mhausenblas 10 • in-tree (upstream)

    via SIG or direct PR • maintain your own fork • built-in customization approaches Customization options in principle
  11. Hit me up on Twitter: @mhausenblas 11 • configuration files

    and flags (kubelet, kube-apiserver, etc.) • extension points • cloud providers • kubelet (plugins for network/devices/storage and container runtimes) • kubectl plugins • access extensions in the API server • custom resources/controllers • extension API servers • scheduler extensions Customization approaches I I A A A I I A I infrastructure API
  12. Hit me up on Twitter: @mhausenblas 12 Extension patterns example:

    manage a CRD example: authn/authz example: network, storage, kubectl
  13. Hit me up on Twitter: @mhausenblas 13 Cloud providers github.com/kubernetes

    • libraries (in-tree)/controller manager • interfaces for things like: • load balancers • network routes • nodes/VMs I
  14. Hit me up on Twitter: @mhausenblas 14 kubelet: network/device/storage plugins

    • Network—standard: CNI
 https://mhausenblas.info/cn-ref/ • Devices—GPUs, FPGAs, etc.
 https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/ • Storage—20+ in-tree, up-and-coming standard: CSI
 https://kubernetes.io/blog/2018/04/10/container-storage-interface-beta/ I
  15. Hit me up on Twitter: @mhausenblas 15 kubelet: container runtimes

    • Container runtime—standard: CRI (since 1.5)
 https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/ • Nowadays multiple options: • containerd • Kata containers • gVisor • hyper.sh http://cri-o.io I
  16. Hit me up on Twitter: @mhausenblas 16 kubectl plugins •

    Extend the set of commands
 https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/ • Write in any programming language (note: these are binary extensions) • Examples: context control, service catalog, user verification I ~/.kube/plugins
  17. Hit me up on Twitter: @mhausenblas

  18. Hit me up on Twitter: @mhausenblas 18 Interlude 1: the

    control plane
  19. Hit me up on Twitter: @mhausenblas 19 Interlude 2: the

    life of an API server request Above is based on Extensible Admission is Beta and Kubernetes deep dive: API Server – part 1 persisting to etcd API HTTP handler authn & authz mutating admission object schema validation validating admission mutating webhooks validating webhooks
  20. Hit me up on Twitter: @mhausenblas 20 Interlude 3: core

    resources (in-tree) A https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/
  21. Hit me up on Twitter: @mhausenblas

  22. Hit me up on Twitter: @mhausenblas 22 Access extensions in

    the API server • Admission controllers (in-tree, via configuration of the API server)
 https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/ • Dynamic Admission Control
 https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/ • Admission Webhooks (beta) • Initializers (alpha)
  23. Hit me up on Twitter: @mhausenblas 23 Custom resources •

    Extend “known” resources beyond core resources (pods, services, etc.)
 https://blog.openshift.com/kubernetes-deep-dive-api-server-part-3a/ • Use API server to store custom resources in etcd for you • Use CLI to interact with custom resources in the usual way:
 kubectl get | create | delete … A
  24. Hit me up on Twitter: @mhausenblas 24 Custom resource—example A

  25. Hit me up on Twitter: @mhausenblas 25 Custom controller •

    Beyond the controller manager (which is in-tree!) • Custom controller • look after core resources
 https://github.com/kelseyhightower/secrets-controller • look after custom resources
 https://github.com/kubernetes/sample-controller A
  26. Hit me up on Twitter: @mhausenblas 26 Operators • operator

    =~ custom resource + controller
 https://coreos.com/blog/introducing-operator-framework • Motivation: application lifecycle management • Use one of over 30 available operators or write your own using the framework
 https://github.com/operator-framework/awesome-operators A github.com/operator-framework
  27. Hit me up on Twitter: @mhausenblas 27 Extension API servers

    • Full control but a lot of effort and responsibility
 https://kubernetes.io/docs/tasks/access-kubernetes-api/setup-extension-api-server/ • Typically more LOC than an controller/operator • Have to manage storage in etcd yourself • And beyond: the Open Service Broker API and the service catalog
 https://www.openservicebrokerapi.org/ A
  28. Hit me up on Twitter: @mhausenblas 28 Scheduler extensions •

    A scheduler selects a node to run your pods on, based on resource requirements, QoS, affinity, etc.
 https://jvns.ca/blog/2017/07/27/how-does-the-kubernetes-scheduler-work/ • You modify policies or run multiple schedulers (with pod opt-in)
 https://embano1.github.io/post/sched-reconcile/ • You can use a webhook 
 https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/scheduler_extender.md I
  29. Hit me up on Twitter: @mhausenblas 29 Other stuff you

    can customize • Monitoring, alerting, logging • Secret management (encryption at rest) • Ingress
 https://kubernetes.io/docs/concepts/services-networking/ingress/ • DNS
 https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/ • kube-proxy
  30. Resources

  31. Hit me up on Twitter: @mhausenblas 31 • https://kubernetes.io/docs/concepts/extend-kubernetes/extend-cluster/ •

    https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/ • https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/ • https://kubernetes.io/docs/tasks/access-kubernetes-api/setup-extension-api-server/ • https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/ • https://kubernetes.io/docs/reference/access-authn-authz/webhook/ • https://kubernetes.io/docs/setup/scratch/#cloud-provider • https://kubernetes.io/blog/2018/01/extensible-admission-is-beta/ Kubernetes docs and blog posts
  32. Hit me up on Twitter: @mhausenblas 32 • Tim Hockin

    & Michael Rubin—Kubernetes Distributions and ‘Kernels'
 https://www.youtube.com/watch?v=fXBjA2hH-CQ • Stefan Schimanski: • Kubernetes as a API driven platform, Reykjavík Kubernetes Meetup
 https://www.youtube.com/watch?v=BiE7oKeEzDU • SIG API Machinery Deep Dive
 https://www.youtube.com/watch?v=XsFH7OEIIvI • James Munnelly—Extending the Kubernetes API: What the Docs Don't Tell You
  33. Hit me up on Twitter: @mhausenblas 33 • Tim Hockin—Kubernetes

 https://speakerdeck.com/thockin/kubernetes-extensibility • Jonathan Berkhahn & Carolyn Van Slyck—Kubectl Plugins 101
 https://kccnceu18.sched.com/event/DqwJ/kubectl-plugins-101-jonathan-berkhahn-ibm-carolyn-van- slyck-microsoft-intermediate-skill-level-slides-attached • Adrien Trouillaud—Kubernetes Custom Resource, Controller & Operator Development Tools
 https://admiralty.io/kubernetes-custom-resource-controller-and-operator-development-tools.html • Toader Sebastian—A complete guide to Kubernetes Operator SDK
 https://banzaicloud.com/blog/operator-sdk/ • Rob Szumski—Building an Kubernetes Operator for Prometheus and Thanos
 https://robszumski.com/building-an-operator/ Articles and slide decks
  34. Hit me up on Twitter: @mhausenblas 34 • https://github.com/kubernetes/kubectl/tree/master/pkg/pluginutils •

    https://github.com/carolynvs/kubectl-flags-plugin • https://github.com/jordanwilson230/kubectl-plugins • https://github.com/kelseyhightower/denyenv-validating-admission-webhook • https://github.com/kubernetes-sigs/controller-tools • https://github.com/kubernetes-sigs/kubebuilder • https://metacontroller.app/ • https://github.com/yaronha/kube-crd • https://github.com/operator-framework/awesome-operators • https://github.com/operator-framework Repos and tools
  35. plus.google.com/+RedHat linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHatNews learn.openshift.com