Bending Kubernetes to Your Needs

Bending Kubernetes to Your Needs


Michael Hausenblas

July 30, 2018


  1. Bending Kubernetes to Your Needs
 An overview of customization options

    in Kubernetes Michael Hausenblas @mhausenblas
 Developer Advocate, Red Hat
 2018-07-30, Cloud Matters, London
  2. Hit me up on Twitter: @mhausenblas 2 • Developer Advocate

    @ Red Hat (Go, Kubernetes, OpenShift) • Developer Advocate @ Mesosphere (Mesos, DC/OS, Kubernetes) • Chief Data Engineer @ MapR (HDFS, HBase, Drill, etc.) • Applied research (4y in Ireland, 7y in Austria) • Nowadays mainly developing tools in Go (Python, Node, Java, C++) • Kinda developer turned ops (aka appops) $ whois mhausenblas
  3. Hit me up on Twitter: @mhausenblas 3 admin SRE developer

    infosec architect PM PHB
  4. A little analogy that might help …

  5. Hit me up on Twitter: @mhausenblas 5 kernel distribution

  6. A quick Kubernetes 101

  7. Hit me up on Twitter: @mhausenblas 7 Moving parts—physical view

  8. Hit me up on Twitter: @mhausenblas 8 Moving parts—logical view

  9. How can I customize Kubernetes?

  10. Hit me up on Twitter: @mhausenblas 10 • in-tree (upstream)

    via SIG or direct PR • maintain your own fork • built-in customization approaches Customization options in principle
  11. Hit me up on Twitter: @mhausenblas 11 • configuration files

    and flags (kubelet, kube-apiserver, etc.) • extension points • cloud providers • kubelet (plugins for network/devices/storage and container runtimes) • kubectl plugins • access extensions in the API server • custom resources/controllers • extension API servers • scheduler extensions Customization approaches I I A A A I I A I infrastructure API
  12. Hit me up on Twitter: @mhausenblas 12 Extension patterns example:

    manage a CRD example: authn/authz example: network, storage, kubectl
  13. Hit me up on Twitter: @mhausenblas 13 Cloud providers

    • libraries (in-tree)/controller manager • interfaces for things like: • load balancers • network routes • nodes/VMs I
  14. Hit me up on Twitter: @mhausenblas 14 kubelet: network/device/storage plugins

    • Network—standard: CNI • Devices—GPUs, FPGAs, etc. • Storage—20+ in-tree, up-and-coming standard: CSI I
  15. Hit me up on Twitter: @mhausenblas 15 kubelet: container runtimes

    • Container runtime—standard: CRI (since 1.5) • Nowadays multiple options: • containerd • Kata containers • gVisor • I
  16. Hit me up on Twitter: @mhausenblas 16 kubectl plugins •

    Extend the set of commands • Write in any programming language (note: these are binary extensions) • Examples: context control, service catalog, user verification I ~/.kube/plugins
  17. Hit me up on Twitter: @mhausenblas

  18. Hit me up on Twitter: @mhausenblas 18 Interlude 1: the

    control plane
  19. Hit me up on Twitter: @mhausenblas 19 Interlude 2: the

    life of an API server request Above is based on Extensible Admission is Beta and Kubernetes deep dive: API Server – part 1 persisting to etcd API HTTP handler authn & authz mutating admission object schema validation validating admission mutating webhooks validating webhooks
  20. Hit me up on Twitter: @mhausenblas 20 Interlude 3: core

    resources (in-tree) A
  21. Hit me up on Twitter: @mhausenblas

  22. Hit me up on Twitter: @mhausenblas 22 Access extensions in

    the API server • Admission controllers (in-tree, via configuration of the API server) • Dynamic Admission Control • Admission Webhooks (beta) • Initializers (alpha)
  23. Hit me up on Twitter: @mhausenblas 23 Custom resources •

    Extend “known” resources beyond core resources (pods, services, etc.) • Use API server to store custom resources in etcd for you • Use CLI to interact with custom resources in the usual way:
 kubectl get | create | delete … A
  24. Hit me up on Twitter: @mhausenblas 24 Custom resource—example A

  25. Hit me up on Twitter: @mhausenblas 25 Custom controller •

    Beyond the controller manager (which is in-tree!) • Custom controller • look after core resources • look after custom resources A
  26. Hit me up on Twitter: @mhausenblas 26 Operators • operator

    =~ custom resource + controller • Motivation: application lifecycle management • Use one of over 30 available operators or write your own using the framework A
  27. Hit me up on Twitter: @mhausenblas 27 Extension API servers

    • Full control but a lot of effort and responsibility • Typically more LOC than an controller/operator • Have to manage storage in etcd yourself • And beyond: the Open Service Broker API and the service catalog A
  28. Hit me up on Twitter: @mhausenblas 28 Scheduler extensions •

    A scheduler selects a node to run your pods on, based on resource requirements, QoS, affinity, etc. • You modify policies or run multiple schedulers (with pod opt-in) • You can use a webhook I
  29. Hit me up on Twitter: @mhausenblas 29 Other stuff you

    can customize • Monitoring, alerting, logging • Secret management (encryption at rest) • Ingress • DNS • kube-proxy
  30. Resources

  31. Hit me up on Twitter: @mhausenblas 31 • • • • • • • • Kubernetes docs and blog posts
  32. Hit me up on Twitter: @mhausenblas 32 • Tim Hockin

    & Michael Rubin—Kubernetes Distributions and ‘Kernels' • Stefan Schimanski: • Kubernetes as a API driven platform, Reykjavík Kubernetes Meetup • SIG API Machinery Deep Dive • James Munnelly—Extending the Kubernetes API: What the Docs Don't Tell You 
  33. Hit me up on Twitter: @mhausenblas 33 • Tim Hockin—Kubernetes

    Extensibility • Jonathan Berkhahn & Carolyn Van Slyck—Kubectl Plugins 101 slyck-microsoft-intermediate-skill-level-slides-attached • Adrien Trouillaud—Kubernetes Custom Resource, Controller & Operator Development Tools • Toader Sebastian—A complete guide to Kubernetes Operator SDK • Rob Szumski—Building an Kubernetes Operator for Prometheus and Thanos Articles and slide decks
  34. Hit me up on Twitter: @mhausenblas 34 • • • • • • • • • • Repos and tools