$30 off During Our Annual Pro Sale. View Details »

Bending Kubernetes to Your Needs

Bending Kubernetes to Your Needs

Michael Hausenblas

July 30, 2018
Tweet

More Decks by Michael Hausenblas

Other Decks in Technology

Transcript

  1. Bending Kubernetes to Your Needs

    An overview of customization options in Kubernetes
    Michael Hausenblas @mhausenblas

    Developer Advocate, Red Hat

    2018-07-30, Cloud Matters, London

    View Slide

  2. Hit me up on Twitter: @mhausenblas
    2
    • Developer Advocate @ Red Hat (Go, Kubernetes, OpenShift)
    • Developer Advocate @ Mesosphere (Mesos, DC/OS, Kubernetes)
    • Chief Data Engineer @ MapR (HDFS, HBase, Drill, etc.)
    • Applied research (4y in Ireland, 7y in Austria)
    • Nowadays mainly developing tools in Go (Python, Node, Java, C++)
    • Kinda developer turned ops (aka appops)
    $ whois mhausenblas

    View Slide

  3. Hit me up on Twitter: @mhausenblas
    3
    admin
    SRE
    developer
    infosec
    architect
    PM
    PHB

    View Slide

  4. A little analogy that might help …

    View Slide

  5. Hit me up on Twitter: @mhausenblas
    5
    kernel distribution

    View Slide

  6. A quick Kubernetes 101

    View Slide

  7. Hit me up on Twitter: @mhausenblas
    7
    Moving parts—physical view

    View Slide

  8. Hit me up on Twitter: @mhausenblas
    8
    Moving parts—logical view

    View Slide

  9. How can I customize Kubernetes?

    View Slide

  10. Hit me up on Twitter: @mhausenblas
    10
    • in-tree (upstream) via SIG or direct PR
    • maintain your own fork
    • built-in customization approaches
    Customization options in principle

    View Slide

  11. Hit me up on Twitter: @mhausenblas
    11
    • configuration files and flags (kubelet, kube-apiserver, etc.)
    • extension points
    • cloud providers
    • kubelet (plugins for network/devices/storage and container runtimes)
    • kubectl plugins
    • access extensions in the API server
    • custom resources/controllers
    • extension API servers
    • scheduler extensions
    Customization approaches
    I
    I
    A
    A
    A
    I
    I
    A
    I
    infrastructure
    API

    View Slide

  12. Hit me up on Twitter: @mhausenblas
    12
    Extension patterns
    example: manage a CRD
    example: authn/authz
    example: network, storage, kubectl

    View Slide

  13. Hit me up on Twitter: @mhausenblas
    13
    Cloud providers
    github.com/kubernetes
    • libraries (in-tree)/controller manager
    • interfaces for things like:
    • load balancers
    • network routes
    • nodes/VMs
    I

    View Slide

  14. Hit me up on Twitter: @mhausenblas
    14
    kubelet: network/device/storage plugins
    • Network—standard: CNI

    https://github.com/containernetworking/cni 

    https://mhausenblas.info/cn-ref/
    • Devices—GPUs, FPGAs, etc.

    https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/
    • Storage—20+ in-tree, up-and-coming standard: CSI

    https://kubernetes.io/docs/concepts/storage/volumes/#types-of-volumes 

    https://kubernetes.io/blog/2018/04/10/container-storage-interface-beta/
    I

    View Slide

  15. Hit me up on Twitter: @mhausenblas
    15
    kubelet: container runtimes
    • Container runtime—standard: CRI (since 1.5)

    https://kubernetes.io/blog/2016/12/container-runtime-interface-cri-in-kubernetes/
    • Nowadays multiple options:
    • containerd
    • Kata containers
    • gVisor
    • hyper.sh
    http://cri-o.io
    I

    View Slide

  16. Hit me up on Twitter: @mhausenblas
    16
    kubectl plugins
    • Extend the set of commands

    https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/
    • Write in any programming language (note: these are binary extensions)
    • Examples: context control, service catalog, user verification
    I
    ~/.kube/plugins

    View Slide

  17. Hit me up on Twitter: @mhausenblas

    View Slide

  18. Hit me up on Twitter: @mhausenblas
    18
    Interlude 1: the control plane

    View Slide

  19. Hit me up on Twitter: @mhausenblas
    19
    Interlude 2: the life of an API server request
    Above is based on Extensible Admission is Beta and Kubernetes deep dive: API Server – part 1
    persisting to
    etcd
    API HTTP
    handler
    authn
    &
    authz
    mutating
    admission
    object schema
    validation
    validating
    admission
    mutating
    webhooks
    validating
    webhooks

    View Slide

  20. Hit me up on Twitter: @mhausenblas
    20
    Interlude 3: core resources (in-tree)
    A
    https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/

    View Slide

  21. Hit me up on Twitter: @mhausenblas

    View Slide

  22. Hit me up on Twitter: @mhausenblas
    22
    Access extensions in the API server
    • Admission controllers (in-tree, via configuration of the API server)

    https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/
    • Dynamic Admission Control

    https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/
    • Admission Webhooks (beta)
    • Initializers (alpha)

    A

    View Slide

  23. Hit me up on Twitter: @mhausenblas
    23
    Custom resources
    • Extend “known” resources beyond core resources (pods, services, etc.)

    https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/

    https://blog.openshift.com/kubernetes-deep-dive-api-server-part-3a/
    • Use API server to store custom resources in etcd for you
    • Use CLI to interact with custom resources in the usual way:

    kubectl get | create | delete …
    A

    View Slide

  24. Hit me up on Twitter: @mhausenblas
    24
    Custom resource—example
    A

    View Slide

  25. Hit me up on Twitter: @mhausenblas
    25
    Custom controller
    • Beyond the controller manager (which is in-tree!)
    • Custom controller
    • look after core resources

    https://github.com/kelseyhightower/secrets-controller
    • look after custom resources

    https://github.com/kubernetes/sample-controller
    A

    View Slide

  26. Hit me up on Twitter: @mhausenblas
    26
    Operators
    • operator =~ custom resource + controller

    https://coreos.com/blog/introducing-operator-framework
    • Motivation: application lifecycle management
    • Use one of over 30 available operators or write your
    own using the framework

    https://github.com/operator-framework/awesome-operators
    A
    github.com/operator-framework

    View Slide

  27. Hit me up on Twitter: @mhausenblas
    27
    Extension API servers
    • Full control but a lot of effort and responsibility

    https://kubernetes.io/docs/tasks/access-kubernetes-api/setup-extension-api-server/
    • Typically more LOC than an controller/operator
    • Have to manage storage in etcd yourself
    • And beyond: the Open Service Broker API and the service catalog

    https://kubernetes.io/docs/concepts/extend-kubernetes/service-catalog/

    https://www.openservicebrokerapi.org/
    A

    View Slide

  28. Hit me up on Twitter: @mhausenblas
    28
    Scheduler extensions
    • A scheduler selects a node to run your pods on, based on resource
    requirements, QoS, affinity, etc.

    https://jvns.ca/blog/2017/07/27/how-does-the-kubernetes-scheduler-work/
    • You modify policies or run multiple schedulers (with pod opt-in)

    https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/

    https://embano1.github.io/post/sched-reconcile/
    • You can use a webhook 

    https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/scheduler_extender.md
    I

    View Slide

  29. Hit me up on Twitter: @mhausenblas
    29
    Other stuff you can customize
    • Monitoring, alerting, logging
    • Secret management (encryption at rest)
    • Ingress

    https://kubernetes.io/docs/concepts/services-networking/ingress/
    • DNS

    https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/
    • kube-proxy

    https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive/

    View Slide

  30. Resources

    View Slide

  31. Hit me up on Twitter: @mhausenblas
    31
    • https://kubernetes.io/docs/concepts/extend-kubernetes/extend-cluster/
    • https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
    • https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/
    • https://kubernetes.io/docs/tasks/access-kubernetes-api/setup-extension-api-server/
    • https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/
    • https://kubernetes.io/docs/reference/access-authn-authz/webhook/
    • https://kubernetes.io/docs/setup/scratch/#cloud-provider
    • https://kubernetes.io/blog/2018/01/extensible-admission-is-beta/
    Kubernetes docs and blog posts

    View Slide

  32. Hit me up on Twitter: @mhausenblas
    32
    • Tim Hockin & Michael Rubin—Kubernetes Distributions and ‘Kernels'

    https://www.youtube.com/watch?v=fXBjA2hH-CQ
    • Stefan Schimanski:
    • Kubernetes as a API driven platform, Reykjavík Kubernetes Meetup

    https://www.youtube.com/watch?v=BiE7oKeEzDU
    • SIG API Machinery Deep Dive

    https://www.youtube.com/watch?v=XsFH7OEIIvI
    • James Munnelly—Extending the Kubernetes API: What the Docs Don't Tell You

    https://www.youtube.com/watch?v=PYLFZVv68lM 

    Videos

    View Slide

  33. Hit me up on Twitter: @mhausenblas
    33
    • Tim Hockin—Kubernetes Extensibility

    https://speakerdeck.com/thockin/kubernetes-extensibility
    • Jonathan Berkhahn & Carolyn Van Slyck—Kubectl Plugins 101

    https://kccnceu18.sched.com/event/DqwJ/kubectl-plugins-101-jonathan-berkhahn-ibm-carolyn-van-
    slyck-microsoft-intermediate-skill-level-slides-attached
    • Adrien Trouillaud—Kubernetes Custom Resource, Controller & Operator Development Tools

    https://admiralty.io/kubernetes-custom-resource-controller-and-operator-development-tools.html
    • Toader Sebastian—A complete guide to Kubernetes Operator SDK

    https://banzaicloud.com/blog/operator-sdk/
    • Rob Szumski—Building an Kubernetes Operator for Prometheus and Thanos

    https://robszumski.com/building-an-operator/
    Articles and slide decks

    View Slide

  34. Hit me up on Twitter: @mhausenblas
    34
    • https://github.com/kubernetes/kubectl/tree/master/pkg/pluginutils
    • https://github.com/carolynvs/kubectl-flags-plugin
    • https://github.com/jordanwilson230/kubectl-plugins
    • https://github.com/kelseyhightower/denyenv-validating-admission-webhook
    • https://github.com/kubernetes-sigs/controller-tools
    • https://github.com/kubernetes-sigs/kubebuilder
    • https://metacontroller.app/
    • https://github.com/yaronha/kube-crd
    • https://github.com/operator-framework/awesome-operators
    • https://github.com/operator-framework
    Repos and tools

    View Slide

  35. plus.google.com/+RedHat
    linkedin.com/company/red-hat
    youtube.com/user/RedHatVideos
    facebook.com/redhatinc
    twitter.com/RedHatNews
    learn.openshift.com

    View Slide