Slide 1

Slide 1 text

Extending Kubernetes The superpower behind the Kubernetes API Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 2

Slide 2 text

apiVersion: batch/v1 kind: Job metadata: name: hello spec: template: # This is the pod template spec: containers: - name: hello image: busybox command: ['sh', '-c', 'echo "Hello, Kubernetes!" && sleep 3600'] restartPolicy: OnFailure # The pod template ends here Example coming from kubernetes.io Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 3

Slide 3 text

Kubernetes is a declarative framework If you use it just as end tool you are missing its real value

Slide 4

Slide 4 text

Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 5

Slide 5 text

k-proxy kubelet sched sched sched Control Plane Node etcd Kubernetes cluster api api api c-c-m c-c-m c-c-m c-m c-m c-m Node Node k-proxy kubelet kubelet k-proxy Control plane Scheduler sched Cloud controller manager (optional) c-c-m Controller manager c-m kubelet kubelet kube-proxy k-proxy (persistence store) etcd etcd Node API server api from kubernetes.io Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 6

Slide 6 text

from SlideShare: Moby CRI Containerd Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 7

Slide 7 text

How you justify Kubernetes Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 8

Slide 8 text

Play your own game. That's why services have their own API. Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 9

Slide 9 text

You know yourself, your team and your product. Build around your requirement Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 10

Slide 10 text

Who am I? Software Engineer at Equinix Metal (Packet) Open Source maintainer for Kubernetes, Docker, TestContainer Docker Captain and CNCF Ambassador When not coding I grow vegetables I am active on Twitter as @gianarb

Slide 11

Slide 11 text

Kubernetes follows the same rule, if kubectl is not enough because you have an half way workflow that you like, or the kubectl makes your solution hard to maintain, it is OK to build something by yourself. Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 12

Slide 12 text

There different ways to extend Kubernetes, here a couple: . A kubectl plugin. . Via Client-GO or any other SDK. . Custom Resource Definition. . Aggregation layer Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 13

Slide 13 text

Let's start with kubectl . It is by far the most flexible and easy way to extend Kubernetes. You have to deliver an executable in your $PATH that starts with kubectl-* . Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 14

Slide 14 text

Examples: A binary called kubectl-ns can be executed as kubectl ns A binary called kubectl-profefe can be executed as kubectl profefe Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 15

Slide 15 text

kubectl-profefe Profefe is an open source project to do continuous profiling of application using pprof, such as Golang. cron job application runtime/pprof net/http/pprof pull Storage developer profefe-collector query push GET /debug/pprof/proļ¬le Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 16

Slide 16 text

I wrote a project called profefe/kube-profefe that acts as a bridge between profefe and Kubernetes. . It serves a binary called kprofefe , it can run as Kubernetes cronjob and it collects profiles targeting application running in Kubernetes. . kubectl-profefe helps you to interact with profefe and Kubernetes. Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 17

Slide 17 text

$ kubectl profefe --help It is a kubectl plugin that you can use to retrieve and manage profiles in Go. Available Commands: capture Capture gathers profiles for a pod or a set of them. If can filter by namespace and via label selector. get Display one or many resources help Help about any command load Load a profile you have locally to profefe Flags: -A, --all-namespaces If present, list the requested object(s) across all namespaces --as string Username to impersonate for the operation --as-group stringArray Group to impersonate for the operation, this flag can be repeated to specify multiple groups. --cluster string The name of the kubeconfig cluster to use --context string The name of the kubeconfig context to use -f, --filename strings identifying the resource. -h, --help help for kubectl-profefe --insecure-skip-tls-verify If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure --kubeconfig string Path to the kubeconfig file to use for CLI requests. -n, --namespace string If present, the namespace scope for this CLI request -l, --selector string Selector (label query) to filter on, supports '=', '==', and '!='.(e.g. -l key1=value1,key2=value2) Use "kubectl-profefe [command] --help" for more information about a command. Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 18

Slide 18 text

Links about kubectl plugins: "My experience with Krew to manage kubectl plugins" "kubectl flags in your plugin" Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 19

Slide 19 text

Client GO and other SDKs Kubernetes API works as any other API. There are client libraries that you can use in many languages or it works via HTTP, and all the languages have an HTTP client available. Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 20

Slide 20 text

Not all the languages are the same There are official supported an unofficial libraries available out there. Go, Javascript, Haskell, Python, Java, Dotnet are supported by the Kubernetes community, you can check out the actual documentation. All of them are in different state, Golang is well done. Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 21

Slide 21 text

kprofefe the binary I told you about before uses the client-go library to do a couple of things: Retrieve pods filtered according to label section or/and per namespace in order to get the right targets for profefe It uses the annotations for a particular pod in order to figure out where the pprof server runs (the right port and path) Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 22

Slide 22 text

Get pods // GetSelectedPods returns all the pods with the profefe annotation enabled // filtered by the selected labels func GetSelectedPods(clientset kubernetes.Interface, namespace string, listOpt metav1.ListOptions) ([]v1.Pod, error) { target := []v1.Pod{} pods, err := clientset.CoreV1().Pods(namespace).List(listOpt) if err != nil { return target, err } for _, pod := range pods.Items { enabled, ok := pod.Annotations[ProfefeEnabledAnnotation] if ok && enabled == "true" && pod.Status.Phase == v1.PodRunning { target = append(target, pod) } } return target, nil } Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 23

Slide 23 text

Port forward programmatically func PortForwardAPod(req PortForwardAPodRequest) error { path := fmt.Sprintf("/api/v1/namespaces/%s/pods/%s/portforward", req.Pod.Namespace, req.Pod.Name) hostIP := strings.TrimLeft(req.RestConfig.Host, "htps:/") transport, upgrader, err := spdy.RoundTripperFor(req.RestConfig) if err != nil { return err } dialer := spdy.NewDialer( upgrader, &http.Client{Transport: transport}, http.MethodPost, &url.URL{Scheme: "https", Path: path, Host: hostIP}) fw, err := portforward.New( dialer, []string{fmt.Sprintf("%d:%d", req.LocalPort, req.PodPort)}, req.StopCh, req.ReadyCh, req.Streams.Out, req.Streams.ErrOut) if err != nil { return err } return fw.ForwardPorts() } Repository: gianarb/kube-port-forward

Slide 24

Slide 24 text

Custom Resource Definition (CRD) Terraform has the concept of modules. A Terraform module is a bridge between an external resource and Terraform. A CRD is the same, but with Kubernetes. Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 25

Slide 25 text

Custom Resource Definition (CRD) Native resources: Pod, Services, Deployment, StatefulSet, Ingress... Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 26

Slide 26 text

AWS Controllers for Kubernetes: Elastic Load Balancer, S3 bucket, CloudFormation Stack. Checkout: ACK Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

client api-se kube crd-cont Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 29

Slide 29 text

Why you should write a CRD? Your team uses Kubernetes a lot, they know how to use the kubectl and client-go. You can make available for them external services like DNS management or a binary/image repository with an user experience they already know. Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 30

Slide 30 text

Kubernetes gives you a bunch of useful features that you can leverage: . Authentication . Authorization . Audit Logs . Event System . CLI (kubectl) and client libraries Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 31

Slide 31 text

Gianluca Arbezzano - gianarb.it - twitter.com/gianarb

Slide 32

Slide 32 text

Thank you Any question? Reach out to me via Twitter @gianarb. Gianluca Arbezzano - gianarb.it - twitter.com/gianarb