Kubernetes Advanced Resource Features - Episode 2

- Services
　- Defining a service
　- Kube-proxy modes
　- Type of Services
- Ingress
- Run Application
　- Stateless Applications
　- Stateful Applications
　- Horizontal Pod Autoscaler
- Jobs
- Daemon sets
- Configmaps

1. Kubernetes Advanced Resource Features - Episode 2 Date: 2018/12/07 Place: ITRI Presenter: Samina (Shan-Jung Fu) Prepare Hands-On Environment: http://bit.ly/2zLHggi All Lab base on K8s@1.11
2. Outline • Services • Ingress • Run Application • Jobs • Daemon sets • Configmaps 2
3. • Services ◦ Defining a service ◦ Kube-proxy modes ◦ Type of Services • Ingress • Run Application • Jobs • Daemon sets • Configmaps Outline 3
4. • To group a set of Pod endpoints into a single resource • An abstraction which defines a logical set of Pods & a policy • “layer 4” (TCP/UDP over IP) construct Defining a service • Defining a service • Kube-proxy modes • Type of Services 4
5. Defining a service • Defining a service • Kube-proxy modes • Type of Services A group of pods that work together • grouped by a selector Defines access policy • “load balanced” or “headless” Can have a stable virtual IP and port • also a DNS name VIP is managed by kube-proxy • watches all services • updates iptables(ipvs table) when backends change 5
6. kube-proxy is responsible for implementing a form of virtual IP for Services • Proxy-mode ◦ userspace ◦ iptables ◦ ipvs Virtual IPs and service proxies • Defining a service • Kube-proxy modes • Type of Services 6
7. kube-proxy is responsible for implementing a form of virtual IP for Services • Proxy-mode ◦ userspace ◦ iptables ◦ ipvs Virtual IPs and service proxies • Defining a service • Kube-proxy modes • Type of Services 7
8. iptables kube-proxy From: https://courses.cs.washington.edu/courses/cse550/16au/notes/kubernetes.pdf 8
9. From: https://courses.cs.washington.edu/courses/cse550/16au/notes/kubernetes.pdf 9
10. From: https://courses.cs.washington.edu/courses/cse550/16au/notes/kubernetes.pdf 10
11. From: https://courses.cs.washington.edu/courses/cse550/16au/notes/kubernetes.pdf 11
12. From: https://courses.cs.washington.edu/courses/cse550/16au/notes/kubernetes.pdf 12
13. From: https://courses.cs.washington.edu/courses/cse550/16au/notes/kubernetes.pdf 13
14. From: https://courses.cs.washington.edu/courses/cse550/16au/notes/kubernetes.pdf 14
15. From: https://courses.cs.washington.edu/courses/cse550/16au/notes/kubernetes.pdf 15
16. From: https://courses.cs.washington.edu/courses/cse550/16au/notes/kubernetes.pdf 16
17. From: https://courses.cs.washington.edu/courses/cse550/16au/notes/kubernetes.pdf 17
18. From: https://courses.cs.washington.edu/courses/cse550/16au/notes/kubernetes.pdf 18
19. From: https://courses.cs.washington.edu/courses/cse550/16au/notes/kubernetes.pdf 19
20. From: https://courses.cs.washington.edu/courses/cse550/16au/notes/kubernetes.pdf 20
21. From: https://courses.cs.washington.edu/courses/cse550/16au/notes/kubernetes.pdf 21
22. kube-proxy is responsible for implementing a form of virtual IP for Services • Proxy-mode ◦ userspace ◦ iptables ◦ ipvs Virtual IPs and service proxies • Defining a service • Kube-proxy modes • Type of Services 22
23. Type of Services • ClusterIP (default) • NodePort • LoadBalancer • ExternalName • External IPs • Defining a service • Kube-proxy modes • Type of Services 23
24. • ClusterIP (default) Internal clients send requests to a stable internal IP address. Note: The member Pod must have a container that is listening on TCP port 8080. Else, clients will see a message like "Failed to connect" or "This site can't be reached". • NodePort • LoadBalancer • ExternalName • External IPs Type of Services (Cont.) • Defining a service • Kube-proxy modes • Type of Services apiVersion: v1 kind: Service metadata: name: my-cip-service labels: app: my-nginx Spec: type: ClusterIP ports: - port: 80 protocol: TCP selector: app: my-nginx 24
25. kubectl create deployment --image nginx my-nginx kubectl create -f service-cip.yaml kubectl get all -o wide curl service_ip kubectl delete service/my-cip-service kubectl delete deployment my-nginx • ClusterIP (default) Type of Services (Cont.) • Defining a service • Kube-proxy modes • Type of Services $ $ $ $ $ $ 25
26. • ClusterIP (default) • NodePort Clients send requests to the IP address of a node on one or more nodePort values that are specified by the Service. (default: 30000-32767) • LoadBalancer • ExternalName • External IPs Type of Services (Cont.) • Defining a service • Kube-proxy modes • Type of Services apiVersion: v1 kind: Service metadata: name: my-np-service labels: app: my-nginx spec: type: NodePort ports: - name: http nodePort: 32660 port: 80 targetPort: 80 protocol: TCP selector: app: my-nginx 26
27. kubectl create deployment --image nginx my-nginx kubectl create -f service-np.yaml kubectl get all -o wide curl service_ip curl 172.17.8.100:32660 kubectl delete service/my-np-service kubectl delete deployment my-nginx • NodePort Type of Services (Cont.) • Defining a service • Kube-proxy modes • Type of Services $ $ $ $ $ $ $ 27
28. apiVersion: v1 kind: Service metadata: name: my-np-service labels: app: my-nginx spec: type: LoadBalancer ports: - name: http port: 80 targetPort: 80 protocol: TCP loadBalancerIP: external_IP selector: app: my-nginx • ClusterIP (default) • NodePort • LoadBalancer Clients send requests to the IP address of a External network load balancer. • ExternalName • External IPs Type of Services (Cont.) • Defining a service • Kube-proxy modes • Type of Services 28
29. kind: Service apiVersion: v1 metadata: name: my-service namespace: prod spec: type: ExternalName externalName: my.database.example.com • ClusterIP (default) • NodePort • LoadBalancer • ExternalName Internal clients use the DNS name of a Service as an alias for an external DNS name. • External IPs Type of Services (Cont.) • Defining a service • Kube-proxy modes • Type of Services 29
30. kind: Service apiVersion: v1 metadata: name: my-service spec: selector: app: MyApp ports: - name: http protocol: TCP port: 80 targetPort: 9376 externalIPs: - 80.11.12.10 • ClusterIP (default) • NodePort • LoadBalancer • ExternalName • External IPs If there are external IPs that route to one or more cluster nodes, Kubernetes services can be exposed on those externalIPs Type of Services (Cont.) • Defining a service • Kube-proxy modes • Type of Services 30
31. • Services • Ingress • Run Application • Jobs • Daemon sets • Configmaps Outline 31
32. Ingress Many apps are HTTP/HTTPS Services are L4 (IP + port) Ingress maps incoming traffic to backend services • by HTTP host headers • by HTTP URL paths 32
33. 1. Install ingress controller (eg. ingress-nginx) 2. Create pod & service 3. Create Ingress resource Ingress # If you have external IPs, please do not exec the following three cmd sudo ifconfig enp0s8:0 172.17.8.101 netmask 255.255.255.0 broadcast 172.17.8.255 sudo ifconfig enp0s8:1 172.17.8.102 netmask 255.255.255.0 broadcast 172.17.8.255 sudo ifconfig enp0s8:2 172.17.8.103 netmask 255.255.255.0 broadcast 172.17.8.255 $ helm search nginx-ingress $ helm install --name nginx-ingress --set "rbac.create=true,controller.service.externalIPs[0]=172.17.8.101,controller.s ervice.externalIPs[1]=172.17.8.102,controller.service.externalIPs[2]=172.17.8 .103” stable/nginx-ingress 33
34. Ingress 1. Install ingress controller (eg. ingress-nginx) 2. Create pod & service 3. Create Ingress resource $ cd ~/hands-on-w-tutorials/2018-12-06_07/ingress $ kubectl create -f apple.yaml $ kubectl create -f banana.yaml 34
35. Ingress 1. Install ingress controller (eg. ingress-nginx) 2. Create pod & service 3. Create Ingress apiVersion: extensions/v1beta1 kind: Ingress metadata: name: example-ingress annotations: ingress.kubernetes.io/rewrite-target: / spec: rules: - http: paths: - path: /apple backend: serviceName: apple-service servicePort: 5678 - path: /banana backend: serviceName: banana-service servicePort: 5678 $ cd ~/hands-on-w-tutorials/2018-12-06_07/i ngress $ kubectl create -f ing.yaml $ curl -kL http://172.17.8.102/apple $ curl -kL http://172.17.8.102/banana 35
36. • Services • Ingress • Run Application ◦ Stateless Applications ◦ Stateful Applications ◦ Horizontal Pod Autoscaler • Jobs • Daemon sets • Configmaps Outline 36
37. Stateless Applications Deployments(Deploys) represent a set of multiple, identical Pods with no unique identities. • Stateless Applications • Stateful Applications • Horizontal Pod Autoscaler 37
38. Stateless Applications apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx replicas: 2 # tells deployment to run 2 pods matching the template template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80 • Stateless Applications • Stateful Applications • Horizontal Pod Autoscaler 38
39. $ $ $ $ $ Update & Scale Deploy • Stateless Applications • Stateful Applications • Horizontal Pod Autoscaler kubectl create -f https://k8s.io/examples/application/deployment.yaml kubectl describe deployment nginx-deployment kubectl get pods -l app=nginx kubectl apply -f https://k8s.io/examples/application/deployment-scale.yaml kubectl describe deployment nginx-deployment | tail -n 13 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ScalingReplicaSet 4m10s deployment-controller Scaled up replica set nginx-deployment-67594d6bf6 to 2 Normal ScalingReplicaSet 116s deployment-controller Scaled up replica set nginx-deployment-67594d6bf6 to 4 Normal ScalingReplicaSet 116s deployment-controller Scaled up replica set nginx-deployment-7fc9b7bd96 to 1 Normal ScalingReplicaSet 116s deployment-controller Scaled down replica set nginx-deployment-67594d6bf6 to 3 Normal ScalingReplicaSet 116s deployment-controller Scaled up replica set nginx-deployment-7fc9b7bd96 to 2 Normal ScalingReplicaSet 102s deployment-controller Scaled down replica set nginx-deployment-67594d6bf6 to 2 Normal ScalingReplicaSet 102s deployment-controller Scaled up replica set nginx-deployment-7fc9b7bd96 to 3 Normal ScalingReplicaSet 101s deployment-controller Scaled down replica set nginx-deployment-67594d6bf6 to 1 Normal ScalingReplicaSet 101s deployment-controller Scaled up replica set nginx-deployment-7fc9b7bd96 to 4 Normal ScalingReplicaSet 98s deployment-controller (combined from similar events): Scaled down replica set nginx-deployment-67594d6bf6 to 0 39
40. Stateful Applications • Single-Instance Stateful Application ◦ PersistentVolume + Deployment • Replicated Stateful Application ◦ StatefulSet is the workload API object used to manage stateful applications. • Stateless Applications • Stateful Applications • Horizontal Pod Autoscaler 40
41. StatefulSet • Manages Pods that are based on an identical container spec • Each pod has a persistent identifier • Support ◦ Stable, unique network identifiers. ◦ Stable, persistent storage. ◦ Ordered, graceful deployment and scaling. ◦ Ordered, automated rolling updates. • Stateless Applications • Stateful Applications • Horizontal Pod Autoscaler 41
42. • Deploy web app Steps 1. StorageClass 2. PersistentVolume 3. StatefulSet 4. Service StatefulSet • Stateless Applications • Stateful Applications • Horizontal Pod Autoscaler kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: stsweb-storage-class annotations: storageclass.beta.kubernetes.io/is-default-class: "true" provisioner: kubernetes.io/host-path 42
43. • Deploy web app Steps 1. StorageClass 2. PersistentVolume 3. StatefulSet 4. Service StatefulSet • Stateless Applications • Stateful Applications • Horizontal Pod Autoscaler kind: PersistentVolume apiVersion: v1 metadata: name: stsweb-pv-volume labels: type: local spec: storageClassName: stsweb-storage-class capacity: storage: 10Gi accessModes: - ReadWriteOnce hostPath: path: "/tmp/stsdata" 43
44. apiVersion: apps/v1 kind: StatefulSet metadata: name: stsweb spec: selector: matchLabels: app: stsweb # has to match .spec.template.metadata.labels serviceName: "stsweb" replicas: 1 # by default is 1 template: metadata: labels: app: stsweb # has to match .spec.selector.matchLabels spec: terminationGracePeriodSeconds: 10 containers: - name: stsweb image: k8s.gcr.io/nginx-slim:0.8 ports: - containerPort: 80 name: stsweb volumeMounts: - name: www • Deploy web app Steps 1. StorageClass 2. PersistentVolume 3. StatefulSet 4. Service StatefulSet • Stateless Applications • Stateful Applications • Horizontal Pod Autoscaler volumeClaimTemplates: - metadata: name: www spec: accessModes: [ "ReadWriteOnce" ] storageClassName: "stsweb-storage-class" resources: requests: storage: 1Gi 44
45. • Deploy web app Steps 1. StorageClass 2. PersistentVolume 3. StatefulSet 4. Service StatefulSet • Stateless Applications • Stateful Applications • Horizontal Pod Autoscaler apiVersion: v1 kind: Service metadata: name: stsweb labels: app: stsweb spec: ports: - port: 80 name: http clusterIP: None selector: app: stsweb 45
46. • Stateless Applications • Stateful Applications HorizontalPodAutoscaler Horizontal Pod Autoscaler Automatically scale pods in a replication controller, deployment or replica set • based on CPU utilization (for now) • custom metrics in Alpha DaemonSets can’t 46
47. • Stateless Applications • Stateful Applications HorizontalPodAutoscaler Horizontal Pod Autoscaler 1. Config kube-controller-manager when install k8s cluster 2. Deploy Metrics-Server 3. Create deployment 4. Create Horizontal Pod Autoscaler • Test increase load 47
48. 1. Config kube-controller-manager when install k8s cluster 2. Deploy Metrics-Server 3. Create Horizontal Pod Autoscaler Test • Increase load • Stop load 4. • Stateless Applications • Stateful Applications HorizontalPodAutoscaler Horizontal Pod Autoscaler --horizontal-pod-autoscaler-cpu-initialization-period duration Default: 5m0s The period after pod start when CPU samples might be skipped. --horizontal-pod-autoscaler-downscale-stabilization duration Default: 5m0s The period for which autoscaler will look backwards and not scale down below any recommendation it made during that period. --horizontal-pod-autoscaler-initial-readiness-delay duration Default: 30s The period after pod start during which readiness changes will be treated as initial readiness. --horizontal-pod-autoscaler-sync-period duration Default: 15s The period for syncing the number of pods in horizontal pod autoscaler. --horizontal-pod-autoscaler-tolerance float Default: 0.1 The minimum change (from 1.0) in the desired-to-actual metrics ratio for the horizontal pod autoscaler to consider scaling. 48
49. • Stateless Applications • Stateful Applications HorizontalPodAutoscaler Horizontal Pod Autoscaler 1. Config kube-controller-manager when install k8s cluster 2. Deploy Metrics-Server (a cluster-wide aggregator of resource usage data) 3. Create deployment 4. Create Horizontal Pod Autoscaler • Test increase load $ cd ~/hands-on-w-tutorials/2018-12-06_07 && kubectl create -f ./metrics-server/ $ kubectl api-versions | grep autoscaling $ kubectl top node $ kubectl get --raw "/apis/metrics.k8s.io/v1beta1" | jq 49
50. $ kubectl run php-apache --image=gcr.io/google_containers /hpa-example --requests=cpu=100m --expose --port=80 1. Config kube-controller-manager when install k8s cluster 2. Deploy Metrics-Server 3. Create deployment 4. Create Horizontal Pod Autoscaler • Test increase load • Stateless Applications • Stateful Applications HorizontalPodAutoscaler Horizontal Pod Autoscaler 50
51. 1. Config kube-controller-manager when install k8s cluster 2. Deploy Metrics-Server 3. Create deployment 4. Create Horizontal Pod Autoscaler • Test increase load • Stateless Applications • Stateful Applications HorizontalPodAutoscaler Horizontal Pod Autoscaler apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: name: php-apache namespace: default spec: scaleTargetRef: apiVersion: apps/v1beta1 kind: Deployment name: php-apache minReplicas: 1 maxReplicas: 10 targetCPUUtilizationPercentage: 50 $ kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10 51
52. • Stateless Applications • Stateful Applications HorizontalPodAutoscaler Horizontal Pod Autoscaler 1. Config kube-controller-manager when install k8s cluster 2. Deploy Metrics-Server 3. Create deployment 4. Create Horizontal Pod Autoscaler • Test increase load $ tmux $ kubectl run -i --tty load-generator --image=busybox /bin/sh $ while true; do wget -q -O- http://php-apache.default.svc.cluste r.local; done kubectl get hpa,deployments,po 52
53. • Services • Ingress • Run Application • Jobs • Daemon sets • Configmaps Outline 53
54. Jobs Run-to-completion, as opposed to run-forever • Express parallelism vs. required completions • Workflow: restart on failure • Build/test: don’t restart on failure Aggregates success/failure counts Built for batch and big-data work https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run- to-completion/ 54
55. Jobs (Cont.) apiVersion: batch/v1 kind: Job metadata: name: pi spec: template: spec: containers: - name: pi image: perl command: ["perl", "-Mbignum=bpi", "-wle", "print bpi(2000)"] restartPolicy: Never backoffLimit: 4 55
56. kubectl create -f https://k8s.io/examples/controllers/job.yaml kubectl describe jobs/pi pods=$(kubectl get pods --selector=job-name=pi --output=jsonpath={.items..metadata.name}) echo $pods kubectl logs $pods Jobs (Cont.) $ $ $ $ 56
57. Outline • Services • Ingress • Run Application • Jobs • Daemon sets • Configmaps 57
58. • Manage groups of replicated Pods • Ensures that all (or some) Nodes run a copy of a Pod • Typical uses ◦ a cluster storage daemon ◦ a logs collection daemon ◦ a node monitoring daemon DaemonSets 58
59. apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd namespace: kube-system labels: k8s-app: fluentd-logging spec: selector: matchLabels: name: fluentd # Label selector that determines which Pods belong to the DaemonSet template: metadata: labels: name: fluentd # Pod template's label selector spec: tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule containers: - name: fluentd image: gcr.io/google-containers/fluentd-elasticsearch:1.20 ... DaemonSets 59
60. $ $ $ kubectl create -f fluentd-ds.yaml kubectl get all -n kube-system -l k8s-app=fluentd-logging kubectl delete ds fluentd -n kube-system DaemonSets 60
61. Outline • Services • Ingress • Run Application • Jobs • Daemon sets • Configmaps 61
62. Configmap • Configure a Pod • Decouple configuration artifacts from image content • Keep containerized applications portable • Useful for storing & sharing non-sensitive, unencrypted configuration information 62
63. Configmap (Cont.) 1. Create configmap.yaml 2. Create configmap via kubectl 3. Using ConfigMap data 3.1. Define container ENV variables 3.2. in Pod commands 3.3. To a Volume 63
64. 1. Create configmap.yaml 2. Create configmap via kubectl 3. Using ConfigMap data 3.1. Define container ENV variables 3.2. in Pod commands 3.3. To a Volume Configmap (Cont.) kind: ConfigMap apiVersion: v1 metadata: name: example-config namespace: default data: # example of using --from-literal example.property.1: hello example.property.2: world # example of defined using --from-file example.property.file: |- property.1=value-1 property.2=value-2 property.3=value-3 64
65. 1. Create configmap.yaml 2. Create configmap via kubectl 3. Using ConfigMap data 3.1. Define container ENV variables 3.2. in Pod commands 3.3. To a Volume Configmap (Cont.) kubectl create -f example-config.yaml kubectl get cm example-config kubectl get cm example-config -o yaml 65
66. Configmap (Cont.) 1. Create configmap.yaml 2. Create configmap via kubectl 3. Using ConfigMap data 3.1. Define container ENV variables 3.2. in Pod commands 3.3. To a Volume apiVersion: v1 kind: Pod metadata: name: test-cm-pod1 spec: containers: - name: test-container image: k8s.gcr.io/busybox command: [ "/bin/sh", "-c", "env" ] env: - name: EXAMPLE_KEY2 valueFrom: configMapKeyRef: name: example-config key: example.property.2 restartPolicy: Never $ kubectl create -f env-pod1.yaml $ kubectl logs test-cm-pod1 66
67. Configmap (Cont.) 1. Create configmap.yaml 2. Create configmap via kubectl 3. Using ConfigMap data 3.1. Define container ENV variables 3.2. in Pod commands 3.3. To a Volume apiVersion: v1 kind: Pod metadata: name: test-cm-pod2 spec: containers: - name: test-container image: k8s.gcr.io/busybox command: [ "/bin/sh", "-c", "echo EXAMPLE_KEY1 is $(EXAMPLE_KEY1)" ] env: - name: EXAMPLE_KEY1 valueFrom: configMapKeyRef: name: example-config key: example.property.1 restartPolicy: Never $ kubectl create -f env-pod2.yaml $ kubectl logs test-cm-pod2 67
68. Configmap (Cont.) 1. Create configmap.yaml 2. Create configmap via kubectl 3. Using ConfigMap data 3.1. Define container ENV variables 3.2. in Pod commands 3.3. To a Volume apiVersion: v1 kind: Pod metadata: name: test-cm-pod3 spec: containers: - name: test-container image: k8s.gcr.io/busybox command: [ "/bin/sh","-c","cat /etc/config/myconfig" ] volumeMounts: - name: config-volume mountPath: /etc/config volumes: - name: config-volume configMap: # Provide the name of the ConfigMap containing the files you want # to add to the container name: example-config items: - key: example.property.file path: myconfig restartPolicy: Never $ kubectl create -f env-pod3.yaml $ kubectl logs test-cm-pod3 68
69. References • https://kubernetes.io • https://medium.com/google-cloud/kubernetes-nodeport-vs-loadbalancer-vs-i ngress-when-should-i-use-what-922f010849e0 • https://courses.cs.washington.edu/courses/cse550/16au/notes/kubernetes.p df • https://matthewpalmer.net/kubernetes-app-developer/articles/kubernetes-in gress-guide-nginx-example.html 69
70. Thanks for your attention. Slides: http://bit.ly/k8s1207 GitHub, Telegram: @sufuf3 Twitter: @sufuf3149 70