Kubernetes: от юнги до капитана

Transcript

1. Kubernetes: от юнги до капитана Игорь Зибарев
2. Адаптация приложения для Kubernetes Способы развертывания приложений Возможности платформы в деталях 2
3. Цель 3
4. Контейнер 4
5. 12factor.net 5
6. Config docker run \ -e SOME_SERVICE_ADDR="https: //some-service.company.tld" \ -e POSTGRES_DSN="postgres: //user@pg.company.tld:5432/db" \ -e LOG_FORMAT="json" \ -e LOG_LEVEL="info" \ -e CACHE_DIR="/var/cache" \ -e API_PORT="8080" \ duck-service:v1.2.3 6
7. Один процесс FROM golang:1.10-alpine AS builder COPY . . RUN go install FROM alpine:3.7 COPY --from=builder /go/bin/duck-service /usr/local/bin/duck-service USER 1000 ENTRYPOINT ["/usr/local/bin/duck-service"] 7
8. Одноразовый контейнер 8
9. Healthcheck 9
10. Graceful shutdown 10
11. var app DuckApp go func() { signals := make(chan os.Signal, 1) signal.Notify(signals, syscall.SIGTERM) <-signals log.Info("Shutting down ...") app.Shutdown() }() app.Run() 11
12. github.com/Yelp/dumb-init ENTRYPOINT ["/usr/bin/dumb-init", " --"] CMD ["bash", "-c", "do-some-pre-start-thing && exec my-app"] 12
13. Logs —Stdout / Stderr —Структурированные записи в JSON log.WithField(“s3.bucket", bucket). WithField("s3.key", key). Info("Upload to S3") 13
14. Metrics — Формат Prometheus — Pull-модель router.Handle("/metrics", PrometheusMetrics()) 14
15. Cloud Native 15
16. Kubernetes как PaaS 16
17. Infrastructure (compute, net, …) Platform (ecosystem) Apps 17
18. С чего начиналось развертывание .kube ├── configmaps │ └── nginx.conf ├── deployments │ ├── app.yaml │ └── web.yaml ├── hpa │ ├── app.yaml │ └── web.yaml └── services ├── app.yaml └── web.yaml 18
19. apiVersion: apps/v1 kind: Deployment metadata: name: duck-backend spec: replicas: 2 selector: matchLabels: app: duck-backend template: metadata: labels: app: duck-backend spec: containers: - name: duck-backend image: registry.example.tld/duck/backend:v1.2.3 envFrom: - configMapRef: name: duck-backend ports: - containerPort: 8080 protocol: TCP 19
20. Дублирование для сред .kube ├── configmaps │ ├── nginx-dev.conf │ └── nginx.conf ├── deployments │ ├── app-dev.yaml │ ├── app.yaml │ ├── web-dev.yaml │ └── web.yaml ├── hpa │ ├── app.yaml │ └── web.yaml └── services ├── app.yaml └── web.yaml 20
21. — sed/replace — Terraform provider — Helm 21
23. values.yaml # Default values for the application. image: repository: registry.example.tld/duck/backend tag: v1.2.3 pullPolicy: Always replicaCount: 3 resources: limits: cpu: 250m memory: 64Mi kafka: connString: "kafka.example.tld:9092" topic: "some-events" groupID: "app-consumers" 23
24. Кастомизация для среды # Values for minikube enviornment. replicaCount: 3 kafka: connString: "kafka.local:9092" config: logLeveL: debug 24
25. Новый сервис — helm create — Шаблон-ориентир — Интерактивная генерация 25
26. Chart Registry 26
27. Альтернатива: kustomize github.com/kubernetes-sigs/kustomize 27
28. Конфигурация Pod-а 28
29. Ресурсы apiVersion: apps/v1 kind: Deployment metadata: name: {{ template "app.fullname" . }} # ... # spec.template.spec.containers.* resources: requests: cpu: 250m memory: 40Mi limits: cpu: 400m memory: 64Mi 29
30. Diag endpoints func NewDiagRouter() *http.ServeMux { mux := http.NewServeMux() mux.Handle("/metrics", prometheusMetrics()) mux.Handle("/healthz", healthCheckHandler()) mux.HandleFunc("/debug/pprof/", pprof.Index) // ... return mux } 30
31. Несколько портов apiVersion: apps/v1 kind: Deployment metadata: name: {{ template "app.fullname" . }} # ... # spec.template.spec.containers.* ports: - containerPort: {{ .Values.apiPort }} protocol: TCP name: api - containerPort: {{ .Values.diagPort }} protocol: TCP name: diag 31
32. Пробы apiVersion: apps/v1 kind: Deployment metadata: name: {{ template "app.fullname" . }} # ... # spec.template.spec.containers.* livenessProbe: httpGet: path: /healthz port: diag periodSeconds: 5 timeoutSeconds: 2 32
33. Метрики apiVersion: apps/v1 kind: Deployment metadata: name: {{ template "app.fullname" . }} # ... # spec.template.annotations annotations: prometheus.io/scrape: "true" 33
34. Вместе с diag apiVersion: apps/v1 kind: Deployment # ... # spec.template.annotations annotations: prometheus.io/scrape: "true" prometheus.io/port: "{{ .Values.diagPort }}" # ... # spec.template.spec.containers.* ports: - containerPort: {{ .Values.apiPort }} protocol: TCP name: api - containerPort: {{ .Values.diagPort }} protocol: TCP name: diag 34
35. Exporter apiVersion: apps/v1 kind: Deployment # ... # spec.template.annotations annotations: prometheus.io/scrape: "true" prometheus.io/port: "metrics" # ... # spec.template.spec containers: - name: nginx image: nginx:1.14 # ... - name: prom-exporter image: sophos/nginx-vts-exporter:v0.9.0 ports: - name: metrics containerPort: 9913 35
36. Graceful shutdown Kubernetes выводит pod: —Pod status: ready ➡ terminating —SIGTERM 36
38. Lifecycle Hooks blog.openshift.com/kubernetes-pods-life/ apiVersion: apps/v1 kind: Deployment # ... # spec.template.spec containers: - name: app image: registry.example.tld/app:v1.2.3 # ... lifecycle: preStop: exec: command: ["sleep", "10"] 38
39. Эксплуатация платформы 39
40. Репликация apiVersion: apps/v1 kind: Deployment metadata: name: {{ template "app.fullname" . }} spec: replicas: {{ .Values.replicaCount }} # ... 40
41. Масштабирование apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: name: {{ template "app.fullname" . }} spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: {{ template "app.fullname" . }} minReplicas: 5 maxReplicas: 20 targetCPUUtilizationPercentage: 70 41
42. Входящий трафик — Service Load Balancer — Ingress 42
43. Service Load Balancer apiVersion: v1 kind: Service metadata: name: {{ template "app.fullname" . }} spec: type: LoadBalancer ports: - name: http port: 80 targetPort: {{ .Values.apiPort }} selector: app: {{ template "app.name" . }} 43
44. Ingress apiVersion: extensions/v1beta1 kind: Ingress metadata: name: {{ template "app.fullname" . }} annotations: kubernetes.io/ingress.class: "nginx" spec: rules: - host: app.public.tld http: paths: - path: / backend: serviceName: {{ template "app.fullname" . }} servicePort: http 44
45. Internal Load Balancer apiVersion: v1 kind: Service metadata: name: {{ template "app.fullname" . }} annotations: service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0 spec: type: LoadBalancer ports: - name: http port: 80 targetPort: {{ .Values.apiPort }} selector: app: {{ template "app.name" . }} 45
46. Internal Ingress apiVersion: extensions/v1beta1 kind: Ingress metadata: name: {{ template "app.fullname" . }} annotations: kubernetes.io/ingress.class: "nginx-internal" # <-- spec: rules: - host: app.private.tld http: paths: - path: / backend: serviceName: {{ template "app.fullname" . }} servicePort: http 46
47. github.com/kubernetes-incubator/external-dns 47
48. external-dns (SLB) apiVersion: v1 kind: Service metadata: name: {{ template "app.fullname" . }} annotations: external-dns.alpha.kubernetes.io/hostname: app.public.tld # <-- spec: type: LoadBalancer ports: - name: http port: 80 targetPort: {{ .Values.apiPort }} selector: app: {{ template "app.name" . }} 48
49. external-dns (Ingress) apiVersion: extensions/v1beta1 kind: Ingress metadata: name: {{ template "app.fullname" . }} annotations: kubernetes.io/ingress.class: "nginx" spec: rules: - host: app.public.tld # <-- http: paths: - path: / backend: serviceName: {{ template "app.fullname" . }} servicePort: http 49
50. TLS (SLB) apiVersion: v1 kind: Service metadata: name: {{ template "app.fullname" . }} annotations: service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:123456 ..." service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443" spec: type: LoadBalancer ports: - name: https port: 443 targetPort: {{ .Values.apiPort }} # ... 50
51. github.com/jetstack/cert-manager 51
52. TLS (Ingress) apiVersion: extensions/v1beta1 kind: Ingress metadata: name: {{ template "app.fullname" . }} annotations: kubernetes.io/ingress.class: "nginx" kubernetes.io/tls-acme: "true" # <-- spec: rules: - host: app.public.tld # ... tls: - hosts: - app.public.tld secretName: {{ template "app.fullname" . }} 52
53. Распределение по нодам —Affinity —Taints —Tolerations 53
54. podAntiAffinity apiVersion: apps/v1 kind: Deployment metadata: name: {{ template "app.fullname" . }} # spec.template.spec affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - topologyKey: "kubernetes.io/hostname" labelSelector: matchLabels: app: {{ template "app.name" . }} release: {{ .Release.Name }} 54
55. tolerations apiVersion: apps/v1 kind: Deployment metadata: name: {{ template "app.fullname" . }} labels: app: {{ template "app.name" . }} # ... # spec.template.spec tolerations: - key: "node-class" operator: Equal value: "preemptible" effect: NoSchedule 55
56. PodDisruptionBudget apiVersion: policy/v1beta1 kind: PodDisruptionBudget metadata: name: {{ template "app.fullname" . }} labels: app: {{ template "app.name" . }} spec: selector: matchLabels: app: {{ template "app.name" . }} release: {{ .Release.Name }} maxUnavailable: 1 # <-- 56
57. И еще океан возможностей RBAC / Admission Webhooks / StatefulSet / Network Policies / CustomResourceDefinition / Security Policies / CronJob / Volumes / Aggregation / … 57
58. Утилиты для разработчиков 58
59. Minikube — Локальная разработка — Без внешних зависимостей Docker for Mac github.com/kubernetes/minikube 59
60. Skaffold — Continuous Development — Build + Push + Deploy — File watcher github.com/GoogleContainerTools/skaffold 60
61. Shell Переключение namespace и cluster github.com/ahmetb/kubectx github.com/jonmosco/kube-ps1 61
62. God Mode alias k=kubectl 62
63. Итог 63
65. Спасибо за внимание! 65 t.me/zibarev github.com/hypnoglow