Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Перевозим Spring на Kubernetes

Перевозим Spring на Kubernetes

Как настроить:
- Service Discovery (Service);
- Configuration (ConfigMap / Secrets);
- Gateway (Ingress);
Spring Cloud Starter Kubernetes

Dmitrii Rygalov

February 27, 2019
Tweet

More Decks by Dmitrii Rygalov

Other Decks in Programming

Transcript

  1. © 2019 True Engineering. All rights reserved. Алексей Зотов и

    Дмитрий Рыгалов ……. . Перевозим Spring на Kubernetes Engineering True
  2. Дмитрий Рыгалов [email protected] @Dymanich @Dymanich Mityi [email protected] Алексей Зотов [email protected]

    @lexzotov @lexzotov lex4hex [email protected] В главных ролях Разработчики из True Engineering по 6 & 8 лет в IT !2
  3. Кто ты и зачем здесь? Кто использует Spring Netflix? Кто

    планирует мигрировать на Kubernetes? Кто уже мигрировал в Kubernetes? !3
  4. О чём будем говорить: Переход со стека NetflixOSS на Kubernetes

    1. Service Discovery 2. Configuration (ConfigMap / Secrets) 3. Gateway !4
  5. !5

  6. Рассказ о том, как мы получили эту архитектуру https://youtu.be/IZ-J77KZ6PA До

    перехода Zuul Nginx (Static) HAProxy Config Server Eureka MS 1 MS 2 MS 3 Actuators JARs systemd RabbitMQ Hazelcast MySQL !7
  7. Датацентр #1 Датацентр #N Hazelcast RabbitMQ До перехода Рассказ о

    том, как мы получили эту архитектуру https://youtu.be/IZ-J77KZ6PA HAProxy JARs HAProxy JARs MySQL MySQL !8
  8. Количество запросов 3500 700 О требованиях Рассказ о том, как

    мы получили эту архитектуру https://youtu.be/IZ-J77KZ6PA !9
  9. DEV OPS 1. Tooling 2. Gitlab CI 3. systemD 4.

    Сonfiguration 5. Infrastructure services 1. Tooling 2. Load Balancing 3. Network 4. systemD 5. Production Deployment 6. Databases 7. Monitoring Зоны ответственности !10
  10. Поводы для беспокойства Microservices Concerns Config management Service Discovery &

    LB Service Security API Management Resilience & Fault Tolerance Centralized Logging Centralized Metrics Scheduling & Deployment Auto Scaling & Self Healing Distributed Tracing !12
  11. Расширяем инструментарий Microservices Concerns Config management Service Discovery & LB

    Service Security API Management Resilience & Fault Tolerance Centralized Logging Centralized Metrics Scheduling & Deployment Auto Scaling & Self Healing Distributed Tracing !13
  12. Добавляем Kubernetes Microservices Concerns Config management Service Discovery & LB

    Service Security API Management Resilience & Fault Tolerance Centralized Logging Centralized Metrics Scheduling & Deployment Auto Scaling & Self Healing Distributed Tracing !14
  13. kubelet Архитектура Kubernetes kube-proxy Kubernetes Minions Kubernetes Master Cloud kube-controller-

    manager cloud-controller- manager kube-api-server kube-scheduler etcd kubelet kube-proxy kubelet kube-proxy !17
  14. Service Ingress ConfigMap Secret Pod 1 Pod 2 Pod 3

    Deployment namespace Kubernetes objects !18
  15. apiVersion: apps/v1 kind: Deployment metadata: name: my-frontend spec: replicas: 3

    template: metadata: labels: name: my-frontend spec: containers: - name: my-frontend image: my-f:1. Deployment !19
  16. kind: ConfigMap metadata: name: example-configmap apiVersion: v1 data: key: example-value

    application.properties: |- property.example1=example-value-1 property.example2=example-value-2 ConfigMap !20
  17. apiVersion: v1 kind: Secret metadata: name: example-secret type: Opaque data:

    username: YWRtaW4= password: MWYyZDFlMmU2N2Rm Secrets !21
  18. kind: Service apiVersion: v1 metadata: name: my-service spec: selector: app:

    MyApp ports: - protocol: TCP port: 80 targetPort: 9376 Service !22
  19. apiVersion: extensions/v1beta1 kind: Ingress metadata: name: my-ingress annotations: rewrite-target: /

    spec: rules: - host: my.local http: paths: - path: / backend: serviceName: client servicePort: 8080 Ingress !23
  20. Spring cloud & Netflix OSS Kubernetes Service Discovery Eureka, Consul

    Service & Ingress Load Balancing Ribbon Service Gateway Spring Gateway, Zuul Service & Ingress Security Spring Vault Secrets Configuration Management Config Server ConfigMap Resilience & Fault Tolerance Actuator, Hystrix Health-check & Self- Healing & Autoscaling Spring Cloud vs Kubernetes !27
  21. plugins { id 'com.palantir.docker' } Dockerfile: FROM openjdk:8-jre-slim ARG JAR_FILE

    ADD ${JAR_FILE} app.jar ENV JAVA_OPTS="" ENTRYPOINT ["sh", "-c", "java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar /app.jar”] Containerization !29
  22. cat > /usr/lib/systemd/system/my-server.service << EOF [Unit] Description=Embedded App - my-server.jar

    After=syslog.target [Service] EnvironmentFile=/etc/sysconfig/myserver.prop User=hrdt ExecStart=/opt/servers/my-server.jar SuccessExitStatus=143 [Install] WantedBy=multi-user.target EOF Deployments (before) !31
  23. apiVersion: apps/v1 kind: Deployment metadata: name: my-server spec: replicas: 1

    template: metadata: labels: name: my-server spec: containers: - name: my-server image: nexus/my-server:1.0 envFrom: - configMapRef: name: myServerProp Deployments (after) !32
  24. @SpringBootApplication @EnableConfigServer public class ConfigServer { public static void main(String[]

    args) { SpringApplication.run(ConfigServer.class, args); } } spring: cloud: config: server: git: uri: file://${user.home}/config-repo Spring-cloud-config-server Как было раньше? !34
  25. Spring’s PropertiesPropertySourceLoader Apache commons-configuration var configuration = new PropertiesConfiguration(filePath); var

    reloadingStrategy = new FileChangedReloadingStrategy(); reloadingStrategy.setRefreshDelay(REFRESH_DELAY); configuration.setReloadingStrategy(reloadingStrategy); configuration.refresh(); String name = "file:" + path.getFile().getAbsolutePath(); List<PropertySource<?>> propertySources = loader.load(name, path); environment.getPropertySources().addLast(propertySource); Как было раньше? !35
  26. Варианты использования ConfigMap 1. Environment Variables 2. Volume Mount 3.

    Spring-cloud-kubernetes-config 4. Spring-cloud-kubernetes-config with Volume Mount !38
  27. - name: EXAMPLE_ENV_PROPERTY valueFrom: configMapKeyRef: name: example-environment-configmap key: env.property.example deployment.yaml

    apiVersion: v1 kind: ConfigMap metadata: name: example-environment-configmap namespace: default data: env.property.example: Hello, I'm from environment configmap.yaml Environment Variables !39
  28. volumeMounts: - mountPath: "/opt/config/" name: config-volume readOnly: true restartPolicy: Always

    volumes: - name: config-volume configMap: name: example-configmap items: - key: application.properties path: application.properties apiVersion: v1 kind: ConfigMap metadata: name: example-configmap namespace: default data: application.properties: |- example.property=Hello! Mount application.properties to volume deployment.yaml configmap.yaml !40
  29. volumeMounts: - mountPath: /volume name: config-volume readOnly: true restartPolicy: Always

    volumes: - name: config-volume configMap: name: volume items: - key: example.properties path: example.properties spring: profiles: volume cloud: kubernetes: config: paths: “/volume/ example.properties” deployment.yaml bootstrap.yaml spring-cloud-kubernetes- config with Volume Mount !41
  30. spec: containers: - name: configmap-test-app image: image imagePullPolicy: Never env:

    - name: SPRING_PROFILES_ACTIVE value: sck-profile spring: profiles: sck-profile cloud: kubernetes: reload: enabled: true mode: polling period: 5000 config: name: reloadable-configmap bootstrap.yaml deployment.yaml @Configuration @ConfigurationProperties(prefix = "reloadable") @Data public class ReloadableProperty { private String property; } Spring-cloud-kubernetes-config !44
  31. env: - name: SPRING_PROFILES_ACTIVE value: secrets-profile volumeMounts: - mountPath: /secrets-volume

    name: secrets-volume readOnly: true restartPolicy: Always volumes: - name: secrets-volume secret: secretName: example-secret spring: profiles: secrets-profile cloud: kubernetes: secrets: paths: "/secrets-volume/" deployment.yaml bootstrap.yaml Secret !47
  32. Service Pod А Labels •app=demo •component=front Pod B Labels •app=demo

    •component=front Service: frontend Selector •app=demo •component=front Pod C Labels •app=demo •component=users Pod D Labels •app=demo •component=users Service: users Selector •app=demo •component=users Client Pod Labels •app=demo •component=client 50
  33. kind: Service metadata: name: my-server spec: selector: app: server-tag type:

    ClusterIP ports: - protocol: TCP port: 8080 targetPort: 8080 kind: Deployment metadata: name: my-server spec: replicas: 1 template: metadata: labels: app: server-tag spec: containers: - name: my-server image: my-srv:1.0 Service selectors 51
  34. Environment variables: • {SVCNAME}_SERVICE_HOST=10.0.0.11 • {SVCNAME}_SERVICE_PORT=6379 DNS (Ex. service name

    = my-service): • my-service • my-service.my-namespace Discovering services 52
  35. var address = ”my-service”; var address = "MY_SERVICE_SERVICE_HOST" + ":"

    + "MY_SERVICE_SERVICE_PORT"; restTemplate .getForObject(address, Response.class); @FeignClient(url = "${service.address}") interface Client {…} Example !53
  36. class KubernetesDiscoveryClient { private KubernetesClient client; // real method have

    65 lines List<ServiceInstance> getInstances(serviceId) { var endpoints = client.endpoints(serviceId); var service = client.services(serviceId); return endpoints.stream() .flatMap(endpoint.getAddresses()) .map((addr, port) -> new KubernetesServiceInstance( serviceId, addr, port, service.getMetadata())) .collect(Collectors.toList()); } } KubernetesDiscoveryClient 56
  37. apiVersion: v1 kind: Endpoints metadata: name: my-server subsets: - addresses:

    #pod ip’s - ip: 10.1.1.175 - ip: 10.1.1.176 ports: - port: 8080 protocol: TCP Endpoints 57
  38. kind: Deployment spec: containers: readinessProbe: httpGet: path: /actuator/health port: 8080

    scheme: HTTP initialDelaySeconds: 10 periodSeconds: 5 failureThreshold: 5 livenessProbe: httpGet: path: /actuator/health port: 8080 scheme: HTTP initialDelaySeconds: 20 periodSeconds: 5 failureThreshold: 5 Readiness/ Liveness Probe 58
  39. Ingress Pods Pods Pod Service 1 Pods Pods Pod Service

    2 Pods Pods Pod Service 1 Pods Pods Pod Service 2 Ingress Controller 61
  40. kind: Ingress metadata: name: my-ingress annotations: nginx.ingress.kubernetes.io/rewrite-target: /$1 spec: rules:

    - host: jug.nsk.ru http: paths: - path: /?(.*) backend: serviceName: frontend servicePort: 8080 - path: /api?(.*) backend: serviceName: root-app servicePort: 8080 Ingress rules 62
  41. kind: Ingress metadata: name: my-ingress annotations: nginx.ingress.kubernetes.io/rewrite-target: /$1 spec: rules:

    - host: jug.nsk.ru http: paths: - path: /v21/?(.*) backend: serviceName: my-server servicePort: 8080 - path: /api?(.*) backend: serviceName: root-app servicePort: 8080 Ingress rules 64
  42. # для локальной разработки application.yaml eureka.client.enabled: true spring.cloud.kubernetes.enabled: false server.port:

    0 # внутри кластера использовать kind: ConfigMap metadata: name: kubernetes-lb-configmap data: application.properties: |- server.port: 8080 spring.cloud.kubernetes.enabled: true eureka.client.enabled: false eureka.client.enabled: true 73
  43. Архитектура внутри кластера DB HZ Rabbit namespace ConfigMap A ConfigMap

    common Secrets B Service DB HZ Rabbit Ingress Service A Service B Pods Pods Pods Deployment А Pods Pods Pods Deployment B !77
  44. DEV OPS 1. Tooling 2. Gitlab CI 3. systemD 4.

    Сonfiguration 5. Infrastructure services 1. Tooling 2. Load Balancing 3. Network 4. systemD 5. Production Deployment 6. Databases 7. Monitoring Зоны ответственности «до» !78
  45. Зоны ответственности «после» DEV OPS 1. Уменьшилось количество сервисов 2.

    systemD scripts -> k8s objects 3. Deployment 4. ConfigMap 5. Service 6. Ingress 7. Helm 1. Rabbit, hazelcast. 2. docker registry 3. Ingress 4. Внешний LB 5. Настройки кластеров 6. Мониторинг 7. Namespace configuration 8. Persistent Volumes (PV) 9. Service 10. Secret (SSL, registry, token, TLS) !79
  46. + Consistency + Scaling + Deploy + Configuration - Локальная

    разработка - Долго и трудно - Неактуальная документация - Есть баги - Порог вхождения Основные +/- !80
  47. Не изменили ли вы свое мнение? Кто всё ещё планирует

    мигрировать на Kubernetes? Кто испытал схожие проблемы при миграции Kubernetes? !82
  48. Примеры использования ConfigMap https://github.com/Lex4hex/k8s-config-map Примеры работы с Service и Ingress

    https://github.com/mityi/demo-migration-to-k8s Как перейти на микросервисы и не разломать prod https://youtu.be/IZ-J77KZ6PA Баги и таски: https://github.com/OpenFeign/feign/issues/872 https://github.com/spring-cloud/spring-cloud-kubernetes/pull/329 https://github.com/spring-cloud/spring-cloud-kubernetes/pull/328 QA !83