Наш опыт с Kubernetes в небольших проектах

93aef1d166a8a3536538eff713f80307?s=47 flant
June 06, 2017

Наш опыт с Kubernetes в небольших проектах

Доклад технического директора компании «Флант» (http://flant.ru/) Дмитрия Столярова на РИТ++ 2017 про устройство Kubernetes и наш опыт его использования в production.

* Текстовый обзор доклада: https://habrahabr.ru/company/flant/blog/331188/
* Видео с выступления: https://www.youtube.com/watch?v=CgCLPYJRxbU
* Анонс доклада на сайте конференции: http://rootconf.ru/2017/abstracts/2701

93aef1d166a8a3536538eff713f80307?s=128

flant

June 06, 2017
Tweet

Transcript

  1. Дмитрий Столяров CTO & Co-founder v2 Наш опыт с Kubernetes

    в небольших проектах
  2. # whoami dmitry.stolyarov # hostname -d flant.ru # cat /etc/motd

    Наш опыт с Kubernetes в небольших проектах Привет!
  3. 24×7×365 L1/L2/L3/+ DevOps SLA

  4. Небольшие проекты?

  5. Небольшие проекты? Kubernetes масштабируется до 5 000 nodes & 150

    000 pods
  6. Небольшие проекты? Маленькие < 50 nodes < 1 500 pods

    Большие > 500 nodes > 15 000 pods Средние 50–500 nodes 1500–15 000 pods Kubernetes масштабируется до 5 000 nodes & 150 000 pods
  7. Небольшие проекты? Маленькие < 50 nodes < 1 500 pods

    Большие > 500 nodes > 15 000 pods Средние 50–500 nodes 1500–15 000 pods Kubernetes масштабируется до 5 000 nodes & 150 000 pods Небольшие
  8. Часть 1: Предыстория 10 минут Часть 2: Kubernetes 30 минут

    Часть 3: Наш опыт 20 минут
  9. Часть 1: Предыстория 10 минут Часть 2: Kubernetes 30 минут

    Часть 3: Наш опыт 20 минут
  10. None
  11. None
  12. DB Backend

  13. DB Backend

  14. DB Backend Cron

  15. DB Backend memcached Cron

  16. Frontend DB Backend memcached Cron

  17. Frontend DB Backend memcached Files Cron

  18. Frontend DB Backend memcached Files Redis Background Jobs Cron

  19. Frontend DB Backend memcached Files Sphinx Background Jobs Redis Cron

  20. Frontend DB Backend memcached Files Sphinx Background Jobs Redis Cron

    Какие проблемы?
  21. None
  22. Frontend Backend memcached Files Sphinx Background Jobs Redis Cron DB

    master
  23. Frontend Backend memcached Files Sphinx Background Jobs Redis Cron DB

    master DB slave
  24. Redis Frontend Sphinx Backend memcached Background Jobs Cron DB master

    DB slave Files Backend Frontend memcached
  25. Frontend Backend memcached Sphinx Background Jobs Redis Cron DB master

    DB slave Frontend memcached Files Backend
  26. Backend memcached Files Sphinx Background Jobs Redis Cron DB master

    DB slave Backend memcached Frontend Frontend
  27. memcached Files Sphinx Background Jobs Redis Cron DB master DB

    slave memcached Frontend Frontend Backend Backend
  28. Frontend Backend memcached Files Sphinx Cron DB slave Frontend Backend

    memcached Redis DB master Background Jobs
  29. Frontend Backend Sphinx memcached Cron DB master DB slave Frontend

    Backend Background Jobs Redis memcached Files nfs-сервер
  30. Frontend Backend Sphinx memcached Cron DB master DB slave Frontend

    Backend Background Jobs Files nfs-сервер memcached Redis
  31. Frontend Backend Sphinx memcached DB master DB slave Frontend Backend

    Redis Files nfs-сервер memcached Cron Background Jobs
  32. Frontend Backend Sphinx memcached DB master DB slave Frontend Backend

    Redis Files nfs-сервер memcached Cron Background Jobs New Project Backend
  33. Frontend Backend Sphinx memcached DB master DB slave Frontend Backend

    Redis Files nfs-сервер memcached Cron Background Jobs New Project Backend Какие проблемы?
  34. None
  35. Cloud

  36. Frontend Frontend Backend Backend Backend Backend Backend Backend DB master

    DB slave Files Memcached Memcached Memcached Redis Cron Background Jobs Sphinx Sphinx
  37. Frontend Frontend Backend Backend Backend Backend Backend Backend DB master

    DB slave Files Memcached Memcached Memcached Redis Cron Background Jobs LB Sphinx Sphinx
  38. Frontend Frontend Backend Backend Backend Backend Backend Backend DB master

    DB slave Files Memcached Memcached Memcached Redis Cron Background Jobs LB Sphinx Sphinx Какие проблемы?
  39. Frontend Frontend Backend Backend Backend Backend Backend Backend DB master

    DB slave Sphinx Sphinx Frontend Frontend Backend Backend Backend Backend Backend Backend DB master DB slave Sphinx Sphinx LB Memcached Memcached Memcached Redis Cron Background Jobs S3 Files
  40. Frontend Frontend Sphinx Sphinx Frontend Frontend Sphinx Sphinx DB master

    DB slave DB master DB slave S3 Files LB Memcached Memcached Memcached Redis Cron Background Jobs Backend-1 Backend-2 Backend-N ... LB
  41. Frontend Frontend Sphinx Sphinx Frontend Frontend Sphinx Sphinx DB master

    DB slave DB master DB slave LB S3 Files Memcached Memcached Memcached Redis Cron Background Jobs Backend-1 Backend-2 Backend-N ...
  42. Backend-1 Backend-2 Backend-N ... Frontend Frontend Sphinx Sphinx Frontend Frontend

    Sphinx Sphinx LB S3 Files Memcached Memcached Memcached Redis Cron Background Jobs DB-1 DB-2 DB-N ...
  43. JOB-1 JOB-2 JOB-N ... Sphinx-1 Spninx-N ... LB DB-1 DB-2

    DB-N ... Backend-1 Backend-2 Backend-N ... Frontend Frontend Frontend Frontend MQ-1 MQ-2 MQ-N ... S3 Files Memcached Memcached Memcached Cron
  44. LB Frontend-1 Frontend-N ... LB Memcached 1 Memcached N ...

    Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... Backend-1 Backend-2 Backend-N ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron
  45. Микросервис S3 Files Cron MQ-1 MQ-2 MQ-N ... JOB-1 JOB-2

    JOB-N ... DB-1 DB-2 DB-N ... LB Backend 1 Backend 2 Backend N ... Sphinx-1 Spрinx-N ... Memcached 1 Memcached N ... Frontend 1 Frontend N ...
  46. Микросервис FOO Message Bus Микросервис BAR Микросервис BAZ ... LB

    Frontend-1 Frontend-N ...
  47. Простая Цирк Облачная Хорошая Идеальная Микросервисная

  48. Frontend 1 Frontend N ... Memcached 1 Memcached N ...

    Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... Backend 1 Backend 2 ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron LB LB Backend N
  49. Frontend 1 Frontend N ... Memcached 1 Memcached N ...

    Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... Backend 1 Backend 2 ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB Backend N
  50. Frontend 1 Frontend N ... Memcached 1 Memcached N ...

    Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... Backend 1 Backend 2 ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB Backend N 3. Supervision
  51. Frontend 1 Frontend N ... Memcached 1 Memcached N ...

    Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... Backend 1 Backend 2 ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB Backend N 3. Supervision 4. Service Discovery 5. Infrastructure Automation
  52. Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... JOB-1 JOB-2 JOB-N

    ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery Memcached 1 Memcached N ... ... Backend 1 Backend 2 ... Backend N Frontend 1 Frontend N 5. Infrastructure Automation
  53. Frontend 1 Frontend N ... Sphinx-1 Spninx-N ... DB-1 DB-2

    DB-N ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery Memcached 1 Memcached N ... Backend 1 Backend 2 ... Backend N 5. Infrastructure Automation
  54. Frontend 1 Frontend N ... Memcached 1 Memcached N ...

    Sphinx-1 Spninx-N ... Backend 1 Backend 2 ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB Backend N 3. Supervision 4. Service Discovery DB-1 DB-2 DB-N ... 5. Infrastructure Automation
  55. Frontend 1 Frontend N ... Memcached 1 Memcached N ...

    Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... Backend 1 Backend 2 ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB Backend N 3. Supervision 4. Service Discovery 5. Infrastructure Automation
  56. 5. Infrastructure Automation Frontend 1 Frontend N ... Memcached 1

    Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... Backend 1 Backend 2 ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB Backend N 3. Supervision 4. Service Discovery 6. (Auto) Scaling
  57. 6. (Auto) Scaling 5. Infrastructure Automation Frontend 1 Frontend N

    ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery 7. CI/CD Backend 1 Backend 2 ... Backend N
  58. 6. (Auto) Scaling 5. Infrastructure Automation Frontend 1 Frontend N

    ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery 7. CI/CD Backend 1 Backend 2 ... Backend N
  59. 6. (Auto) Scaling 5. Infrastructure Automation Frontend 1 Frontend N

    ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery 7. CI/CD Backend 1 Backend 2 ... Backend N JOB-1 JOB-2 JOB-N ...
  60. 6. (Auto) Scaling 5. Infrastructure Automation Frontend 1 Frontend N

    ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... MQ-1 MQ-2 MQ-N ... S3 Files 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery 7. CI/CD Backend 1 Backend 2 ... Backend N JOB-1 JOB-2 JOB-N ... Cron
  61. 6. (Auto) Scaling 5. Infrastructure Automation Frontend 1 Frontend N

    ... Memcached 1 Memcached N ... DB-1 DB-2 DB-N ... MQ-1 MQ-2 MQ-N ... S3 Files 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery 7. CI/CD Backend 1 Backend 2 ... Backend N JOB-1 JOB-2 JOB-N ... Cron Sphinx-1 Spninx-N ...
  62. 6. (Auto) Scaling 5. Infrastructure Automation Memcached 1 Memcached N

    ... DB-1 DB-2 DB-N ... MQ-1 MQ-2 MQ-N ... S3 Files 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery 7. CI/CD Backend 1 Backend 2 ... Backend N JOB-1 JOB-2 JOB-N ... Cron Sphinx-1 Spninx-N ... Frontend 1 Frontend N ...
  63. Лучшие практики Continuous Delivery c Docker – 2016 habrahabr.ru/company/flant/blog/322686/

  64. Главный паттерн деплоя Старая версия работает 1

  65. Главный паттерн деплоя Старая версия работает 1 Новая версия “прогревается”

    2
  66. Главный паттерн деплоя Старая версия работает 1 Новая версия “прогревается”

    2 Переключили трафик 3
  67. Главный паттерн деплоя Старая версия работает Новая версия “прогревается” Переключили

    трафик Старая версия остановлена 1 2 3 4
  68. 6. (Auto) Scaling 5. Infrastructure Automation Memcached 1 Memcached N

    ... DB-1 DB-2 DB-N ... MQ-1 MQ-2 MQ-N ... S3 Files 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery 7. CI/CD Backend 1 Backend 2 ... Backend N JOB-1 JOB-2 JOB-N ... Cron Sphinx-1 Spninx-N ... Frontend 1 Frontend N ...
  69. 7. CI/CD 6. (Auto) Scaling 5. Infrastructure Automation Frontend 1

    Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery Backend 1 Backend 2 ... Backend N 8. Vendor lock-in
  70. 7. CI/CD 6. (Auto) Scaling 5. Infrastructure Automation Frontend 1

    Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery Backend 1 Backend 2 ... Backend N 8. Vendor lock-in
  71. None
  72. Часть 1: Предыстория 10 минут Часть 2: Kubernetes 30 минут

    Часть 3: Наш опыт 20 минут
  73. Архитектура Kubernetes

  74. master Архитектура Kubernetes

  75. node-1 node-2 ... master Архитектура Kubernetes

  76. node-1 node-2 ... master Архитектура Kubernetes Docker Docker

  77. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet

  78. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet

    kube-proxy kube-proxy
  79. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet

    kube-proxy kube-proxy
  80. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet

    kube-proxy kube-proxy API Server
  81. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet

    kube-proxy kube-proxy API Server
  82. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet

    kube-proxy kube-proxy API Server etcd
  83. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet

    kube-proxy kube-proxy API Server etcd
  84. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet

    kube-proxy kube-proxy API Server etcd Scheduler
  85. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet

    kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager
  86. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet

    kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager kubectl
  87. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet

    kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager kubectl YAML (Declarative DSL)
  88. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet

    kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager kubectl YAML (Declarative DSL)
  89. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet

    kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager kubectl YAML (Declarative DSL)
  90. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet

    kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager kubectl YAML (Declarative DSL)
  91. Набор строительных блоков

  92. Container

  93. Container Image Cmd

  94. Container Image Cmd 1. Logging

  95. Container Image Cmd 3. Supervision 1. Logging

  96. Pod

  97. Pod Container 1 Container 2 ...

  98. Pod Container 1 Container 2 ... localhost eth0

  99. Pod Container 1 Container 2 ... localhost eth0 volume1 volume2

  100. Pod Container 1 Container 2 ... localhost eth0 volume1 volume2

    tier: backend version: 0.15.7
  101. Label & Selector

  102. Label & Selector app: foo

  103. Label & Selector app: foo tier: backend version: 0.15.7

  104. Label & Selector app: foo tier: backend version: 0.15.7 partition:

    customer_a
  105. Label & Selector app: foo tier: backend version: 0.15.7 partition:

    customer_a app in (foo, bar)
  106. Label & Selector app: foo tier: backend version: 0.15.7 partition:

    customer_a app in (foo, bar) tier == backend
  107. Label & Selector app: foo tier: backend version: 0.15.7 partition:

    customer_a app in (foo, bar) tier == backend partition != customer_a
  108. Pod Container 1 Container 2 ... localhost eth0 volume1 volume2

    tier: backend version: 0.15.7
  109. Pod Container 1 Container 2 ... localhost eth0 volume1 volume2

    tier: backend 5. Infrastructure Automation version: 0.15.7
  110. 3. Supervision Pod Container 1 Container 2 ... localhost eth0

    volume1 volume2 tier: backend 5. Infrastructure Automation version: 0.15.7
  111. 3. Supervision Pod Container 1 Container 2 ... localhost eth0

    volume1 volume2 tier: backend 5. Infrastructure Automation 4. Service Discovery version: 0.15.7
  112. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet

    kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager kubectl YAML (Declarative DSL)
  113. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet

    kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager kubectl YAML (Declarative DSL)
  114. apiVersion: v1 kind: Pod metadata: name: manual-bash spec: containers: -

    name: bash image: ubuntu:16.04 command: bash args: [-c, “while true; do sleep 1; date; done”]
  115. apiVersion: v1 kind: Pod metadata: name: manual-bash spec: containers: -

    name: bash image: ubuntu:16.04 command: bash args: [-c, “while true; do sleep 1; date; done”]
  116. apiVersion: v1 kind: Pod metadata: name: manual-bash spec: containers: -

    name: bash image: ubuntu:16.04 command: bash args: [-c, “while true; do sleep 1; date; done”]
  117. apiVersion: v1 kind: Pod metadata: name: manual-bash spec: containers: -

    name: bash image: ubuntu:16.04 command: bash args: [-c, “while true; do sleep 1; date; done”]
  118. apiVersion: v1 kind: Pod metadata: name: manual-bash spec: containers: -

    name: bash image: ubuntu:16.04 command: bash args: [-c, “while true; do sleep 1; date; done”]
  119. apiVersion: v1 kind: Pod metadata: name: manual-bash spec: containers: -

    name: bash image: ubuntu:16.04 command: bash args: [-c, “while true; do sleep 1; date; done”]
  120. apiVersion: v1 kind: Pod metadata: name: manual-backend labels: tier: backend

    spec: containers: - name: web image: myregistry.local/backend:0.15.9 command: /usr/bin/php-fpm
  121. apiVersion: v1 kind: Pod metadata: name: manual-backend labels: tier: backend

    spec: containers: - name: web image: myregistry.local/backend:0.15.9 command: /usr/bin/php-fpm
  122. apiVersion: v1 kind: Pod metadata: name: manual-backend labels: tier: backend

    spec: containers: - name: web image: myregistry.local/backend:0.15.9 command: /usr/bin/php-fpm
  123. ReplicaSet

  124. ReplicaSet N

  125. None
  126. Current Ready Status Desired 2 2 2

  127. 2 2 2 Current Ready Status Desired 3 2 2

  128. 2 2 2 3 2 2 Current Ready Status Desired

    3 3 2
  129. 2 2 2 3 2 2 3 3 2 Current

    Ready Status Desired 3 3 3
  130. 2 2 2 3 2 2 3 3 2 3

    3 3 Current Ready Status Desired 1 3 3
  131. 2 2 2 3 2 2 3 3 2 3

    3 3 1 3 3 Current Ready Status Desired 1 3 1
  132. 2 2 2 3 2 2 3 3 2 3

    3 3 1 3 3 1 3 1 Current Ready Status Desired 1 1 1
  133. ReplicaSet N 6. (Auto) Scaling 3. Supervision

  134. apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: name: backend spec: replicas: 3

    selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.7 command: php-fpm
  135. apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: name: backend spec: replicas: 3

    selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.7 command: php-fpm
  136. apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: name: backend spec: replicas: 3

    selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.7 command: php-fpm
  137. apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: name: backend spec: replicas: 3

    selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.7 command: php-fpm
  138. apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: name: backend spec: replicas: 3

    selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.7 command: php-fpm
  139. apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: name: backend spec: replicas: 3

    selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.7 command: php-fpm
  140. apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: name: backend spec: replicas: 3

    selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.7 command: php-fpm
  141. apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: name: backend spec: replicas: 3

    selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.7 command: php-fpm
  142. apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: name: backend spec: replicas: 3

    selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.7 command: php-fpm
  143. Deployment

  144. Deployment N

  145. Deployment N + History 0 0 ...

  146. Deployment N + History 0 0 ... + Deploy

  147. Deployment N + History 0 0 ... + Deploy Pod

    = N
  148. 3 0.15.7

  149. 3 0.15.7

  150. 3 0.15.7 0 0.15.9

  151. 3 0.15.7 1 0.15.9

  152. 3 0.15.7 1 0.15.9

  153. 3 0.15.7 1 0.15.9

  154. 2 0.15.7 1 0.15.9

  155. 2 0.15.7 1 0.15.9

  156. 2 0.15.7 1 0.15.9

  157. 2 0.15.7 2 0.15.9

  158. 1 0.15.7 2 0.15.9

  159. 1 0.15.7 3 0.15.9

  160. 0 0.15.7 3 0.15.9

  161. 0 0.15.7 3 0.15.9

  162. 0 0.15.7 3 0.15.9 0 0.16.3

  163. 0 0.15.7 2 0.15.9 1 0.16.3

  164. 0 0.15.7 1 0.15.9 2 0.16.3

  165. 0 0.15.7 0 0.15.9 3 0.16.3

  166. 0 0.15.7 0 0.15.9 3 0.16.3

  167. Deployment Pod + = + History Deploy N N 0

    0 ... 7. CI/CD
  168. apiVersion: apps/v1beta1 kind: Deployment metadata: name: backend spec: replicas: 3

    selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.9 command: php-fpm
  169. apiVersion: apps/v1beta1 kind: Deployment metadata: name: backend spec: replicas: 3

    selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.9 command: php-fpm
  170. N Deployment

  171. Service DNS

  172. Service DNS + Virtual IP

  173. Service DNS Load Balancer + Virtual IP +

  174. Selector Service DNS Load Balancer + Virtual IP + +

  175. Selector Service Service DNS Load Balancer + = Virtual IP

    + +
  176. None
  177. tier: backend tier: backend tier: backend tier: backend tier: backend

    tier: backend
  178. tier: backend tier: backend tier: backend tier: backend tier: backend

    tier: backend version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.7
  179. tier: backend tier: backend tier: backend tier: backend tier: backend

    tier: backend version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.9 version: 0.15.9
  180. tier: backend tier: backend tier: backend tier: backend tier: backend

    tier: backend version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.9 version: 0.15.9 tier: db tier: db tier: db tier: sidekiq
  181. tier: backend tier: backend tier: backend tier: backend tier: backend

    tier: backend version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.9 version: 0.15.9 tier: sidekiq tier: db tier: db tier: db backend tier==backend
  182. tier: db tier: sidekiq tier: db tier: db tier: backend

    tier: backend tier: backend tier: backend tier: backend tier: backend version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.9 version: 0.15.9 backend tier==backend
  183. tier: db tier: sidekiq tier: db tier: db tier: backend

    tier: backend tier: backend tier: backend tier: backend tier: backend version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.9 version: 0.15.9 backend tier==backend
  184. tier: backend version: 0.15.7 tier: db tier: sidekiq tier: db

    tier: db tier: backend tier: backend tier: backend version: 0.15.7 version: 0.15.7 version: 0.15.7 backend tier==backend tier==backend new-backend version==0.15.9 tier: backend version: 0.15.9 tier: backend version: 0.15.9
  185. kind: Service apiVersion: v1 metadata: name: backend spec: selector: tier:

    backend ports: - protocol: TCP port: 80 targetPort: 8080
  186. kind: Service apiVersion: v1 metadata: name: backend spec: selector: tier:

    backend ports: - protocol: TCP port: 80 targetPort: 8080
  187. kind: Service apiVersion: v1 metadata: name: backend spec: selector: tier:

    backend ports: - protocol: TCP port: 80 targetPort: 8080
  188. N Deployment Service

  189. 6. Job

  190. 6. Job Pod

  191. 6. Job Pod Successful Completion +

  192. 6. Job Pod Successful Completion Job + =

  193. apiVersion: batch/v1 kind: Job metadata: name: migrate spec: template: metadata:

    name: migrate spec: containers: - name: migrate image: myregistry.local/backend:0.15.9 command: [yiic, migrate] restartPolicy: Never
  194. apiVersion: batch/v1 kind: Job metadata: name: migrate spec: template: metadata:

    name: migrate spec: containers: - name: migrate image: myregistry.local/backend:0.15.9 command: [yiic, migrate] restartPolicy: Never
  195. apiVersion: batch/v1 kind: Job metadata: name: migrate spec: template: metadata:

    name: migrate spec: containers: - name: migrate image: myregistry.local/backend:0.15.9 command: [yiic, migrate] restartPolicy: Never
  196. N Deployment Service Job

  197. CronJob

  198. CronJob Job

  199. Shedule * * * * * CronJob Job +

  200. Shedule * * * * * CronJob Job CronJob +

    =
  201. apiVersion: batch/v2alpha1 kind: CronJob metadata: name: cron-foo spec: schedule: "*/1

    * * * *" jobTemplate: spec: template: spec: containers: - name: main image: myregistry.local/backend:0.15.9 args: [yiic, cron-foo] restartPolicy: OnFailure
  202. apiVersion: batch/v2alpha1 kind: CronJob metadata: name: cron-foo spec: schedule: "*/1

    * * * *" jobTemplate: spec: template: spec: containers: - name: main image: myregistry.local/backend:0.15.9 args: [yiic, cron-foo] restartPolicy: OnFailure
  203. apiVersion: batch/v2alpha1 kind: CronJob metadata: name: cron-foo spec: schedule: "*/1

    * * * *" jobTemplate: spec: template: spec: containers: - name: main image: myregistry.local/backend:0.15.9 args: [yiic, cron-foo] restartPolicy: OnFailure
  204. apiVersion: batch/v2alpha1 kind: CronJob metadata: name: cron-foo spec: schedule: "*/1

    * * * *" jobTemplate: spec: template: spec: containers: - name: main image: myregistry.local/backend:0.15.9 args: [yiic, cron-foo] restartPolicy: OnFailure
  205. N Deployment Service Job CronJob

  206. Volume

  207. Volume Volume Storage Class Access Type Size + + =

  208. Volume Volume Storage Class Access Type Size + + =

    ReadOnly Many ReadWrite Once ReadWrite Many
  209. Volume Volume Storage Class Access Type Size + + =

    normal slow fast Admin defined Storage Classes
  210. Volume Volume Storage Class Access Type Size + + =

    AWSElasticBlockStore GCEPersistentDisk AzureFile FC (Fibre Channel) AzureDisk FlexVolume Flocker iSCSI NFS RBD Cinder CephFS Glusterfs Quobyte Volumes VsphereVolume HostPath VMware Photon ScaleIO Volumes Portworx Volumes normal slow fast Admin defined Storage Classes
  211. Volume Volume Storage Class Access Type Size + + =

    AWSElasticBlockStore GCEPersistentDisk AzureFile FC (Fibre Channel) AzureDisk FlexVolume Flocker iSCSI NFS RBD Cinder CephFS Glusterfs Quobyte Volumes VsphereVolume HostPath VMware Photon ScaleIO Volumes Portworx Volumes normal slow fast Admin defined Storage Classes Using Interface Implementation
  212. Pod Volume

  213. N ReplicaSet Pod Volume

  214. N ReplicaSet Pod Volume ...

  215. N ReplicaSet Pod Volume Volume ...

  216. N N Deployment ReplicaSet Pod Volume Volume ... Volume ...

  217. Volume Volume ... Volume ... N N ReadWriteOnce ReadWriteOnce ReadWriteOnce

    Deployment ReplicaSet Pod
  218. StatefulSet

  219. StatefulSet 0 = N 1 N-1 ...

  220. foo 3 Deployment

  221. foo 3 Deployment foo-khku8 foo-nacti foo-z9gth

  222. bar foo 3 Deployment 3 StatefulSet foo-khku8 foo-nacti foo-z9gth

  223. bar foo 3 Deployment 3 StatefulSet foo-khku8 foo-nacti foo-z9gth bar-0

    bar-1 bar-2
  224. Volume ... N Deployment

  225. Volume ... N ... ... N StatefulSet Deployment

  226. N Deployment StatefulSet Service Job CronJob N

  227. foo bar foo N bar N Ingress

  228. foo bar Internet foo N bar N Ingress

  229. foo bar Internet foo N bar N Ingress public ip

    public ip
  230. foo bar foo N bar N Ingress foo bar Internet

    foo N bar N public ip public ip
  231. foo bar foo N bar N Ingress foo bar Internet

    foo N bar N public ip public ip Ingress
  232. foo bar foo N bar N Ingress foo bar Internet

    foo N bar N public ip public ip Ingress public ip
  233. foo bar foo N bar N Ingress foo bar Internet

    foo N bar N public ip public ip Ingress public ip Internet
  234. Ingress Ingress Service + = Hostname Path + + TLS

  235. apiVersion: extensions/v1beta1 kind: Ingress metadata: name: main spec: rules: -

    host: foo.bar.com http: paths: - path: /assets backend: serviceName: frontend servicePort: 80 - path: / backend: serviceName: backend servicePort: 80
  236. apiVersion: extensions/v1beta1 kind: Ingress metadata: name: main spec: rules: -

    host: foo.bar.com http: paths: - path: /assets backend: serviceName: frontend servicePort: 80 - path: / backend: serviceName: backend servicePort: 80
  237. apiVersion: extensions/v1beta1 kind: Ingress metadata: name: main spec: rules: -

    host: foo.bar.com http: paths: - path: /assets backend: serviceName: frontend servicePort: 80 - path: / backend: serviceName: backend servicePort: 80
  238. apiVersion: extensions/v1beta1 kind: Ingress metadata: name: main spec: rules: -

    host: foo.bar.com http: paths: - path: /assets backend: serviceName: frontend servicePort: 80 - path: / backend: serviceName: backend servicePort: 80
  239. apiVersion: extensions/v1beta1 kind: Ingress metadata: name: main spec: rules: -

    host: foo.bar.com http: paths: - path: /assets backend: serviceName: frontend servicePort: 80 - path: / backend: serviceName: backend servicePort: 80
  240. Ingress Using Interface Implementation

  241. Ingress Using Interface Implementation Nginx GCE Ingress Controllers

  242. N Deployment StatefulSet Service Ingress Job CronJob N

  243. 4. Service Discovery 1. Logging 2. Metrics 3. Supervision 6.

    (Auto) Scaling 7. CI/CD 8. Vendor lock-in Вызовы 5. Infrastructure Automation
  244. 4. Service Discovery 3. Supervision 6. (Auto) Scaling 7. CI/CD

    8. Vendor lock-in Вызовы 5. Infrastructure Automation 1. Logging 2. Metrics
  245. 4. Service Discovery 6. (Auto) Scaling 7. CI/CD 8. Vendor

    lock-in Вызовы 5. Infrastructure Automation 1. Logging 2. Metrics 3. Supervision
  246. 6. (Auto) Scaling 7. CI/CD 8. Vendor lock-in Вызовы 5.

    Infrastructure Automation 1. Logging 2. Metrics 3. Supervision 4. Service Discovery
  247. 6. (Auto) Scaling 7. CI/CD 8. Vendor lock-in Вызовы 1.

    Logging 2. Metrics 3. Supervision 4. Service Discovery 5. Infrastructure Automation
  248. 7. CI/CD 8. Vendor lock-in Вызовы 1. Logging 2. Metrics

    3. Supervision 4. Service Discovery 5. Infrastructure Automation 6. (Auto) Scaling
  249. 8. Vendor lock-in Вызовы 1. Logging 2. Metrics 3. Supervision

    4. Service Discovery 5. Infrastructure Automation 6. (Auto) Scaling 7. CI/CD
  250. Вызовы 1. Logging 2. Metrics 3. Supervision 4. Service Discovery

    5. Infrastructure Automation 6. (Auto) Scaling 7. CI/CD 8. Vendor lock-in
  251. None
  252. Часть 1: Предыстория 10 минут Часть 2: Kubernetes 30 минут

    Часть 3: Наш опыт 20 минут
  253. Маленький проект на Bare metal

  254. Маленький проект на Bare metal hv-1 hv-2 hv-3

  255. Маленький проект на Bare metal hv-1 hv-2 hv-3

  256. Маленький проект на Bare metal hv-1 hv-2 hv-3

  257. Маленький проект на Bare metal hv-1 hv-2 hv-3

  258. Маленький проект на Bare metal hv-1 hv-2 hv-3

  259. Маленький проект на Bare metal hv-1 hv-2 hv-3 kube-master-1 kube-master-2

    kube-master-3
  260. Маленький проект на Bare metal hv-1 hv-2 hv-3 kube-master-1 kube-master-2

    kube-master-3 kube-node-1 kube-node-2 kube-node-3
  261. Маленький проект на Bare metal hv-1 hv-2 hv-3 kube-master-1 kube-master-2

    kube-master-3 kube-node-1 kube-node-2 kube-node-3 kube-front-1 kube-front-2 kube-front-3
  262. Маленький проект на Bare metal hv-1 hv-2 hv-3 kube-master-1 kube-master-2

    kube-master-3 kube-node-1 kube-node-2 kube-node-3 kube-front-1 kube-front-2 kube-front-3 ... ... ...
  263. Маленький проект на Bare metal hv-1 hv-2 hv-3 kube-node-1 kube-front-1

    kube-node-2 kube-front-2 kube-node-3 kube-front-3 gitlab vpn kube-master-1 kube-master-2 kube-master-3 nfs ... ... ... ...
  264. Что мы пока не размещаем в Kubernetes?

  265. Что мы пока не размещаем в Kubernetes? 1. High IO

    MySQL, MongoDB, PostgreSQL, Cassandra
  266. Что мы пока не размещаем в Kubernetes? * Durable (non-shared)

    local storage management https://github.com/kubernetes/features/issues/121 1. High IO * MySQL, MongoDB, PostgreSQL, Cassandra
  267. Что мы пока не размещаем в Kubernetes? * Durable (non-shared)

    local storage management https://github.com/kubernetes/features/issues/121 1. High IO * MySQL, MongoDB, PostgreSQL, Cassandra 2. Cloud non native cluster leader election, fencing
  268. Что мы пока не размещаем в Kubernetes? * Durable (non-shared)

    local storage management https://github.com/kubernetes/features/issues/121 1. High IO * MySQL, MongoDB, PostgreSQL, Cassandra 2. Cloud non native cluster leader election, fencing … а также:
  269. Что мы пока не размещаем в Kubernetes? * Durable (non-shared)

    local storage management https://github.com/kubernetes/features/issues/121 1. High IO * MySQL, MongoDB, PostgreSQL, Cassandra 2. Cloud non native cluster leader election, fencing 3. Страх “большие” базы … а также:
  270. Что мы пока не размещаем в Kubernetes? * Durable (non-shared)

    local storage management https://github.com/kubernetes/features/issues/121 1. High IO * MySQL, MongoDB, PostgreSQL, Cassandra 2. Cloud non native cluster leader election, fencing 3. Страх “большие” базы 4. Лень VPN server, NFS server … а также:
  271. Что мы пока не размещаем в Kubernetes? * Durable (non-shared)

    local storage management https://github.com/kubernetes/features/issues/121 1. High IO * MySQL, MongoDB, PostgreSQL, Cassandra 2. Cloud non native cluster leader election, fencing 3. Страх “большие” базы 4. Лень VPN server, NFS server 5. Инерция Gitlab, Ceph OSD … а также:
  272. Как мы делаем Ingress

  273. Как мы делаем Ingress kube-front-1 kube-front-2 kube-front-3

  274. Как мы делаем Ingress kube-front-1 kube-front-2 kube-front-3 Nginx Ingress Controller

    Nginx Ingress Controller Nginx Ingress Controller
  275. Как мы делаем Ingress kube-front-1 kube-front-2 kube-front-3 Nginx Ingress Controller

    Nginx Ingress Controller Nginx Ingress Controller Highly Available Public IP 3 x
  276. Как мы делаем Ingress kube-front-1 kube-front-2 kube-front-3 Nginx Ingress Controller

    Nginx Ingress Controller Nginx Ingress Controller Highly Available Public IP 3 x VRRP VRRP VRRP
  277. Как мы делаем Ingress kube-front-1 kube-front-2 kube-front-3 Nginx Ingress Controller

    Nginx Ingress Controller Nginx Ingress Controller Highly Available Public IP 3 x Pacemaker Pacemaker Pacemaker
  278. Nginx Ingress Controller Nginx Ingress Controller Nginx Ingress Controller Как

    мы делаем Ingress kube-front-1 kube-front-2 kube-front-3 Highly Available Public IP 3 x DNS Load Balancing Pacemaker Pacemaker Pacemaker
  279. git build test release operate Как мы делаем Continuous delivery

  280. None
  281. foo bar baz nats mongo

  282. foo bar baz nats mongo production staging testing

  283. foo bar baz nats mongo production staging testing foo foo

    foo
  284. foo bar baz nats mongo production staging testing foo bar

    baz nats mongo foo bar baz nats mongo foo bar baz nats mongo
  285. git build test release operate Dockerfile .kube/ backend.yaml frontend.yaml cron.yaml

  286. git build test release operate Dockerfile .kube/ backend.yaml frontend.yaml cron.yaml

    docker build Docker- image
  287. git build test release operate Dockerfile .kube/ backend.yaml frontend.yaml cron.yaml

    docker build Docker- image Docker registry
  288. git build test release operate Dockerfile .kube/ backend.yaml frontend.yaml cron.yaml

    docker build Docker- image Docker registry Docker- image tests +
  289. git build test release operate Dockerfile .kube/ backend.yaml frontend.yaml cron.yaml

    docker build Docker- image Docker registry Docker- image tests + kubectl
  290. git build test release operate Dockerfile .kube/ backend.yaml frontend.yaml cron.yaml

    docker build Docker- image Docker registry Docker- image tests + kubectl
  291. .kube/ backend.yaml frontend.yaml cron.yaml Docker- image Docker registry Docker- image

    tests + kubectl git build test release operate Dockerfile docker build
  292. .kube/ backend.yaml frontend.yaml cron.yaml Docker- image Docker registry Docker- image

    tests + kubectl git build test release operate Dappfile build
  293. Docker- image Docker registry Docker- image tests + git build

    test release operate Dappfile .kube/ backend.yaml frontend.yaml cron.yaml kubectl build
  294. Docker- image Docker registry Docker- image tests + git build

    test release operate Dappfile .kube/ backend.yaml frontend.yaml cron.yaml build
  295. Docker- image Docker registry Docker- image tests + git build

    test release operate Dappfile .kube/ backend.yaml frontend.yaml cron.yaml deploy build
  296. Мы хотим сделать dapp действительно крутым и полезным Open Source-решением

    Нам нужен ВАШ опыт использования Будем рады любой помощи Поставьте звёздочку ;) Обещаем очень быстро отвечать на все issues и очень быстро исправлять все баги. Для нас dapp — это приоритетный проект. Если есть желание помочь с переводом документации на английский или с её доработкой, да или вообще помочь — будем очень рады и ждем pull-реквесты. Мы старались. github.com/flant/dapp
  297. Собираем Docker- образы быстро и удобно habrahabr.ru/company/flant/blog/324274/

  298. Инфраструктура НЕбольшого проекта

  299. Инфраструктура НЕбольшого проекта

  300. Инфраструктура НЕбольшого проекта Storage Ceph AWS GCE

  301. Инфраструктура НЕбольшого проекта Storage Ceph AWS GCE Ingress

  302. Инфраструктура НЕбольшого проекта Ingress Highly Available VM Storage Ceph AWS

    GCE
  303. Инфраструктура НЕбольшого проекта Ingress Highly Available VM IOPS VM Storage

    Ceph AWS GCE
  304. Frontend DB Backend memcached Cron Было

  305. Highly Available VM IOPS VM Frontend DB Backend memcached Cron

    Было Стало Storage Ceph AWS GCE
  306. Highly Available VM IOPS VM Frontend DB Backend memcached Cron

    Было Стало backend 3 Storage Ceph AWS GCE
  307. Highly Available VM IOPS VM Frontend DB Backend memcached Cron

    Было Стало backend 3 DB Storage Ceph AWS GCE
  308. Highly Available VM IOPS VM Frontend DB Backend memcached Cron

    Было Стало backend 3 DB cron-foo cron-bar cron-baz Storage Ceph AWS GCE
  309. Highly Available VM IOPS VM Frontend DB Backend memcached Cron

    Было Стало backend 3 DB cron-foo cron-bar cron-baz frontend Storage Ceph AWS GCE
  310. Highly Available VM IOPS VM Frontend DB Backend memcached Cron

    Было Стало backend 3 DB cron-foo cron-bar cron-baz frontend memcached 3 Storage Ceph AWS GCE
  311. Highly Available VM IOPS VM DB frontend backend 3 memcached

    3 cron-foo cron-bar cron-baz Frontend DB Backend memcached Cron Было Стало Files Storage Ceph AWS GCE
  312. Highly Available VM IOPS VM Storage Ceph AWS GCE DB

    frontend backend 3 memcached 3 cron-foo cron-bar cron-baz Frontend DB Backend memcached Cron Было Стало Files Background Jobs Redis Redis 3 Background Jobs 3
  313. Highly Available VM IOPS VM DB frontend backend 3 memcached

    3 cron-foo cron-bar cron-baz Frontend DB Backend memcached Cron Было Стало Files Background Jobs Redis Redis 3 Background Jobs 3 Sphinx 2 Sphinx Storage Ceph AWS GCE
  314. Highly Available VM IOPS VM frontend backend 3 memcached 3

    cron-foo cron-bar cron-baz Frontend DB Backend memcached Cron Было Стало Files Background Jobs Redis Redis 3 Background Jobs 3 Sphinx 2 Sphinx Storage Ceph AWS GCE DB 3
  315. Простая Цирк Облачная Хорошая Идеальная Микросервисная

  316. Хорошая Облачная Хорошая Идеальная Микросервисная Цирк

  317. Хорошая Облачная Хорошая Идеальная Микросервисная Хорошая

  318. Хорошая Хорошая Хорошая Хорошая Идеальная Микросервисная

  319. Хорошая Хорошая Хорошая Идеальная Микросервисная Хорошая

  320. Хорошая Хорошая Хорошая Хорошая Идеальная Микросервисная

  321. Хорошая Хорошая Хорошая Хорошая Идеальная Микросервисная

  322. Наш блог на Хабрахабре habrahabr.ru/company/flant/ Тут мы рассказываем о нашем

    опыте и актуальных технологиях. Подписывайтесь, чтобы быть в курсе!
  323. Наши Docker-проекты github.com/flant/dapp github.com/flant/docker_penetration_experiment github.com/flant/pam_docker github.com/flant/php_fpm_docker github.com/flant/superhosting github.com/flant/buildizer Дмитрий Столяров

    dmitry.stolyarov@flant.ru linkedin.com/in/distol github.com/distol Всем спасибо! Наш блог на Хабрахабре habrahabr.ru/company/flant/