Наш опыт с Kubernetes в небольших проектах

Доклад технического директора компании «Флант» (http://flant.ru/) Дмитрия Столярова на РИТ++ 2017 про устройство Kubernetes и наш опыт его использования в production.

* Текстовый обзор доклада: https://habrahabr.ru/company/flant/blog/331188/
* Видео с выступления: https://www.youtube.com/watch?v=CgCLPYJRxbU
* Анонс доклада на сайте конференции: http://rootconf.ru/2017/abstracts/2701

Transcript

1. Дмитрий Столяров CTO & Co-founder v2 Наш опыт с Kubernetes в небольших проектах
2. # whoami dmitry.stolyarov # hostname -d flant.ru # cat /etc/motd Наш опыт с Kubernetes в небольших проектах Привет!
3. 24×7×365 L1/L2/L3/+ DevOps SLA
4. Небольшие проекты?
5. Небольшие проекты? Kubernetes масштабируется до 5 000 nodes & 150 000 pods
6. Небольшие проекты? Маленькие < 50 nodes < 1 500 pods Большие > 500 nodes > 15 000 pods Средние 50–500 nodes 1500–15 000 pods Kubernetes масштабируется до 5 000 nodes & 150 000 pods
7. Небольшие проекты? Маленькие < 50 nodes < 1 500 pods Большие > 500 nodes > 15 000 pods Средние 50–500 nodes 1500–15 000 pods Kubernetes масштабируется до 5 000 nodes & 150 000 pods Небольшие
8. Часть 1: Предыстория 10 минут Часть 2: Kubernetes 30 минут Часть 3: Наш опыт 20 минут
9. Часть 1: Предыстория 10 минут Часть 2: Kubernetes 30 минут Часть 3: Наш опыт 20 минут
12. DB Backend
13. DB Backend
14. DB Backend Cron
15. DB Backend memcached Cron
16. Frontend DB Backend memcached Cron
17. Frontend DB Backend memcached Files Cron
18. Frontend DB Backend memcached Files Redis Background Jobs Cron
19. Frontend DB Backend memcached Files Sphinx Background Jobs Redis Cron
20. Frontend DB Backend memcached Files Sphinx Background Jobs Redis Cron Какие проблемы?
22. Frontend Backend memcached Files Sphinx Background Jobs Redis Cron DB master
23. Frontend Backend memcached Files Sphinx Background Jobs Redis Cron DB master DB slave
24. Redis Frontend Sphinx Backend memcached Background Jobs Cron DB master DB slave Files Backend Frontend memcached
25. Frontend Backend memcached Sphinx Background Jobs Redis Cron DB master DB slave Frontend memcached Files Backend
26. Backend memcached Files Sphinx Background Jobs Redis Cron DB master DB slave Backend memcached Frontend Frontend
27. memcached Files Sphinx Background Jobs Redis Cron DB master DB slave memcached Frontend Frontend Backend Backend
28. Frontend Backend memcached Files Sphinx Cron DB slave Frontend Backend memcached Redis DB master Background Jobs
29. Frontend Backend Sphinx memcached Cron DB master DB slave Frontend Backend Background Jobs Redis memcached Files nfs-сервер
30. Frontend Backend Sphinx memcached Cron DB master DB slave Frontend Backend Background Jobs Files nfs-сервер memcached Redis
31. Frontend Backend Sphinx memcached DB master DB slave Frontend Backend Redis Files nfs-сервер memcached Cron Background Jobs
32. Frontend Backend Sphinx memcached DB master DB slave Frontend Backend Redis Files nfs-сервер memcached Cron Background Jobs New Project Backend
33. Frontend Backend Sphinx memcached DB master DB slave Frontend Backend Redis Files nfs-сервер memcached Cron Background Jobs New Project Backend Какие проблемы?
35. Cloud
36. Frontend Frontend Backend Backend Backend Backend Backend Backend DB master DB slave Files Memcached Memcached Memcached Redis Cron Background Jobs Sphinx Sphinx
37. Frontend Frontend Backend Backend Backend Backend Backend Backend DB master DB slave Files Memcached Memcached Memcached Redis Cron Background Jobs LB Sphinx Sphinx
38. Frontend Frontend Backend Backend Backend Backend Backend Backend DB master DB slave Files Memcached Memcached Memcached Redis Cron Background Jobs LB Sphinx Sphinx Какие проблемы?
39. Frontend Frontend Backend Backend Backend Backend Backend Backend DB master DB slave Sphinx Sphinx Frontend Frontend Backend Backend Backend Backend Backend Backend DB master DB slave Sphinx Sphinx LB Memcached Memcached Memcached Redis Cron Background Jobs S3 Files
40. Frontend Frontend Sphinx Sphinx Frontend Frontend Sphinx Sphinx DB master DB slave DB master DB slave S3 Files LB Memcached Memcached Memcached Redis Cron Background Jobs Backend-1 Backend-2 Backend-N ... LB
41. Frontend Frontend Sphinx Sphinx Frontend Frontend Sphinx Sphinx DB master DB slave DB master DB slave LB S3 Files Memcached Memcached Memcached Redis Cron Background Jobs Backend-1 Backend-2 Backend-N ...
42. Backend-1 Backend-2 Backend-N ... Frontend Frontend Sphinx Sphinx Frontend Frontend Sphinx Sphinx LB S3 Files Memcached Memcached Memcached Redis Cron Background Jobs DB-1 DB-2 DB-N ...
43. JOB-1 JOB-2 JOB-N ... Sphinx-1 Spninx-N ... LB DB-1 DB-2 DB-N ... Backend-1 Backend-2 Backend-N ... Frontend Frontend Frontend Frontend MQ-1 MQ-2 MQ-N ... S3 Files Memcached Memcached Memcached Cron
44. LB Frontend-1 Frontend-N ... LB Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... Backend-1 Backend-2 Backend-N ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron
45. Микросервис S3 Files Cron MQ-1 MQ-2 MQ-N ... JOB-1 JOB-2 JOB-N ... DB-1 DB-2 DB-N ... LB Backend 1 Backend 2 Backend N ... Sphinx-1 Spрinx-N ... Memcached 1 Memcached N ... Frontend 1 Frontend N ...
46. Микросервис FOO Message Bus Микросервис BAR Микросервис BAZ ... LB Frontend-1 Frontend-N ...
47. Простая Цирк Облачная Хорошая Идеальная Микросервисная
48. Frontend 1 Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... Backend 1 Backend 2 ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron LB LB Backend N
49. Frontend 1 Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... Backend 1 Backend 2 ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB Backend N
50. Frontend 1 Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... Backend 1 Backend 2 ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB Backend N 3. Supervision
51. Frontend 1 Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... Backend 1 Backend 2 ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB Backend N 3. Supervision 4. Service Discovery 5. Infrastructure Automation
52. Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery Memcached 1 Memcached N ... ... Backend 1 Backend 2 ... Backend N Frontend 1 Frontend N 5. Infrastructure Automation
53. Frontend 1 Frontend N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery Memcached 1 Memcached N ... Backend 1 Backend 2 ... Backend N 5. Infrastructure Automation
54. Frontend 1 Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... Backend 1 Backend 2 ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB Backend N 3. Supervision 4. Service Discovery DB-1 DB-2 DB-N ... 5. Infrastructure Automation
55. Frontend 1 Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... Backend 1 Backend 2 ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB Backend N 3. Supervision 4. Service Discovery 5. Infrastructure Automation
56. 5. Infrastructure Automation Frontend 1 Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... Backend 1 Backend 2 ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB Backend N 3. Supervision 4. Service Discovery 6. (Auto) Scaling
57. 6. (Auto) Scaling 5. Infrastructure Automation Frontend 1 Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery 7. CI/CD Backend 1 Backend 2 ... Backend N
58. 6. (Auto) Scaling 5. Infrastructure Automation Frontend 1 Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery 7. CI/CD Backend 1 Backend 2 ... Backend N
59. 6. (Auto) Scaling 5. Infrastructure Automation Frontend 1 Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery 7. CI/CD Backend 1 Backend 2 ... Backend N JOB-1 JOB-2 JOB-N ...
60. 6. (Auto) Scaling 5. Infrastructure Automation Frontend 1 Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... MQ-1 MQ-2 MQ-N ... S3 Files 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery 7. CI/CD Backend 1 Backend 2 ... Backend N JOB-1 JOB-2 JOB-N ... Cron
61. 6. (Auto) Scaling 5. Infrastructure Automation Frontend 1 Frontend N ... Memcached 1 Memcached N ... DB-1 DB-2 DB-N ... MQ-1 MQ-2 MQ-N ... S3 Files 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery 7. CI/CD Backend 1 Backend 2 ... Backend N JOB-1 JOB-2 JOB-N ... Cron Sphinx-1 Spninx-N ...
62. 6. (Auto) Scaling 5. Infrastructure Automation Memcached 1 Memcached N ... DB-1 DB-2 DB-N ... MQ-1 MQ-2 MQ-N ... S3 Files 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery 7. CI/CD Backend 1 Backend 2 ... Backend N JOB-1 JOB-2 JOB-N ... Cron Sphinx-1 Spninx-N ... Frontend 1 Frontend N ...
63. Лучшие практики Continuous Delivery c Docker – 2016 habrahabr.ru/company/flant/blog/322686/
64. Главный паттерн деплоя Старая версия работает 1
65. Главный паттерн деплоя Старая версия работает 1 Новая версия “прогревается” 2
66. Главный паттерн деплоя Старая версия работает 1 Новая версия “прогревается” 2 Переключили трафик 3
67. Главный паттерн деплоя Старая версия работает Новая версия “прогревается” Переключили трафик Старая версия остановлена 1 2 3 4
68. 6. (Auto) Scaling 5. Infrastructure Automation Memcached 1 Memcached N ... DB-1 DB-2 DB-N ... MQ-1 MQ-2 MQ-N ... S3 Files 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery 7. CI/CD Backend 1 Backend 2 ... Backend N JOB-1 JOB-2 JOB-N ... Cron Sphinx-1 Spninx-N ... Frontend 1 Frontend N ...
69. 7. CI/CD 6. (Auto) Scaling 5. Infrastructure Automation Frontend 1 Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery Backend 1 Backend 2 ... Backend N 8. Vendor lock-in
70. 7. CI/CD 6. (Auto) Scaling 5. Infrastructure Automation Frontend 1 Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery Backend 1 Backend 2 ... Backend N 8. Vendor lock-in
72. Часть 1: Предыстория 10 минут Часть 2: Kubernetes 30 минут Часть 3: Наш опыт 20 минут
73. Архитектура Kubernetes
74. master Архитектура Kubernetes
75. node-1 node-2 ... master Архитектура Kubernetes
76. node-1 node-2 ... master Архитектура Kubernetes Docker Docker
77. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet
78. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy
79. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy
80. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server
81. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server
82. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server etcd
83. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server etcd
84. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server etcd Scheduler
85. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager
86. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager kubectl
87. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager kubectl YAML (Declarative DSL)
88. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager kubectl YAML (Declarative DSL)
89. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager kubectl YAML (Declarative DSL)
90. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager kubectl YAML (Declarative DSL)
91. Набор строительных блоков
92. Container
93. Container Image Cmd
94. Container Image Cmd 1. Logging
95. Container Image Cmd 3. Supervision 1. Logging
96. Pod
97. Pod Container 1 Container 2 ...
98. Pod Container 1 Container 2 ... localhost eth0
99. Pod Container 1 Container 2 ... localhost eth0 volume1 volume2
100. Pod Container 1 Container 2 ... localhost eth0 volume1 volume2 tier: backend version: 0.15.7
101. Label & Selector
102. Label & Selector app: foo
103. Label & Selector app: foo tier: backend version: 0.15.7
104. Label & Selector app: foo tier: backend version: 0.15.7 partition: customer_a
105. Label & Selector app: foo tier: backend version: 0.15.7 partition: customer_a app in (foo, bar)
106. Label & Selector app: foo tier: backend version: 0.15.7 partition: customer_a app in (foo, bar) tier == backend
107. Label & Selector app: foo tier: backend version: 0.15.7 partition: customer_a app in (foo, bar) tier == backend partition != customer_a
108. Pod Container 1 Container 2 ... localhost eth0 volume1 volume2 tier: backend version: 0.15.7
109. Pod Container 1 Container 2 ... localhost eth0 volume1 volume2 tier: backend 5. Infrastructure Automation version: 0.15.7
110. 3. Supervision Pod Container 1 Container 2 ... localhost eth0 volume1 volume2 tier: backend 5. Infrastructure Automation version: 0.15.7
111. 3. Supervision Pod Container 1 Container 2 ... localhost eth0 volume1 volume2 tier: backend 5. Infrastructure Automation 4. Service Discovery version: 0.15.7
112. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager kubectl YAML (Declarative DSL)
113. node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager kubectl YAML (Declarative DSL)
114. apiVersion: v1 kind: Pod metadata: name: manual-bash spec: containers: - name: bash image: ubuntu:16.04 command: bash args: [-c, “while true; do sleep 1; date; done”]
115. apiVersion: v1 kind: Pod metadata: name: manual-bash spec: containers: - name: bash image: ubuntu:16.04 command: bash args: [-c, “while true; do sleep 1; date; done”]
116. apiVersion: v1 kind: Pod metadata: name: manual-bash spec: containers: - name: bash image: ubuntu:16.04 command: bash args: [-c, “while true; do sleep 1; date; done”]
117. apiVersion: v1 kind: Pod metadata: name: manual-bash spec: containers: - name: bash image: ubuntu:16.04 command: bash args: [-c, “while true; do sleep 1; date; done”]
118. apiVersion: v1 kind: Pod metadata: name: manual-bash spec: containers: - name: bash image: ubuntu:16.04 command: bash args: [-c, “while true; do sleep 1; date; done”]
119. apiVersion: v1 kind: Pod metadata: name: manual-bash spec: containers: - name: bash image: ubuntu:16.04 command: bash args: [-c, “while true; do sleep 1; date; done”]
120. apiVersion: v1 kind: Pod metadata: name: manual-backend labels: tier: backend spec: containers: - name: web image: myregistry.local/backend:0.15.9 command: /usr/bin/php-fpm
121. apiVersion: v1 kind: Pod metadata: name: manual-backend labels: tier: backend spec: containers: - name: web image: myregistry.local/backend:0.15.9 command: /usr/bin/php-fpm
122. apiVersion: v1 kind: Pod metadata: name: manual-backend labels: tier: backend spec: containers: - name: web image: myregistry.local/backend:0.15.9 command: /usr/bin/php-fpm
123. ReplicaSet
124. ReplicaSet N
126. Current Ready Status Desired 2 2 2
127. 2 2 2 Current Ready Status Desired 3 2 2
128. 2 2 2 3 2 2 Current Ready Status Desired 3 3 2
129. 2 2 2 3 2 2 3 3 2 Current Ready Status Desired 3 3 3
130. 2 2 2 3 2 2 3 3 2 3 3 3 Current Ready Status Desired 1 3 3
131. 2 2 2 3 2 2 3 3 2 3 3 3 1 3 3 Current Ready Status Desired 1 3 1
132. 2 2 2 3 2 2 3 3 2 3 3 3 1 3 3 1 3 1 Current Ready Status Desired 1 1 1
133. ReplicaSet N 6. (Auto) Scaling 3. Supervision
134. apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: name: backend spec: replicas: 3 selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.7 command: php-fpm
135. apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: name: backend spec: replicas: 3 selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.7 command: php-fpm
136. apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: name: backend spec: replicas: 3 selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.7 command: php-fpm
137. apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: name: backend spec: replicas: 3 selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.7 command: php-fpm
138. apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: name: backend spec: replicas: 3 selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.7 command: php-fpm
139. apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: name: backend spec: replicas: 3 selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.7 command: php-fpm
140. apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: name: backend spec: replicas: 3 selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.7 command: php-fpm
141. apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: name: backend spec: replicas: 3 selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.7 command: php-fpm
142. apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: name: backend spec: replicas: 3 selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.7 command: php-fpm
143. Deployment
144. Deployment N
145. Deployment N + History 0 0 ...
146. Deployment N + History 0 0 ... + Deploy
147. Deployment N + History 0 0 ... + Deploy Pod = N
148. 3 0.15.7
149. 3 0.15.7
150. 3 0.15.7 0 0.15.9
151. 3 0.15.7 1 0.15.9
152. 3 0.15.7 1 0.15.9
153. 3 0.15.7 1 0.15.9
154. 2 0.15.7 1 0.15.9
155. 2 0.15.7 1 0.15.9
156. 2 0.15.7 1 0.15.9
157. 2 0.15.7 2 0.15.9
158. 1 0.15.7 2 0.15.9
159. 1 0.15.7 3 0.15.9
160. 0 0.15.7 3 0.15.9
161. 0 0.15.7 3 0.15.9
162. 0 0.15.7 3 0.15.9 0 0.16.3
163. 0 0.15.7 2 0.15.9 1 0.16.3
164. 0 0.15.7 1 0.15.9 2 0.16.3
165. 0 0.15.7 0 0.15.9 3 0.16.3
166. 0 0.15.7 0 0.15.9 3 0.16.3
167. Deployment Pod + = + History Deploy N N 0 0 ... 7. CI/CD
168. apiVersion: apps/v1beta1 kind: Deployment metadata: name: backend spec: replicas: 3 selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.9 command: php-fpm
169. apiVersion: apps/v1beta1 kind: Deployment metadata: name: backend spec: replicas: 3 selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.9 command: php-fpm
170. N Deployment
171. Service DNS
172. Service DNS + Virtual IP
173. Service DNS Load Balancer + Virtual IP +
174. Selector Service DNS Load Balancer + Virtual IP + +
175. Selector Service Service DNS Load Balancer + = Virtual IP + +
177. tier: backend tier: backend tier: backend tier: backend tier: backend tier: backend
178. tier: backend tier: backend tier: backend tier: backend tier: backend tier: backend version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.7
179. tier: backend tier: backend tier: backend tier: backend tier: backend tier: backend version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.9 version: 0.15.9
180. tier: backend tier: backend tier: backend tier: backend tier: backend tier: backend version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.9 version: 0.15.9 tier: db tier: db tier: db tier: sidekiq
181. tier: backend tier: backend tier: backend tier: backend tier: backend tier: backend version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.9 version: 0.15.9 tier: sidekiq tier: db tier: db tier: db backend tier==backend
182. tier: db tier: sidekiq tier: db tier: db tier: backend tier: backend tier: backend tier: backend tier: backend tier: backend version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.9 version: 0.15.9 backend tier==backend
183. tier: db tier: sidekiq tier: db tier: db tier: backend tier: backend tier: backend tier: backend tier: backend tier: backend version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.9 version: 0.15.9 backend tier==backend
184. tier: backend version: 0.15.7 tier: db tier: sidekiq tier: db tier: db tier: backend tier: backend tier: backend version: 0.15.7 version: 0.15.7 version: 0.15.7 backend tier==backend tier==backend new-backend version==0.15.9 tier: backend version: 0.15.9 tier: backend version: 0.15.9
185. kind: Service apiVersion: v1 metadata: name: backend spec: selector: tier: backend ports: - protocol: TCP port: 80 targetPort: 8080
186. kind: Service apiVersion: v1 metadata: name: backend spec: selector: tier: backend ports: - protocol: TCP port: 80 targetPort: 8080
187. kind: Service apiVersion: v1 metadata: name: backend spec: selector: tier: backend ports: - protocol: TCP port: 80 targetPort: 8080
188. N Deployment Service
189. 6. Job
190. 6. Job Pod
191. 6. Job Pod Successful Completion +
192. 6. Job Pod Successful Completion Job + =
193. apiVersion: batch/v1 kind: Job metadata: name: migrate spec: template: metadata: name: migrate spec: containers: - name: migrate image: myregistry.local/backend:0.15.9 command: [yiic, migrate] restartPolicy: Never
194. apiVersion: batch/v1 kind: Job metadata: name: migrate spec: template: metadata: name: migrate spec: containers: - name: migrate image: myregistry.local/backend:0.15.9 command: [yiic, migrate] restartPolicy: Never
195. apiVersion: batch/v1 kind: Job metadata: name: migrate spec: template: metadata: name: migrate spec: containers: - name: migrate image: myregistry.local/backend:0.15.9 command: [yiic, migrate] restartPolicy: Never
196. N Deployment Service Job
197. CronJob
198. CronJob Job
199. Shedule * * * * * CronJob Job +
200. Shedule * * * * * CronJob Job CronJob + =
201. apiVersion: batch/v2alpha1 kind: CronJob metadata: name: cron-foo spec: schedule: "*/1 * * * *" jobTemplate: spec: template: spec: containers: - name: main image: myregistry.local/backend:0.15.9 args: [yiic, cron-foo] restartPolicy: OnFailure
202. apiVersion: batch/v2alpha1 kind: CronJob metadata: name: cron-foo spec: schedule: "*/1 * * * *" jobTemplate: spec: template: spec: containers: - name: main image: myregistry.local/backend:0.15.9 args: [yiic, cron-foo] restartPolicy: OnFailure
203. apiVersion: batch/v2alpha1 kind: CronJob metadata: name: cron-foo spec: schedule: "*/1 * * * *" jobTemplate: spec: template: spec: containers: - name: main image: myregistry.local/backend:0.15.9 args: [yiic, cron-foo] restartPolicy: OnFailure
204. apiVersion: batch/v2alpha1 kind: CronJob metadata: name: cron-foo spec: schedule: "*/1 * * * *" jobTemplate: spec: template: spec: containers: - name: main image: myregistry.local/backend:0.15.9 args: [yiic, cron-foo] restartPolicy: OnFailure
205. N Deployment Service Job CronJob
206. Volume
207. Volume Volume Storage Class Access Type Size + + =
208. Volume Volume Storage Class Access Type Size + + = ReadOnly Many ReadWrite Once ReadWrite Many
209. Volume Volume Storage Class Access Type Size + + = normal slow fast Admin defined Storage Classes
210. Volume Volume Storage Class Access Type Size + + = AWSElasticBlockStore GCEPersistentDisk AzureFile FC (Fibre Channel) AzureDisk FlexVolume Flocker iSCSI NFS RBD Cinder CephFS Glusterfs Quobyte Volumes VsphereVolume HostPath VMware Photon ScaleIO Volumes Portworx Volumes normal slow fast Admin defined Storage Classes
211. Volume Volume Storage Class Access Type Size + + = AWSElasticBlockStore GCEPersistentDisk AzureFile FC (Fibre Channel) AzureDisk FlexVolume Flocker iSCSI NFS RBD Cinder CephFS Glusterfs Quobyte Volumes VsphereVolume HostPath VMware Photon ScaleIO Volumes Portworx Volumes normal slow fast Admin defined Storage Classes Using Interface Implementation
212. Pod Volume
213. N ReplicaSet Pod Volume
214. N ReplicaSet Pod Volume ...
215. N ReplicaSet Pod Volume Volume ...
216. N N Deployment ReplicaSet Pod Volume Volume ... Volume ...
217. Volume Volume ... Volume ... N N ReadWriteOnce ReadWriteOnce ReadWriteOnce Deployment ReplicaSet Pod
218. StatefulSet
219. StatefulSet 0 = N 1 N-1 ...
220. foo 3 Deployment
221. foo 3 Deployment foo-khku8 foo-nacti foo-z9gth
222. bar foo 3 Deployment 3 StatefulSet foo-khku8 foo-nacti foo-z9gth
223. bar foo 3 Deployment 3 StatefulSet foo-khku8 foo-nacti foo-z9gth bar-0 bar-1 bar-2
224. Volume ... N Deployment
225. Volume ... N ... ... N StatefulSet Deployment
226. N Deployment StatefulSet Service Job CronJob N
227. foo bar foo N bar N Ingress
228. foo bar Internet foo N bar N Ingress
229. foo bar Internet foo N bar N Ingress public ip public ip
230. foo bar foo N bar N Ingress foo bar Internet foo N bar N public ip public ip
231. foo bar foo N bar N Ingress foo bar Internet foo N bar N public ip public ip Ingress
232. foo bar foo N bar N Ingress foo bar Internet foo N bar N public ip public ip Ingress public ip
233. foo bar foo N bar N Ingress foo bar Internet foo N bar N public ip public ip Ingress public ip Internet
234. Ingress Ingress Service + = Hostname Path + + TLS
235. apiVersion: extensions/v1beta1 kind: Ingress metadata: name: main spec: rules: - host: foo.bar.com http: paths: - path: /assets backend: serviceName: frontend servicePort: 80 - path: / backend: serviceName: backend servicePort: 80
236. apiVersion: extensions/v1beta1 kind: Ingress metadata: name: main spec: rules: - host: foo.bar.com http: paths: - path: /assets backend: serviceName: frontend servicePort: 80 - path: / backend: serviceName: backend servicePort: 80
237. apiVersion: extensions/v1beta1 kind: Ingress metadata: name: main spec: rules: - host: foo.bar.com http: paths: - path: /assets backend: serviceName: frontend servicePort: 80 - path: / backend: serviceName: backend servicePort: 80
238. apiVersion: extensions/v1beta1 kind: Ingress metadata: name: main spec: rules: - host: foo.bar.com http: paths: - path: /assets backend: serviceName: frontend servicePort: 80 - path: / backend: serviceName: backend servicePort: 80
239. apiVersion: extensions/v1beta1 kind: Ingress metadata: name: main spec: rules: - host: foo.bar.com http: paths: - path: /assets backend: serviceName: frontend servicePort: 80 - path: / backend: serviceName: backend servicePort: 80
240. Ingress Using Interface Implementation
241. Ingress Using Interface Implementation Nginx GCE Ingress Controllers
242. N Deployment StatefulSet Service Ingress Job CronJob N
243. 4. Service Discovery 1. Logging 2. Metrics 3. Supervision 6. (Auto) Scaling 7. CI/CD 8. Vendor lock-in Вызовы 5. Infrastructure Automation
244. 4. Service Discovery 3. Supervision 6. (Auto) Scaling 7. CI/CD 8. Vendor lock-in Вызовы 5. Infrastructure Automation 1. Logging 2. Metrics
245. 4. Service Discovery 6. (Auto) Scaling 7. CI/CD 8. Vendor lock-in Вызовы 5. Infrastructure Automation 1. Logging 2. Metrics 3. Supervision
246. 6. (Auto) Scaling 7. CI/CD 8. Vendor lock-in Вызовы 5. Infrastructure Automation 1. Logging 2. Metrics 3. Supervision 4. Service Discovery
247. 6. (Auto) Scaling 7. CI/CD 8. Vendor lock-in Вызовы 1. Logging 2. Metrics 3. Supervision 4. Service Discovery 5. Infrastructure Automation
248. 7. CI/CD 8. Vendor lock-in Вызовы 1. Logging 2. Metrics 3. Supervision 4. Service Discovery 5. Infrastructure Automation 6. (Auto) Scaling
249. 8. Vendor lock-in Вызовы 1. Logging 2. Metrics 3. Supervision 4. Service Discovery 5. Infrastructure Automation 6. (Auto) Scaling 7. CI/CD
250. Вызовы 1. Logging 2. Metrics 3. Supervision 4. Service Discovery 5. Infrastructure Automation 6. (Auto) Scaling 7. CI/CD 8. Vendor lock-in
252. Часть 1: Предыстория 10 минут Часть 2: Kubernetes 30 минут Часть 3: Наш опыт 20 минут
253. Маленький проект на Bare metal
254. Маленький проект на Bare metal hv-1 hv-2 hv-3
255. Маленький проект на Bare metal hv-1 hv-2 hv-3
256. Маленький проект на Bare metal hv-1 hv-2 hv-3
257. Маленький проект на Bare metal hv-1 hv-2 hv-3
258. Маленький проект на Bare metal hv-1 hv-2 hv-3
259. Маленький проект на Bare metal hv-1 hv-2 hv-3 kube-master-1 kube-master-2 kube-master-3
260. Маленький проект на Bare metal hv-1 hv-2 hv-3 kube-master-1 kube-master-2 kube-master-3 kube-node-1 kube-node-2 kube-node-3
261. Маленький проект на Bare metal hv-1 hv-2 hv-3 kube-master-1 kube-master-2 kube-master-3 kube-node-1 kube-node-2 kube-node-3 kube-front-1 kube-front-2 kube-front-3
262. Маленький проект на Bare metal hv-1 hv-2 hv-3 kube-master-1 kube-master-2 kube-master-3 kube-node-1 kube-node-2 kube-node-3 kube-front-1 kube-front-2 kube-front-3 ... ... ...
263. Маленький проект на Bare metal hv-1 hv-2 hv-3 kube-node-1 kube-front-1 kube-node-2 kube-front-2 kube-node-3 kube-front-3 gitlab vpn kube-master-1 kube-master-2 kube-master-3 nfs ... ... ... ...
264. Что мы пока не размещаем в Kubernetes?
265. Что мы пока не размещаем в Kubernetes? 1. High IO MySQL, MongoDB, PostgreSQL, Cassandra
266. Что мы пока не размещаем в Kubernetes? * Durable (non-shared) local storage management https://github.com/kubernetes/features/issues/121 1. High IO * MySQL, MongoDB, PostgreSQL, Cassandra
267. Что мы пока не размещаем в Kubernetes? * Durable (non-shared) local storage management https://github.com/kubernetes/features/issues/121 1. High IO * MySQL, MongoDB, PostgreSQL, Cassandra 2. Cloud non native cluster leader election, fencing
268. Что мы пока не размещаем в Kubernetes? * Durable (non-shared) local storage management https://github.com/kubernetes/features/issues/121 1. High IO * MySQL, MongoDB, PostgreSQL, Cassandra 2. Cloud non native cluster leader election, fencing … а также:
269. Что мы пока не размещаем в Kubernetes? * Durable (non-shared) local storage management https://github.com/kubernetes/features/issues/121 1. High IO * MySQL, MongoDB, PostgreSQL, Cassandra 2. Cloud non native cluster leader election, fencing 3. Страх “большие” базы … а также:
270. Что мы пока не размещаем в Kubernetes? * Durable (non-shared) local storage management https://github.com/kubernetes/features/issues/121 1. High IO * MySQL, MongoDB, PostgreSQL, Cassandra 2. Cloud non native cluster leader election, fencing 3. Страх “большие” базы 4. Лень VPN server, NFS server … а также:
271. Что мы пока не размещаем в Kubernetes? * Durable (non-shared) local storage management https://github.com/kubernetes/features/issues/121 1. High IO * MySQL, MongoDB, PostgreSQL, Cassandra 2. Cloud non native cluster leader election, fencing 3. Страх “большие” базы 4. Лень VPN server, NFS server 5. Инерция Gitlab, Ceph OSD … а также:
272. Как мы делаем Ingress
273. Как мы делаем Ingress kube-front-1 kube-front-2 kube-front-3
274. Как мы делаем Ingress kube-front-1 kube-front-2 kube-front-3 Nginx Ingress Controller Nginx Ingress Controller Nginx Ingress Controller
275. Как мы делаем Ingress kube-front-1 kube-front-2 kube-front-3 Nginx Ingress Controller Nginx Ingress Controller Nginx Ingress Controller Highly Available Public IP 3 x
276. Как мы делаем Ingress kube-front-1 kube-front-2 kube-front-3 Nginx Ingress Controller Nginx Ingress Controller Nginx Ingress Controller Highly Available Public IP 3 x VRRP VRRP VRRP
277. Как мы делаем Ingress kube-front-1 kube-front-2 kube-front-3 Nginx Ingress Controller Nginx Ingress Controller Nginx Ingress Controller Highly Available Public IP 3 x Pacemaker Pacemaker Pacemaker
278. Nginx Ingress Controller Nginx Ingress Controller Nginx Ingress Controller Как мы делаем Ingress kube-front-1 kube-front-2 kube-front-3 Highly Available Public IP 3 x DNS Load Balancing Pacemaker Pacemaker Pacemaker
279. git build test release operate Как мы делаем Continuous delivery
281. foo bar baz nats mongo
282. foo bar baz nats mongo production staging testing
283. foo bar baz nats mongo production staging testing foo foo foo
284. foo bar baz nats mongo production staging testing foo bar baz nats mongo foo bar baz nats mongo foo bar baz nats mongo
285. git build test release operate Dockerfile .kube/ backend.yaml frontend.yaml cron.yaml
286. git build test release operate Dockerfile .kube/ backend.yaml frontend.yaml cron.yaml docker build Docker- image
287. git build test release operate Dockerfile .kube/ backend.yaml frontend.yaml cron.yaml docker build Docker- image Docker registry
288. git build test release operate Dockerfile .kube/ backend.yaml frontend.yaml cron.yaml docker build Docker- image Docker registry Docker- image tests +
289. git build test release operate Dockerfile .kube/ backend.yaml frontend.yaml cron.yaml docker build Docker- image Docker registry Docker- image tests + kubectl
290. git build test release operate Dockerfile .kube/ backend.yaml frontend.yaml cron.yaml docker build Docker- image Docker registry Docker- image tests + kubectl
291. .kube/ backend.yaml frontend.yaml cron.yaml Docker- image Docker registry Docker- image tests + kubectl git build test release operate Dockerfile docker build
292. .kube/ backend.yaml frontend.yaml cron.yaml Docker- image Docker registry Docker- image tests + kubectl git build test release operate Dappfile build
293. Docker- image Docker registry Docker- image tests + git build test release operate Dappfile .kube/ backend.yaml frontend.yaml cron.yaml kubectl build
294. Docker- image Docker registry Docker- image tests + git build test release operate Dappfile .kube/ backend.yaml frontend.yaml cron.yaml build
295. Docker- image Docker registry Docker- image tests + git build test release operate Dappfile .kube/ backend.yaml frontend.yaml cron.yaml deploy build
296. Мы хотим сделать dapp действительно крутым и полезным Open Source-решением Нам нужен ВАШ опыт использования Будем рады любой помощи Поставьте звёздочку ;) Обещаем очень быстро отвечать на все issues и очень быстро исправлять все баги. Для нас dapp — это приоритетный проект. Если есть желание помочь с переводом документации на английский или с её доработкой, да или вообще помочь — будем очень рады и ждем pull-реквесты. Мы старались. github.com/flant/dapp
297. Собираем Docker- образы быстро и удобно habrahabr.ru/company/flant/blog/324274/
298. Инфраструктура НЕбольшого проекта
299. Инфраструктура НЕбольшого проекта
300. Инфраструктура НЕбольшого проекта Storage Ceph AWS GCE
301. Инфраструктура НЕбольшого проекта Storage Ceph AWS GCE Ingress
302. Инфраструктура НЕбольшого проекта Ingress Highly Available VM Storage Ceph AWS GCE
303. Инфраструктура НЕбольшого проекта Ingress Highly Available VM IOPS VM Storage Ceph AWS GCE
304. Frontend DB Backend memcached Cron Было
305. Highly Available VM IOPS VM Frontend DB Backend memcached Cron Было Стало Storage Ceph AWS GCE
306. Highly Available VM IOPS VM Frontend DB Backend memcached Cron Было Стало backend 3 Storage Ceph AWS GCE
307. Highly Available VM IOPS VM Frontend DB Backend memcached Cron Было Стало backend 3 DB Storage Ceph AWS GCE
308. Highly Available VM IOPS VM Frontend DB Backend memcached Cron Было Стало backend 3 DB cron-foo cron-bar cron-baz Storage Ceph AWS GCE
309. Highly Available VM IOPS VM Frontend DB Backend memcached Cron Было Стало backend 3 DB cron-foo cron-bar cron-baz frontend Storage Ceph AWS GCE
310. Highly Available VM IOPS VM Frontend DB Backend memcached Cron Было Стало backend 3 DB cron-foo cron-bar cron-baz frontend memcached 3 Storage Ceph AWS GCE
311. Highly Available VM IOPS VM DB frontend backend 3 memcached 3 cron-foo cron-bar cron-baz Frontend DB Backend memcached Cron Было Стало Files Storage Ceph AWS GCE
312. Highly Available VM IOPS VM Storage Ceph AWS GCE DB frontend backend 3 memcached 3 cron-foo cron-bar cron-baz Frontend DB Backend memcached Cron Было Стало Files Background Jobs Redis Redis 3 Background Jobs 3
313. Highly Available VM IOPS VM DB frontend backend 3 memcached 3 cron-foo cron-bar cron-baz Frontend DB Backend memcached Cron Было Стало Files Background Jobs Redis Redis 3 Background Jobs 3 Sphinx 2 Sphinx Storage Ceph AWS GCE
314. Highly Available VM IOPS VM frontend backend 3 memcached 3 cron-foo cron-bar cron-baz Frontend DB Backend memcached Cron Было Стало Files Background Jobs Redis Redis 3 Background Jobs 3 Sphinx 2 Sphinx Storage Ceph AWS GCE DB 3
315. Простая Цирк Облачная Хорошая Идеальная Микросервисная
316. Хорошая Облачная Хорошая Идеальная Микросервисная Цирк
317. Хорошая Облачная Хорошая Идеальная Микросервисная Хорошая
318. Хорошая Хорошая Хорошая Хорошая Идеальная Микросервисная
319. Хорошая Хорошая Хорошая Идеальная Микросервисная Хорошая
320. Хорошая Хорошая Хорошая Хорошая Идеальная Микросервисная
321. Хорошая Хорошая Хорошая Хорошая Идеальная Микросервисная
322. Наш блог на Хабрахабре habrahabr.ru/company/flant/ Тут мы рассказываем о нашем опыте и актуальных технологиях. Подписывайтесь, чтобы быть в курсе!
323. Наши Docker-проекты github.com/flant/dapp github.com/flant/docker_penetration_experiment github.com/flant/pam_docker github.com/flant/php_fpm_docker github.com/flant/superhosting github.com/flant/buildizer Дмитрий Столяров dmitry.stolyarov@flant.ru linkedin.com/in/distol github.com/distol Всем спасибо! Наш блог на Хабрахабре habrahabr.ru/company/flant/