Наш опыт с Kubernetes в небольших проектах

by flant

Slide 1

Slide 1 text

Дмитрий Столяров CTO & Co-founder v2 Наш опыт с Kubernetes в небольших проектах

Slide 2

Slide 2 text

# whoami dmitry.stolyarov # hostname -d flant.ru # cat /etc/motd Наш опыт с Kubernetes в небольших проектах Привет!

Slide 3

Slide 3 text

24×7×365 L1/L2/L3/+ DevOps SLA

Slide 4

Slide 4 text

Небольшие проекты?

Slide 5

Slide 5 text

Небольшие проекты? Kubernetes масштабируется до 5 000 nodes & 150 000 pods

Slide 6

Slide 6 text

Небольшие проекты? Маленькие < 50 nodes < 1 500 pods Большие > 500 nodes > 15 000 pods Средние 50–500 nodes 1500–15 000 pods Kubernetes масштабируется до 5 000 nodes & 150 000 pods

Slide 7

Slide 7 text

Небольшие проекты? Маленькие < 50 nodes < 1 500 pods Большие > 500 nodes > 15 000 pods Средние 50–500 nodes 1500–15 000 pods Kubernetes масштабируется до 5 000 nodes & 150 000 pods Небольшие

Slide 8

Slide 8 text

Часть 1: Предыстория 10 минут Часть 2: Kubernetes 30 минут Часть 3: Наш опыт 20 минут

Slide 9

Slide 9 text

Часть 1: Предыстория 10 минут Часть 2: Kubernetes 30 минут Часть 3: Наш опыт 20 минут

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

DB Backend

Slide 13

Slide 13 text

DB Backend

Slide 14

Slide 14 text

DB Backend Cron

Slide 15

Slide 15 text

DB Backend memcached Cron

Slide 16

Slide 16 text

Frontend DB Backend memcached Cron

Slide 17

Slide 17 text

Frontend DB Backend memcached Files Cron

Slide 18

Slide 18 text

Frontend DB Backend memcached Files Redis Background Jobs Cron

Slide 19

Slide 19 text

Frontend DB Backend memcached Files Sphinx Background Jobs Redis Cron

Slide 20

Slide 20 text

Frontend DB Backend memcached Files Sphinx Background Jobs Redis Cron Какие проблемы?

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

Frontend Backend memcached Files Sphinx Background Jobs Redis Cron DB master

Slide 23

Slide 23 text

Frontend Backend memcached Files Sphinx Background Jobs Redis Cron DB master DB slave

Slide 24

Slide 24 text

Redis Frontend Sphinx Backend memcached Background Jobs Cron DB master DB slave Files Backend Frontend memcached

Slide 25

Slide 25 text

Frontend Backend memcached Sphinx Background Jobs Redis Cron DB master DB slave Frontend memcached Files Backend

Slide 26

Slide 26 text

Backend memcached Files Sphinx Background Jobs Redis Cron DB master DB slave Backend memcached Frontend Frontend

Slide 27

Slide 27 text

memcached Files Sphinx Background Jobs Redis Cron DB master DB slave memcached Frontend Frontend Backend Backend

Slide 28

Slide 28 text

Frontend Backend memcached Files Sphinx Cron DB slave Frontend Backend memcached Redis DB master Background Jobs

Slide 29

Slide 29 text

Frontend Backend Sphinx memcached Cron DB master DB slave Frontend Backend Background Jobs Redis memcached Files nfs-сервер

Slide 30

Slide 30 text

Frontend Backend Sphinx memcached Cron DB master DB slave Frontend Backend Background Jobs Files nfs-сервер memcached Redis

Slide 31

Slide 31 text

Frontend Backend Sphinx memcached DB master DB slave Frontend Backend Redis Files nfs-сервер memcached Cron Background Jobs

Slide 32

Slide 32 text

Frontend Backend Sphinx memcached DB master DB slave Frontend Backend Redis Files nfs-сервер memcached Cron Background Jobs New Project Backend

Slide 33

Slide 33 text

Frontend Backend Sphinx memcached DB master DB slave Frontend Backend Redis Files nfs-сервер memcached Cron Background Jobs New Project Backend Какие проблемы?

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

Cloud

Slide 36

Slide 36 text

Frontend Frontend Backend Backend Backend Backend Backend Backend DB master DB slave Files Memcached Memcached Memcached Redis Cron Background Jobs Sphinx Sphinx

Slide 37

Slide 37 text

Frontend Frontend Backend Backend Backend Backend Backend Backend DB master DB slave Files Memcached Memcached Memcached Redis Cron Background Jobs LB Sphinx Sphinx

Slide 38

Slide 38 text

Frontend Frontend Backend Backend Backend Backend Backend Backend DB master DB slave Files Memcached Memcached Memcached Redis Cron Background Jobs LB Sphinx Sphinx Какие проблемы?

Slide 39

Slide 39 text

Frontend Frontend Backend Backend Backend Backend Backend Backend DB master DB slave Sphinx Sphinx Frontend Frontend Backend Backend Backend Backend Backend Backend DB master DB slave Sphinx Sphinx LB Memcached Memcached Memcached Redis Cron Background Jobs S3 Files

Slide 40

Slide 40 text

Frontend Frontend Sphinx Sphinx Frontend Frontend Sphinx Sphinx DB master DB slave DB master DB slave S3 Files LB Memcached Memcached Memcached Redis Cron Background Jobs Backend-1 Backend-2 Backend-N ... LB

Slide 41

Slide 41 text

Frontend Frontend Sphinx Sphinx Frontend Frontend Sphinx Sphinx DB master DB slave DB master DB slave LB S3 Files Memcached Memcached Memcached Redis Cron Background Jobs Backend-1 Backend-2 Backend-N ...

Slide 42

Slide 42 text

Backend-1 Backend-2 Backend-N ... Frontend Frontend Sphinx Sphinx Frontend Frontend Sphinx Sphinx LB S3 Files Memcached Memcached Memcached Redis Cron Background Jobs DB-1 DB-2 DB-N ...

Slide 43

Slide 43 text

JOB-1 JOB-2 JOB-N ... Sphinx-1 Spninx-N ... LB DB-1 DB-2 DB-N ... Backend-1 Backend-2 Backend-N ... Frontend Frontend Frontend Frontend MQ-1 MQ-2 MQ-N ... S3 Files Memcached Memcached Memcached Cron

Slide 44

Slide 44 text

LB Frontend-1 Frontend-N ... LB Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... Backend-1 Backend-2 Backend-N ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron

Slide 45

Slide 45 text

Микросервис S3 Files Cron MQ-1 MQ-2 MQ-N ... JOB-1 JOB-2 JOB-N ... DB-1 DB-2 DB-N ... LB Backend 1 Backend 2 Backend N ... Sphinx-1 Spрinx-N ... Memcached 1 Memcached N ... Frontend 1 Frontend N ...

Slide 46

Slide 46 text

Микросервис FOO Message Bus Микросервис BAR Микросервис BAZ ... LB Frontend-1 Frontend-N ...

Slide 47

Slide 47 text

Простая Цирк Облачная Хорошая Идеальная Микросервисная

Slide 48

Slide 48 text

Frontend 1 Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... Backend 1 Backend 2 ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron LB LB Backend N

Slide 49

Slide 49 text

Frontend 1 Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... Backend 1 Backend 2 ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB Backend N

Slide 50

Slide 50 text

Slide 51

Slide 51 text

Slide 52

Slide 52 text

Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery Memcached 1 Memcached N ... ... Backend 1 Backend 2 ... Backend N Frontend 1 Frontend N 5. Infrastructure Automation

Slide 53

Slide 53 text

Frontend 1 Frontend N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery Memcached 1 Memcached N ... Backend 1 Backend 2 ... Backend N 5. Infrastructure Automation

Slide 54

Slide 54 text

Frontend 1 Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... Backend 1 Backend 2 ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB Backend N 3. Supervision 4. Service Discovery DB-1 DB-2 DB-N ... 5. Infrastructure Automation

Slide 55

Slide 55 text

Slide 56

Slide 56 text

5. Infrastructure Automation Frontend 1 Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... Backend 1 Backend 2 ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB Backend N 3. Supervision 4. Service Discovery 6. (Auto) Scaling

Slide 57

Slide 57 text

6. (Auto) Scaling 5. Infrastructure Automation Frontend 1 Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery 7. CI/CD Backend 1 Backend 2 ... Backend N

Slide 58

Slide 58 text

Slide 59

Slide 59 text

6. (Auto) Scaling 5. Infrastructure Automation Frontend 1 Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery 7. CI/CD Backend 1 Backend 2 ... Backend N JOB-1 JOB-2 JOB-N ...

Slide 60

Slide 60 text

6. (Auto) Scaling 5. Infrastructure Automation Frontend 1 Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... MQ-1 MQ-2 MQ-N ... S3 Files 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery 7. CI/CD Backend 1 Backend 2 ... Backend N JOB-1 JOB-2 JOB-N ... Cron

Slide 61

Slide 61 text

6. (Auto) Scaling 5. Infrastructure Automation Frontend 1 Frontend N ... Memcached 1 Memcached N ... DB-1 DB-2 DB-N ... MQ-1 MQ-2 MQ-N ... S3 Files 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery 7. CI/CD Backend 1 Backend 2 ... Backend N JOB-1 JOB-2 JOB-N ... Cron Sphinx-1 Spninx-N ...

Slide 62

Slide 62 text

6. (Auto) Scaling 5. Infrastructure Automation Memcached 1 Memcached N ... DB-1 DB-2 DB-N ... MQ-1 MQ-2 MQ-N ... S3 Files 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery 7. CI/CD Backend 1 Backend 2 ... Backend N JOB-1 JOB-2 JOB-N ... Cron Sphinx-1 Spninx-N ... Frontend 1 Frontend N ...

Slide 63

Slide 63 text

Лучшие практики Continuous Delivery c Docker – 2016 habrahabr.ru/company/flant/blog/322686/

Slide 64

Slide 64 text

Главный паттерн деплоя Старая версия работает 1

Slide 65

Slide 65 text

Главный паттерн деплоя Старая версия работает 1 Новая версия “прогревается” 2

Slide 66

Slide 66 text

Главный паттерн деплоя Старая версия работает 1 Новая версия “прогревается” 2 Переключили трафик 3

Slide 67

Slide 67 text

Главный паттерн деплоя Старая версия работает Новая версия “прогревается” Переключили трафик Старая версия остановлена 1 2 3 4

Slide 68

Slide 68 text

Slide 69

Slide 69 text

7. CI/CD 6. (Auto) Scaling 5. Infrastructure Automation Frontend 1 Frontend N ... Memcached 1 Memcached N ... Sphinx-1 Spninx-N ... DB-1 DB-2 DB-N ... JOB-1 JOB-2 JOB-N ... MQ-1 MQ-2 MQ-N ... S3 Files Cron 1. Logging 2. Metrics LB LB 3. Supervision 4. Service Discovery Backend 1 Backend 2 ... Backend N 8. Vendor lock-in

Slide 70

Slide 70 text

Slide 71

Slide 71 text

No content

Slide 72

Slide 72 text

Часть 1: Предыстория 10 минут Часть 2: Kubernetes 30 минут Часть 3: Наш опыт 20 минут

Slide 73

Slide 73 text

Архитектура Kubernetes

Slide 74

Slide 74 text

master Архитектура Kubernetes

Slide 75

Slide 75 text

node-1 node-2 ... master Архитектура Kubernetes

Slide 76

Slide 76 text

node-1 node-2 ... master Архитектура Kubernetes Docker Docker

Slide 77

Slide 77 text

node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet

Slide 78

Slide 78 text

node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy

Slide 79

Slide 79 text

node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy

Slide 80

Slide 80 text

node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server

Slide 81

Slide 81 text

node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server

Slide 82

Slide 82 text

node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server etcd

Slide 83

Slide 83 text

node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server etcd

Slide 84

Slide 84 text

node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server etcd Scheduler

Slide 85

Slide 85 text

node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager

Slide 86

Slide 86 text

node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager kubectl

Slide 87

Slide 87 text

node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager kubectl YAML (Declarative DSL)

Slide 88

Slide 88 text

node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager kubectl YAML (Declarative DSL)

Slide 89

Slide 89 text

node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager kubectl YAML (Declarative DSL)

Slide 90

Slide 90 text

node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager kubectl YAML (Declarative DSL)

Slide 91

Slide 91 text

Набор строительных блоков

Slide 92

Slide 92 text

Container

Slide 93

Slide 93 text

Container Image Cmd

Slide 94

Slide 94 text

Container Image Cmd 1. Logging

Slide 95

Slide 95 text

Container Image Cmd 3. Supervision 1. Logging

Slide 96

Slide 96 text

Pod

Slide 97

Slide 97 text

Pod Container 1 Container 2 ...

Slide 98

Slide 98 text

Pod Container 1 Container 2 ... localhost eth0

Slide 99

Slide 99 text

Pod Container 1 Container 2 ... localhost eth0 volume1 volume2

Slide 100

Slide 100 text

Pod Container 1 Container 2 ... localhost eth0 volume1 volume2 tier: backend version: 0.15.7

Slide 101

Slide 101 text

Label & Selector

Slide 102

Slide 102 text

Label & Selector app: foo

Slide 103

Slide 103 text

Label & Selector app: foo tier: backend version: 0.15.7

Slide 104

Slide 104 text

Label & Selector app: foo tier: backend version: 0.15.7 partition: customer_a

Slide 105

Slide 105 text

Label & Selector app: foo tier: backend version: 0.15.7 partition: customer_a app in (foo, bar)

Slide 106

Slide 106 text

Label & Selector app: foo tier: backend version: 0.15.7 partition: customer_a app in (foo, bar) tier == backend

Slide 107

Slide 107 text

Label & Selector app: foo tier: backend version: 0.15.7 partition: customer_a app in (foo, bar) tier == backend partition != customer_a

Slide 108

Slide 108 text

Pod Container 1 Container 2 ... localhost eth0 volume1 volume2 tier: backend version: 0.15.7

Slide 109

Slide 109 text

Pod Container 1 Container 2 ... localhost eth0 volume1 volume2 tier: backend 5. Infrastructure Automation version: 0.15.7

Slide 110

Slide 110 text

3. Supervision Pod Container 1 Container 2 ... localhost eth0 volume1 volume2 tier: backend 5. Infrastructure Automation version: 0.15.7

Slide 111

Slide 111 text

3. Supervision Pod Container 1 Container 2 ... localhost eth0 volume1 volume2 tier: backend 5. Infrastructure Automation 4. Service Discovery version: 0.15.7

Slide 112

Slide 112 text

node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager kubectl YAML (Declarative DSL)

Slide 113

Slide 113 text

node-1 node-2 ... master Архитектура Kubernetes Docker Docker kubelet kubelet kube-proxy kube-proxy API Server etcd Scheduler Сontroller -manager kubectl YAML (Declarative DSL)

Slide 114

Slide 114 text

apiVersion: v1 kind: Pod metadata: name: manual-bash spec: containers: - name: bash image: ubuntu:16.04 command: bash args: [-c, “while true; do sleep 1; date; done”]

Slide 115

Slide 115 text

apiVersion: v1 kind: Pod metadata: name: manual-bash spec: containers: - name: bash image: ubuntu:16.04 command: bash args: [-c, “while true; do sleep 1; date; done”]

Slide 116

Slide 116 text

apiVersion: v1 kind: Pod metadata: name: manual-bash spec: containers: - name: bash image: ubuntu:16.04 command: bash args: [-c, “while true; do sleep 1; date; done”]

Slide 117

Slide 117 text

apiVersion: v1 kind: Pod metadata: name: manual-bash spec: containers: - name: bash image: ubuntu:16.04 command: bash args: [-c, “while true; do sleep 1; date; done”]

Slide 118

Slide 118 text

apiVersion: v1 kind: Pod metadata: name: manual-bash spec: containers: - name: bash image: ubuntu:16.04 command: bash args: [-c, “while true; do sleep 1; date; done”]

Slide 119

Slide 119 text

apiVersion: v1 kind: Pod metadata: name: manual-bash spec: containers: - name: bash image: ubuntu:16.04 command: bash args: [-c, “while true; do sleep 1; date; done”]

Slide 120

Slide 120 text

apiVersion: v1 kind: Pod metadata: name: manual-backend labels: tier: backend spec: containers: - name: web image: myregistry.local/backend:0.15.9 command: /usr/bin/php-fpm

Slide 121

Slide 121 text

apiVersion: v1 kind: Pod metadata: name: manual-backend labels: tier: backend spec: containers: - name: web image: myregistry.local/backend:0.15.9 command: /usr/bin/php-fpm

Slide 122

Slide 122 text

apiVersion: v1 kind: Pod metadata: name: manual-backend labels: tier: backend spec: containers: - name: web image: myregistry.local/backend:0.15.9 command: /usr/bin/php-fpm

Slide 123

Slide 123 text

ReplicaSet

Slide 124

Slide 124 text

ReplicaSet N

Slide 125

Slide 125 text

No content

Slide 126

Slide 126 text

Current Ready Status Desired 2 2 2

Slide 127

Slide 127 text

2 2 2 Current Ready Status Desired 3 2 2

Slide 128

Slide 128 text

2 2 2 3 2 2 Current Ready Status Desired 3 3 2

Slide 129

Slide 129 text

2 2 2 3 2 2 3 3 2 Current Ready Status Desired 3 3 3

Slide 130

Slide 130 text

2 2 2 3 2 2 3 3 2 3 3 3 Current Ready Status Desired 1 3 3

Slide 131

Slide 131 text

2 2 2 3 2 2 3 3 2 3 3 3 1 3 3 Current Ready Status Desired 1 3 1

Slide 132

Slide 132 text

2 2 2 3 2 2 3 3 2 3 3 3 1 3 3 1 3 1 Current Ready Status Desired 1 1 1

Slide 133

Slide 133 text

ReplicaSet N 6. (Auto) Scaling 3. Supervision

Slide 134

Slide 134 text

apiVersion: extensions/v1beta1 kind: ReplicaSet metadata: name: backend spec: replicas: 3 selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.7 command: php-fpm

Slide 135

Slide 135 text

Slide 136

Slide 136 text

Slide 137

Slide 137 text

Slide 138

Slide 138 text

Slide 139

Slide 139 text

Slide 140

Slide 140 text

Slide 141

Slide 141 text

Slide 142

Slide 142 text

Slide 143

Slide 143 text

Deployment

Slide 144

Slide 144 text

Deployment N

Slide 145

Slide 145 text

Deployment N + History 0 0 ...

Slide 146

Slide 146 text

Deployment N + History 0 0 ... + Deploy

Slide 147

Slide 147 text

Deployment N + History 0 0 ... + Deploy Pod = N

Slide 148

Slide 148 text

3 0.15.7

Slide 149

Slide 149 text

3 0.15.7

Slide 150

Slide 150 text

3 0.15.7 0 0.15.9

Slide 151

Slide 151 text

3 0.15.7 1 0.15.9

Slide 152

Slide 152 text

3 0.15.7 1 0.15.9

Slide 153

Slide 153 text

3 0.15.7 1 0.15.9

Slide 154

Slide 154 text

2 0.15.7 1 0.15.9

Slide 155

Slide 155 text

2 0.15.7 1 0.15.9

Slide 156

Slide 156 text

2 0.15.7 1 0.15.9

Slide 157

Slide 157 text

2 0.15.7 2 0.15.9

Slide 158

Slide 158 text

1 0.15.7 2 0.15.9

Slide 159

Slide 159 text

1 0.15.7 3 0.15.9

Slide 160

Slide 160 text

0 0.15.7 3 0.15.9

Slide 161

Slide 161 text

0 0.15.7 3 0.15.9

Slide 162

Slide 162 text

0 0.15.7 3 0.15.9 0 0.16.3

Slide 163

Slide 163 text

0 0.15.7 2 0.15.9 1 0.16.3

Slide 164

Slide 164 text

0 0.15.7 1 0.15.9 2 0.16.3

Slide 165

Slide 165 text

0 0.15.7 0 0.15.9 3 0.16.3

Slide 166

Slide 166 text

0 0.15.7 0 0.15.9 3 0.16.3

Slide 167

Slide 167 text

Deployment Pod + = + History Deploy N N 0 0 ... 7. CI/CD

Slide 168

Slide 168 text

apiVersion: apps/v1beta1 kind: Deployment metadata: name: backend spec: replicas: 3 selector: matchLabels: tier: backend template: metadata: labels: tier: backend spec: containers: - name: fpm image: myregistry.local/backend:0.15.9 command: php-fpm

Slide 169

Slide 169 text

Slide 170

Slide 170 text

N Deployment

Slide 171

Slide 171 text

Service DNS

Slide 172

Slide 172 text

Service DNS + Virtual IP

Slide 173

Slide 173 text

Service DNS Load Balancer + Virtual IP +

Slide 174

Slide 174 text

Selector Service DNS Load Balancer + Virtual IP + +

Slide 175

Slide 175 text

Selector Service Service DNS Load Balancer + = Virtual IP + +

Slide 176

Slide 176 text

No content

Slide 177

Slide 177 text

tier: backend tier: backend tier: backend tier: backend tier: backend tier: backend

Slide 178

Slide 178 text

tier: backend tier: backend tier: backend tier: backend tier: backend tier: backend version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.7

Slide 179

Slide 179 text

tier: backend tier: backend tier: backend tier: backend tier: backend tier: backend version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.9 version: 0.15.9

Slide 180

Slide 180 text

tier: backend tier: backend tier: backend tier: backend tier: backend tier: backend version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.9 version: 0.15.9 tier: db tier: db tier: db tier: sidekiq

Slide 181

Slide 181 text

Slide 182

Slide 182 text

tier: db tier: sidekiq tier: db tier: db tier: backend tier: backend tier: backend tier: backend tier: backend tier: backend version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.7 version: 0.15.9 version: 0.15.9 backend tier==backend

Slide 183

Slide 183 text

Slide 184

Slide 184 text

tier: backend version: 0.15.7 tier: db tier: sidekiq tier: db tier: db tier: backend tier: backend tier: backend version: 0.15.7 version: 0.15.7 version: 0.15.7 backend tier==backend tier==backend new-backend version==0.15.9 tier: backend version: 0.15.9 tier: backend version: 0.15.9

Slide 185

Slide 185 text

kind: Service apiVersion: v1 metadata: name: backend spec: selector: tier: backend ports: - protocol: TCP port: 80 targetPort: 8080

Slide 186

Slide 186 text

kind: Service apiVersion: v1 metadata: name: backend spec: selector: tier: backend ports: - protocol: TCP port: 80 targetPort: 8080

Slide 187

Slide 187 text

kind: Service apiVersion: v1 metadata: name: backend spec: selector: tier: backend ports: - protocol: TCP port: 80 targetPort: 8080

Slide 188

Slide 188 text

N Deployment Service

Slide 189

Slide 189 text

6. Job

Slide 190

Slide 190 text

6. Job Pod

Slide 191

Slide 191 text

6. Job Pod Successful Completion +

Slide 192

Slide 192 text

6. Job Pod Successful Completion Job + =

Slide 193

Slide 193 text

apiVersion: batch/v1 kind: Job metadata: name: migrate spec: template: metadata: name: migrate spec: containers: - name: migrate image: myregistry.local/backend:0.15.9 command: [yiic, migrate] restartPolicy: Never

Slide 194

Slide 194 text

Slide 195

Slide 195 text

Slide 196

Slide 196 text

N Deployment Service Job

Slide 197

Slide 197 text

CronJob

Slide 198

Slide 198 text

CronJob Job

Slide 199

Slide 199 text

Shedule * * * * * CronJob Job +

Slide 200

Slide 200 text

Shedule * * * * * CronJob Job CronJob + =

Slide 201

Slide 201 text

apiVersion: batch/v2alpha1 kind: CronJob metadata: name: cron-foo spec: schedule: "*/1 * * * *" jobTemplate: spec: template: spec: containers: - name: main image: myregistry.local/backend:0.15.9 args: [yiic, cron-foo] restartPolicy: OnFailure

Slide 202

Slide 202 text

Slide 203

Slide 203 text

Slide 204

Slide 204 text

Slide 205

Slide 205 text

N Deployment Service Job CronJob

Slide 206

Slide 206 text

Volume

Slide 207

Slide 207 text

Volume Volume Storage Class Access Type Size + + =

Slide 208

Slide 208 text

Volume Volume Storage Class Access Type Size + + = ReadOnly Many ReadWrite Once ReadWrite Many

Slide 209

Slide 209 text

Volume Volume Storage Class Access Type Size + + = normal slow fast Admin defined Storage Classes

Slide 210

Slide 210 text

Volume Volume Storage Class Access Type Size + + = AWSElasticBlockStore GCEPersistentDisk AzureFile FC (Fibre Channel) AzureDisk FlexVolume Flocker iSCSI NFS RBD Cinder CephFS Glusterfs Quobyte Volumes VsphereVolume HostPath VMware Photon ScaleIO Volumes Portworx Volumes normal slow fast Admin defined Storage Classes

Slide 211

Slide 211 text

Slide 212

Slide 212 text

Pod Volume

Slide 213

Slide 213 text

N ReplicaSet Pod Volume

Slide 214

Slide 214 text

N ReplicaSet Pod Volume ...

Slide 215

Slide 215 text

N ReplicaSet Pod Volume Volume ...

Slide 216

Slide 216 text

N N Deployment ReplicaSet Pod Volume Volume ... Volume ...

Slide 217

Slide 217 text

Volume Volume ... Volume ... N N ReadWriteOnce ReadWriteOnce ReadWriteOnce Deployment ReplicaSet Pod

Slide 218

Slide 218 text

StatefulSet

Slide 219

Slide 219 text

StatefulSet 0 = N 1 N-1 ...

Slide 220

Slide 220 text

foo 3 Deployment

Slide 221

Slide 221 text

foo 3 Deployment foo-khku8 foo-nacti foo-z9gth

Slide 222

Slide 222 text

bar foo 3 Deployment 3 StatefulSet foo-khku8 foo-nacti foo-z9gth

Slide 223

Slide 223 text

bar foo 3 Deployment 3 StatefulSet foo-khku8 foo-nacti foo-z9gth bar-0 bar-1 bar-2

Slide 224

Slide 224 text

Volume ... N Deployment

Slide 225

Slide 225 text

Volume ... N ... ... N StatefulSet Deployment

Slide 226

Slide 226 text

N Deployment StatefulSet Service Job CronJob N

Slide 227

Slide 227 text

foo bar foo N bar N Ingress

Slide 228

Slide 228 text

foo bar Internet foo N bar N Ingress

Slide 229

Slide 229 text

foo bar Internet foo N bar N Ingress public ip public ip

Slide 230

Slide 230 text

foo bar foo N bar N Ingress foo bar Internet foo N bar N public ip public ip

Slide 231

Slide 231 text

foo bar foo N bar N Ingress foo bar Internet foo N bar N public ip public ip Ingress

Slide 232

Slide 232 text

foo bar foo N bar N Ingress foo bar Internet foo N bar N public ip public ip Ingress public ip

Slide 233

Slide 233 text

foo bar foo N bar N Ingress foo bar Internet foo N bar N public ip public ip Ingress public ip Internet

Slide 234

Slide 234 text

Ingress Ingress Service + = Hostname Path + + TLS

Slide 235

Slide 235 text

apiVersion: extensions/v1beta1 kind: Ingress metadata: name: main spec: rules: - host: foo.bar.com http: paths: - path: /assets backend: serviceName: frontend servicePort: 80 - path: / backend: serviceName: backend servicePort: 80

Slide 236

Slide 236 text

Slide 237

Slide 237 text

Slide 238

Slide 238 text

Slide 239

Slide 239 text

Slide 240

Slide 240 text

Ingress Using Interface Implementation

Slide 241

Slide 241 text

Ingress Using Interface Implementation Nginx GCE Ingress Controllers

Slide 242

Slide 242 text

N Deployment StatefulSet Service Ingress Job CronJob N

Slide 243

Slide 243 text

4. Service Discovery 1. Logging 2. Metrics 3. Supervision 6. (Auto) Scaling 7. CI/CD 8. Vendor lock-in Вызовы 5. Infrastructure Automation

Slide 244

Slide 244 text

4. Service Discovery 3. Supervision 6. (Auto) Scaling 7. CI/CD 8. Vendor lock-in Вызовы 5. Infrastructure Automation 1. Logging 2. Metrics

Slide 245

Slide 245 text

4. Service Discovery 6. (Auto) Scaling 7. CI/CD 8. Vendor lock-in Вызовы 5. Infrastructure Automation 1. Logging 2. Metrics 3. Supervision

Slide 246

Slide 246 text

6. (Auto) Scaling 7. CI/CD 8. Vendor lock-in Вызовы 5. Infrastructure Automation 1. Logging 2. Metrics 3. Supervision 4. Service Discovery

Slide 247

Slide 247 text

6. (Auto) Scaling 7. CI/CD 8. Vendor lock-in Вызовы 1. Logging 2. Metrics 3. Supervision 4. Service Discovery 5. Infrastructure Automation

Slide 248

Slide 248 text

7. CI/CD 8. Vendor lock-in Вызовы 1. Logging 2. Metrics 3. Supervision 4. Service Discovery 5. Infrastructure Automation 6. (Auto) Scaling

Slide 249

Slide 249 text

8. Vendor lock-in Вызовы 1. Logging 2. Metrics 3. Supervision 4. Service Discovery 5. Infrastructure Automation 6. (Auto) Scaling 7. CI/CD

Slide 250

Slide 250 text

Вызовы 1. Logging 2. Metrics 3. Supervision 4. Service Discovery 5. Infrastructure Automation 6. (Auto) Scaling 7. CI/CD 8. Vendor lock-in

Slide 251

Slide 251 text

No content

Slide 252

Slide 252 text

Часть 1: Предыстория 10 минут Часть 2: Kubernetes 30 минут Часть 3: Наш опыт 20 минут

Slide 253

Slide 253 text

Маленький проект на Bare metal

Slide 254

Slide 254 text

Маленький проект на Bare metal hv-1 hv-2 hv-3

Slide 255

Slide 255 text

Маленький проект на Bare metal hv-1 hv-2 hv-3

Slide 256

Slide 256 text

Маленький проект на Bare metal hv-1 hv-2 hv-3

Slide 257

Slide 257 text

Маленький проект на Bare metal hv-1 hv-2 hv-3

Slide 258

Slide 258 text

Маленький проект на Bare metal hv-1 hv-2 hv-3

Slide 259

Slide 259 text

Маленький проект на Bare metal hv-1 hv-2 hv-3 kube-master-1 kube-master-2 kube-master-3

Slide 260

Slide 260 text

Маленький проект на Bare metal hv-1 hv-2 hv-3 kube-master-1 kube-master-2 kube-master-3 kube-node-1 kube-node-2 kube-node-3

Slide 261

Slide 261 text

Маленький проект на Bare metal hv-1 hv-2 hv-3 kube-master-1 kube-master-2 kube-master-3 kube-node-1 kube-node-2 kube-node-3 kube-front-1 kube-front-2 kube-front-3

Slide 262

Slide 262 text

Маленький проект на Bare metal hv-1 hv-2 hv-3 kube-master-1 kube-master-2 kube-master-3 kube-node-1 kube-node-2 kube-node-3 kube-front-1 kube-front-2 kube-front-3 ... ... ...

Slide 263

Slide 263 text

Маленький проект на Bare metal hv-1 hv-2 hv-3 kube-node-1 kube-front-1 kube-node-2 kube-front-2 kube-node-3 kube-front-3 gitlab vpn kube-master-1 kube-master-2 kube-master-3 nfs ... ... ... ...

Slide 264

Slide 264 text

Что мы пока не размещаем в Kubernetes?

Slide 265

Slide 265 text

Что мы пока не размещаем в Kubernetes? 1. High IO MySQL, MongoDB, PostgreSQL, Cassandra

Slide 266

Slide 266 text

Slide 267

Slide 267 text

Slide 268

Slide 268 text

Что мы пока не размещаем в Kubernetes? * Durable (non-shared) local storage management https://github.com/kubernetes/features/issues/121 1. High IO * MySQL, MongoDB, PostgreSQL, Cassandra 2. Cloud non native cluster leader election, fencing … а также:

Slide 269

Slide 269 text

Slide 270

Slide 270 text

Slide 271

Slide 271 text

Slide 272

Slide 272 text

Как мы делаем Ingress

Slide 273

Slide 273 text

Как мы делаем Ingress kube-front-1 kube-front-2 kube-front-3

Slide 274

Slide 274 text

Как мы делаем Ingress kube-front-1 kube-front-2 kube-front-3 Nginx Ingress Controller Nginx Ingress Controller Nginx Ingress Controller

Slide 275

Slide 275 text

Как мы делаем Ingress kube-front-1 kube-front-2 kube-front-3 Nginx Ingress Controller Nginx Ingress Controller Nginx Ingress Controller Highly Available Public IP 3 x

Slide 276

Slide 276 text

Как мы делаем Ingress kube-front-1 kube-front-2 kube-front-3 Nginx Ingress Controller Nginx Ingress Controller Nginx Ingress Controller Highly Available Public IP 3 x VRRP VRRP VRRP

Slide 277

Slide 277 text

Как мы делаем Ingress kube-front-1 kube-front-2 kube-front-3 Nginx Ingress Controller Nginx Ingress Controller Nginx Ingress Controller Highly Available Public IP 3 x Pacemaker Pacemaker Pacemaker

Slide 278

Slide 278 text

Nginx Ingress Controller Nginx Ingress Controller Nginx Ingress Controller Как мы делаем Ingress kube-front-1 kube-front-2 kube-front-3 Highly Available Public IP 3 x DNS Load Balancing Pacemaker Pacemaker Pacemaker

Slide 279

Slide 279 text

git build test release operate Как мы делаем Continuous delivery

Slide 280

Slide 280 text

No content

Slide 281

Slide 281 text

foo bar baz nats mongo

Slide 282

Slide 282 text

foo bar baz nats mongo production staging testing

Slide 283

Slide 283 text

foo bar baz nats mongo production staging testing foo foo foo

Slide 284

Slide 284 text

foo bar baz nats mongo production staging testing foo bar baz nats mongo foo bar baz nats mongo foo bar baz nats mongo

Slide 285

Slide 285 text

git build test release operate Dockerfile .kube/ backend.yaml frontend.yaml cron.yaml

Slide 286

Slide 286 text

git build test release operate Dockerfile .kube/ backend.yaml frontend.yaml cron.yaml docker build Docker- image

Slide 287

Slide 287 text

git build test release operate Dockerfile .kube/ backend.yaml frontend.yaml cron.yaml docker build Docker- image Docker registry

Slide 288

Slide 288 text

git build test release operate Dockerfile .kube/ backend.yaml frontend.yaml cron.yaml docker build Docker- image Docker registry Docker- image tests +

Slide 289

Slide 289 text

git build test release operate Dockerfile .kube/ backend.yaml frontend.yaml cron.yaml docker build Docker- image Docker registry Docker- image tests + kubectl

Slide 290

Slide 290 text

git build test release operate Dockerfile .kube/ backend.yaml frontend.yaml cron.yaml docker build Docker- image Docker registry Docker- image tests + kubectl

Slide 291

Slide 291 text

.kube/ backend.yaml frontend.yaml cron.yaml Docker- image Docker registry Docker- image tests + kubectl git build test release operate Dockerfile docker build

Slide 292

Slide 292 text

.kube/ backend.yaml frontend.yaml cron.yaml Docker- image Docker registry Docker- image tests + kubectl git build test release operate Dappfile build

Slide 293

Slide 293 text

Docker- image Docker registry Docker- image tests + git build test release operate Dappfile .kube/ backend.yaml frontend.yaml cron.yaml kubectl build

Slide 294

Slide 294 text

Docker- image Docker registry Docker- image tests + git build test release operate Dappfile .kube/ backend.yaml frontend.yaml cron.yaml build

Slide 295

Slide 295 text

Docker- image Docker registry Docker- image tests + git build test release operate Dappfile .kube/ backend.yaml frontend.yaml cron.yaml deploy build

Slide 296

Slide 296 text

Мы хотим сделать dapp действительно крутым и полезным Open Source-решением Нам нужен ВАШ опыт использования Будем рады любой помощи Поставьте звёздочку ;) Обещаем очень быстро отвечать на все issues и очень быстро исправлять все баги. Для нас dapp — это приоритетный проект. Если есть желание помочь с переводом документации на английский или с её доработкой, да или вообще помочь — будем очень рады и ждем pull-реквесты. Мы старались. github.com/flant/dapp

Slide 297

Slide 297 text

Собираем Docker- образы быстро и удобно habrahabr.ru/company/flant/blog/324274/

Slide 298

Slide 298 text

Инфраструктура НЕбольшого проекта

Slide 299

Slide 299 text

Инфраструктура НЕбольшого проекта

Slide 300

Slide 300 text

Инфраструктура НЕбольшого проекта Storage Ceph AWS GCE

Slide 301

Slide 301 text

Инфраструктура НЕбольшого проекта Storage Ceph AWS GCE Ingress

Slide 302

Slide 302 text

Инфраструктура НЕбольшого проекта Ingress Highly Available VM Storage Ceph AWS GCE

Slide 303

Slide 303 text

Инфраструктура НЕбольшого проекта Ingress Highly Available VM IOPS VM Storage Ceph AWS GCE

Slide 304

Slide 304 text

Frontend DB Backend memcached Cron Было

Slide 305

Slide 305 text

Highly Available VM IOPS VM Frontend DB Backend memcached Cron Было Стало Storage Ceph AWS GCE

Slide 306

Slide 306 text

Highly Available VM IOPS VM Frontend DB Backend memcached Cron Было Стало backend 3 Storage Ceph AWS GCE

Slide 307

Slide 307 text

Highly Available VM IOPS VM Frontend DB Backend memcached Cron Было Стало backend 3 DB Storage Ceph AWS GCE

Slide 308

Slide 308 text

Highly Available VM IOPS VM Frontend DB Backend memcached Cron Было Стало backend 3 DB cron-foo cron-bar cron-baz Storage Ceph AWS GCE

Slide 309

Slide 309 text

Highly Available VM IOPS VM Frontend DB Backend memcached Cron Было Стало backend 3 DB cron-foo cron-bar cron-baz frontend Storage Ceph AWS GCE

Slide 310

Slide 310 text

Highly Available VM IOPS VM Frontend DB Backend memcached Cron Было Стало backend 3 DB cron-foo cron-bar cron-baz frontend memcached 3 Storage Ceph AWS GCE

Slide 311

Slide 311 text

Highly Available VM IOPS VM DB frontend backend 3 memcached 3 cron-foo cron-bar cron-baz Frontend DB Backend memcached Cron Было Стало Files Storage Ceph AWS GCE

Slide 312

Slide 312 text

Highly Available VM IOPS VM Storage Ceph AWS GCE DB frontend backend 3 memcached 3 cron-foo cron-bar cron-baz Frontend DB Backend memcached Cron Было Стало Files Background Jobs Redis Redis 3 Background Jobs 3

Slide 313

Slide 313 text

Highly Available VM IOPS VM DB frontend backend 3 memcached 3 cron-foo cron-bar cron-baz Frontend DB Backend memcached Cron Было Стало Files Background Jobs Redis Redis 3 Background Jobs 3 Sphinx 2 Sphinx Storage Ceph AWS GCE

Slide 314

Slide 314 text

Highly Available VM IOPS VM frontend backend 3 memcached 3 cron-foo cron-bar cron-baz Frontend DB Backend memcached Cron Было Стало Files Background Jobs Redis Redis 3 Background Jobs 3 Sphinx 2 Sphinx Storage Ceph AWS GCE DB 3

Slide 315

Slide 315 text

Простая Цирк Облачная Хорошая Идеальная Микросервисная

Slide 316

Slide 316 text

Хорошая Облачная Хорошая Идеальная Микросервисная Цирк

Slide 317

Slide 317 text

Хорошая Облачная Хорошая Идеальная Микросервисная Хорошая

Slide 318

Slide 318 text

Хорошая Хорошая Хорошая Хорошая Идеальная Микросервисная

Slide 319

Slide 319 text

Хорошая Хорошая Хорошая Идеальная Микросервисная Хорошая

Slide 320

Slide 320 text

Хорошая Хорошая Хорошая Хорошая Идеальная Микросервисная

Slide 321

Slide 321 text

Хорошая Хорошая Хорошая Хорошая Идеальная Микросервисная

Slide 322

Slide 322 text

Наш блог на Хабрахабре habrahabr.ru/company/flant/ Тут мы рассказываем о нашем опыте и актуальных технологиях. Подписывайтесь, чтобы быть в курсе!

Slide 323

Slide 323 text

Наши Docker-проекты github.com/flant/dapp github.com/flant/docker_penetration_experiment github.com/flant/pam_docker github.com/flant/php_fpm_docker github.com/flant/superhosting github.com/flant/buildizer Дмитрий Столяров [email protected] linkedin.com/in/distol github.com/distol Всем спасибо! Наш блог на Хабрахабре habrahabr.ru/company/flant/