Slide 1
Slide 1 text
Дмитрий Столяров
Базы данных
и Kubernetes
CTO & Co-founder
v4
Slide 2
Slide 2 text
А можно базу
в Kubernetes?
?
?
?
?
?
?
Slide 3
Slide 3 text
А можно базу
в Kubernetes?
?
?
?
?
?
?
В целом да, но...
!
Slide 4
Slide 4 text
?
?
?
?
?
?
В целом да, но...
!
А можно базу
в Kubernetes?
Slide 5
Slide 5 text
?
?
?
?
?
?
В целом да, но...
!
А можно базу
в Kubernetes?
stateful
Slide 6
Slide 6 text
?
?
?
?
?
?
А можно базу
в Kubernetes?
stateful
Зачем он нам в Kubernetes?
Slide 7
Slide 7 text
?
?
?
?
?
?
А можно базу
в Kubernetes?
stateful
Зачем он нам в Kubernetes?
О каком stateful идет речь?
Slide 8
Slide 8 text
О каком stateful идет речь?
Relational Database NoSQL Database
Slide 9
Slide 9 text
О каком stateful идет речь?
Relational Database NoSQL Database In-memory Database
Slide 10
Slide 10 text
О каком stateful идет речь?
Relational Database NoSQL Database In-memory Database Message Queue
Slide 11
Slide 11 text
О каком stateful идет речь?
Search Engine
Relational Database NoSQL Database In-memory Database Message Queue
Slide 12
Slide 12 text
О каком stateful идет речь?
Search Engine Object Storage
Relational Database NoSQL Database In-memory Database Message Queue
Slide 13
Slide 13 text
О каком stateful идет речь?
Search Engine Object Storage
Relational Database NoSQL Database In-memory Database Message Queue
Configuration &
Service Discovery
Slide 14
Slide 14 text
О каком stateful идет речь?
...
Search Engine Object Storage
Relational Database NoSQL Database In-memory Database Message Queue
Configuration &
Service Discovery
Slide 15
Slide 15 text
Философия
высокой доступности
в Kubernetes
Хранение данных
и Kubernetes
Гарантии
согласованности
в Kubernetes
Slide 16
Slide 16 text
#1
Философия
высокой доступности
в Kubernetes
Slide 17
Slide 17 text
cattle
pets
Slide 18
Slide 18 text
No content
Slide 19
Slide 19 text
No content
Slide 20
Slide 20 text
No content
Slide 21
Slide 21 text
RAID
RAID
Slide 22
Slide 22 text
RAID MySQL
RAID MySQL
Slide 23
Slide 23 text
RAID MySQL
RAID MySQL
Slide 24
Slide 24 text
RAID MySQL
keepalived
RAID MySQL
keepalived
Slide 25
Slide 25 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Slide 26
Slide 26 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Slide 27
Slide 27 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Slide 28
Slide 28 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Slide 29
Slide 29 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Slide 30
Slide 30 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Slide 31
Slide 31 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Slide 32
Slide 32 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Slide 33
Slide 33 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Slide 34
Slide 34 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Slide 35
Slide 35 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Slide 36
Slide 36 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Slide 37
Slide 37 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Slide 38
Slide 38 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Slide 39
Slide 39 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Slide 40
Slide 40 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
...
Slide 41
Slide 41 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
...
Slide 42
Slide 42 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
...
Slide 43
Slide 43 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
...
Slide 44
Slide 44 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
...
Slide 45
Slide 45 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
PHP
PHP
...
PHP
Slide 46
Slide 46 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
PHP
...
PHP
PHP
Slide 47
Slide 47 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
PHP
PHP
...
PHP
Slide 48
Slide 48 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
PHP
PHP
...
PHP
PHP
Slide 49
Slide 49 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Механизмы high availability
PHP
PHP
...
PHP
PHP
Slide 50
Slide 50 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Механизмы high availability
Deployment и StatefulSet
PHP
PHP
...
PHP
PHP
Slide 51
Slide 51 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Механизмы high availability
Deployment и StatefulSet
PodAntiAffinity
PHP
PHP
...
PHP
PHP
Slide 52
Slide 52 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Deployment и StatefulSet
PodAntiAffinity
PodDisruptionBudgets
Механизмы high availability
PHP
PHP
...
PHP
PHP
Slide 53
Slide 53 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Deployment и StatefulSet
PodAntiAffinity
PodDisruptionBudgets
Механизмы high availability
PHP
PHP
...
PHP
PHP
Slide 54
Slide 54 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Deployment и StatefulSet
PodAntiAffinity
PodDisruptionBudgets
Механизмы high availability
PHP
PHP
...
PHP
PHP
Slide 55
Slide 55 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Deployment и StatefulSet
PodAntiAffinity
PodDisruptionBudgets
Механизмы high availability
PHP
PHP
...
PHP
Slide 56
Slide 56 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Deployment и StatefulSet
PodAntiAffinity
PodDisruptionBudgets
Механизмы high availability
PHP
PHP
...
PHP
PHP
Slide 57
Slide 57 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
Deployment и StatefulSet
PodAntiAffinity
PodDisruptionBudgets
Механизмы high availability
PHP
PHP
...
PHP
PHP
Slide 58
Slide 58 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
PHP
PHP
...
PHP
PHP
Slide 59
Slide 59 text
Итог
Slide 60
Slide 60 text
Есть мощный фундамент
Итог
Slide 61
Slide 61 text
Есть мощный фундамент
Требуются другие подходы
Итог
Slide 62
Slide 62 text
Есть мощный фундамент
Требуются другие подходы
Для stateful важна надежность узлов
Итог
Slide 63
Slide 63 text
#2
Гарантии
согласованности
в Kubernetes
Slide 64
Slide 64 text
DB
IP
DB
IP
keepalived
keepalived
Как оно было раньше?
Slide 65
Slide 65 text
App
App
App
App
App
DB
IP
DB
IP
App
App
App
App
App
keepalived
keepalived
Как оно было раньше?
Slide 66
Slide 66 text
App
App
App
App
App
DB
IP
DB
IP
App
App
App
App
App
Balancer
Balancer
keepalived
keepalived
Как оно было раньше?
Slide 67
Slide 67 text
App
App
App
App
App
DB
IP
DB
IP
App
App
App
App
App
Balancer
Balancer
keepalived
keepalived
Как оно было раньше?
Slide 68
Slide 68 text
App
App
App
App
App
DB
IP
DB
IP
App
App
App
App
App
Balancer
Balancer
keepalived
keepalived
Как оно было раньше?
Slide 69
Slide 69 text
App
App
App
App
App
DB
IP
DB
IP
App
App
App
App
App
Balancer
Balancer
keepalived
keepalived
Как оно было раньше?
Slide 70
Slide 70 text
App
App
App
App
App
DB
IP
DB
IP
App
App
App
App
App
Balancer
Balancer
keepalived
keepalived
Как оно было раньше?
Slide 71
Slide 71 text
App
App
App
App
App
DB
IP
DB
IP
App
App
App
App
App
Balancer
Balancer
keepalived
keepalived
Как оно было раньше?
Slide 72
Slide 72 text
App
App
App
App
App
DB
IP
DB
IP
App
App
App
App
App
Balancer
Balancer
keepalived
keepalived
IP
Как оно было раньше?
Slide 73
Slide 73 text
App
App
App
App
App
DB
IP
DB
IP
App
App
App
App
App
Balancer
Balancer
keepalived
keepalived
IP
Как оно было раньше?
Slide 74
Slide 74 text
App
App
App
App
App
DB
IP
DB
IP
App
App
App
App
App
Balancer
Balancer
keepalived
keepalived
Как оно было раньше?
Slide 75
Slide 75 text
App
App
App
App
App
DB
IP
DB
IP
App
App
App
App
App
Balancer
Balancer
Как оно было раньше?
Slide 76
Slide 76 text
App
App
App
App
App
DB
IP
DB
IP
App
App
App
App
App
Balancer
Balancer
corosync
corosync
corosync
Как оно было раньше?
Slide 77
Slide 77 text
App
App
App
App
App
DB
IP
DB
IP
App
App
App
App
App
Balancer
Balancer
corosync
corosync
corosync
Как оно было раньше?
Slide 78
Slide 78 text
App
App
App
App
App
DB
IP
DB
IP
App
App
App
App
App
Balancer
Balancer
corosync
corosync
corosync
Как оно было раньше?
Slide 79
Slide 79 text
App
App
App
App
App
DB
IP
DB
IP
App
App
App
App
App
Balancer
Balancer
corosync
corosync
corosync
Как оно было раньше?
Slide 80
Slide 80 text
App
App
App
App
App
DB
IP
DB
IP
App
App
App
App
App
Balancer
Balancer
corosync
corosync
corosync
Как оно было раньше?
Slide 81
Slide 81 text
App
App
App
App
App
DB
IP
DB
IP
App
App
App
App
App
Balancer
Balancer
corosync
corosync
corosync
Как оно было раньше?
Slide 82
Slide 82 text
App
App
App
App
App
DB
IP
App
App
App
App
App
Balancer
Balancer
corosync
corosync
DB
IP
corosync
Как оно было раньше?
Slide 83
Slide 83 text
App
App
App
App
App
App
App
App
App
App
Balancer
Balancer
DB
IP
corosync
Как оно было раньше?
DB
IP
corosync
corosync
Slide 84
Slide 84 text
App
App
App
App
App
App
App
App
App
App
Balancer
Balancer
DB
IP
corosync
Как оно было раньше?
DB
IP
corosync
corosync
1
Slide 85
Slide 85 text
App
App
App
App
App
App
App
App
App
App
Balancer
Balancer
DB
IP
corosync
Как оно было раньше?
DB
IP
corosync
IP
2
corosync
2
1
Slide 86
Slide 86 text
App
App
App
App
App
App
App
App
App
App
Balancer
Balancer
corosync
DB
IP
corosync
Как оно было раньше?
DB
IP
corosync
IP
Slide 87
Slide 87 text
App
App
App
App
App
App
App
App
App
App
Balancer
Balancer
corosync
DB
IP
corosync
Как оно было раньше?
DB
IP
corosync
IP
Slide 88
Slide 88 text
App
App
App
App
App
App
App
App
App
App
Balancer
Balancer
corosync
DB
IP
corosync
Как оно было раньше?
DB
IP
corosync
IP
At-most-once guarantee
Slide 89
Slide 89 text
А как там в Kubernetes?
Slide 90
Slide 90 text
app-csm52
app-59ptg app-xfvr7
Deployment
А как там в Kubernetes?
Slide 91
Slide 91 text
app-csm52
app-59ptg app-xfvr7
db-0
db-1
Deployment
StatefulSet
А как там в Kubernetes?
Slide 92
Slide 92 text
app-csm52
app-59ptg app-xfvr7
Deployment
StatefulSet
А как там в Kubernetes?
db-0
db-1
Slide 93
Slide 93 text
app-csm52
app-59ptg app-xfvr7
Deployment
StatefulSet
А как там в Kubernetes?
db-0
db-1
Slide 94
Slide 94 text
app-csm52
app-59ptg app-xfvr7
app-ck9cm
Deployment
StatefulSet
А как там в Kubernetes?
db-0
db-1
Slide 95
Slide 95 text
app-csm52
app-59ptg app-xfvr7
app-ck9cm
Deployment
StatefulSet
А как там в Kubernetes?
At-most-once guarantee
db-0
db-1
Slide 96
Slide 96 text
app-59ptg app-xfvr7
app-ck9cm
Deployment
StatefulSet
А как там в Kubernetes?
db-1
app-csm52
db-0
Slide 97
Slide 97 text
app-59ptg app-xfvr7
app-ck9cm
Deployment
StatefulSet
А как там в Kubernetes?
db-1
app-csm52
Может продолжить
писать в базу!
db-0
Slide 98
Slide 98 text
app-csm52
app-59ptg app-xfvr7
app-ck9cm
Deployment
StatefulSet
А как там в Kubernetes?
At-most-once guarantee
но также требуется fencing
db-0
db-1
Slide 99
Slide 99 text
app-csm52
app-59ptg app-xfvr7
app-ck9cm
Deployment
StatefulSet
А как там в Kubernetes?
At-most-once guarantee
но так же требуется fencing
db-0
db-1
Все еще требуется fencing!!
Slide 100
Slide 100 text
app-csm52
app-59ptg app-xfvr7
app-ck9cm
Deployment
StatefulSet
А как там в Kubernetes?
At-most-once guarantee
но так же требуется fencing
db-0
db-1
Все еще требуется fencing!!
...или?
Slide 101
Slide 101 text
Другой подход
Slide 102
Slide 102 text
DB
IP
DB
IP
App
App
App
App
App
App
App
App
App
App
Balancer
Balancer
Другой подход
Slide 103
Slide 103 text
DB
IP
DB
IP
App
App
App
App
App
App
App
App
App
App
Balancer
Balancer
Другой подход
Slide 104
Slide 104 text
DB
IP
DB
IP
App
App
App
App
App
App
App
App
App
App
Balancer
Balancer
Другой подход
LB
Slide 105
Slide 105 text
DB
IP
DB
IP
App
App
App
App
App
App
App
App
App
App
Balancer
Balancer
Другой подход
LB
Slide 106
Slide 106 text
DB
IP
DB
IP
App
App
App
App
App
App
App
App
App
App
Balancer
Balancer
Другой подход
LB
Slide 107
Slide 107 text
DB
IP
DB
IP
App
App
App
App
App
App
App
App
App
App
Balancer
Balancer
Другой подход
LB
Slide 108
Slide 108 text
DB
IP
DB
IP
App
App
App
App
App
App
App
App
App
App
Balancer
Balancer
Другой подход
LB
Slide 109
Slide 109 text
DB
IP
DB
IP
App
App
App
App
App
App
App
App
App
App
Balancer
Balancer
Другой подход
LB
Slide 110
Slide 110 text
DB
IP
DB
IP
App
App
App
App
App
App
App
App
App
App
Balancer
Balancer
Другой подход
LB
“Consistent Switchover”
Slide 111
Slide 111 text
DB
IP
DB
IP
App
App
App
App
App
App
App
App
App
App
Balancer
Balancer
Другой подход
LB
Но как сделать generic?
Slide 112
Slide 112 text
DB
IP
DB
IP
App
App
App
App
App
App
App
App
App
App
Balancer
Balancer
Другой подход
Slide 113
Slide 113 text
App
App
App
App
App
DB
IP
DB
IP
App
P
App
P
App
P
App
P
App
P
Balancer
Balancer
Другой подход
Slide 114
Slide 114 text
App
App
App
App
App
DB
IP
DB
IP
App
P
App
P
App
P
App
P
App
P
Balancer
Balancer
Другой подход
Slide 115
Slide 115 text
App
App
App
App
App
DB
IP
DB
IP
App
P
App
P
App
P
App
P
App
P
Balancer
Balancer
Другой подход
Slide 116
Slide 116 text
App
App
App
App
App
DB
IP
DB
IP
App
P
App
P
App
P
App
P
App
P
Balancer
Balancer
Другой подход
Slide 117
Slide 117 text
App
App
App
App
App
DB
IP
DB
IP
App
P
App
P
App
P
App
P
App
P
Balancer
Balancer
Другой подход
Slide 118
Slide 118 text
App
App
App
App
App
DB
IP
DB
IP
App
P
App
P
App
P
App
P
App
P
Balancer
Balancer
Другой подход
Slide 119
Slide 119 text
App
App
App
App
App
DB
IP
DB
IP
App
P
App
P
App
P
App
P
App
P
Balancer
Balancer
Другой подход
Slide 120
Slide 120 text
App
App
App
App
App
DB
IP
DB
IP
App
P
App
P
App
P
App
P
App
P
Balancer
Balancer
Другой подход
Slide 121
Slide 121 text
App
App
App
App
App
DB
IP
DB
IP
App
P
App
P
App
P
App
P
App
P
Balancer
Balancer
Другой подход
Slide 122
Slide 122 text
App
App
App
App
App
DB
IP
DB
IP
App
P
App
P
App
P
App
P
App
P
Balancer
Balancer
Другой подход
Slide 123
Slide 123 text
App
App
App
App
App
DB
IP
DB
IP
App
P
App
P
App
P
App
P
App
P
Balancer
Balancer
Другой подход
Slide 124
Slide 124 text
App
App
App
App
App
DB
IP
DB
IP
App
P
App
P
App
P
App
P
App
P
Balancer
Balancer
Другой подход
Slide 125
Slide 125 text
App
App
App
App
App
DB
IP
DB
IP
App
P
App
P
App
P
App
P
App
P
Balancer
Balancer
“Consistent Switchover”
Другой подход
Slide 126
Slide 126 text
Итог
Slide 127
Slide 127 text
Есть семантика at-most-once
Итог
Slide 128
Slide 128 text
Есть семантика at-most-once
Нет fencing’а, хотя можно и сделать
Итог
Slide 129
Slide 129 text
Есть семантика at-most-once
Нет fencing’а, хотя можно и сделать
Очень не хватает Consistent Switchover
Итог
Slide 130
Slide 130 text
#3
Хранение данных
и Kubernetes
Slide 131
Slide 131 text
Network Block Device
aka SAN
ReadWriteOnce
Kubernetes Access Mode:
Slide 132
Slide 132 text
Network Block Device
aka SAN
ReadWriteOnce
Kubernetes Access Mode:
Public cloud:
AzureDisk
AWSElasticBlockStore
GCEPersistentDisk
Slide 133
Slide 133 text
Network Block Device
aka SAN
ReadWriteOnce
Kubernetes Access Mode:
Public cloud:
AzureDisk
AWSElasticBlockStore
GCEPersistentDisk
Private cloud:
Cinder
VsphereVolume
Slide 134
Slide 134 text
Network Block Device
aka SAN
ReadWriteOnce
Kubernetes Access Mode:
Public cloud:
AzureDisk
AWSElasticBlockStore
GCEPersistentDisk
Private cloud:
Cinder
VsphereVolume
Bare metal or private cloud:
Ceph RBD
Fibre Channel
iSCSI
Slide 135
Slide 135 text
Network Block Device
db-0
Slide 136
Slide 136 text
Network Block Device
db-0
Slide 137
Slide 137 text
Network Block Device
db-0
Slide 138
Slide 138 text
Network Block Device
db-0
Slide 139
Slide 139 text
Network Block Device
db-0
Slide 140
Slide 140 text
Network Block Device
db-0
Slide 141
Slide 141 text
Network Block Device
Slide 142
Slide 142 text
Network Block Device
db-0
Slide 143
Slide 143 text
Network Block Device
db-0
Slide 144
Slide 144 text
Network Block Device
db-0
Slide 145
Slide 145 text
Network Block Device
db-0
Slide 146
Slide 146 text
Network Block Device
db-0
Slide 147
Slide 147 text
Network Block Device
db-0
Slide 148
Slide 148 text
Network Block Device
db-0
Slide 149
Slide 149 text
Network Block Device
db-0
Slide 150
Slide 150 text
Network Block Device
db-0
Slide 151
Slide 151 text
Network Block Device
db-0
Slide 152
Slide 152 text
IOPS & Latency
Slide 153
Slide 153 text
IOPS & Latency
NVMe
MySQL
100µs
Slide 154
Slide 154 text
IOPS & Latency
NVMe
MySQL
RDMA
NVMe
MySQL
150µs
100µs
Slide 155
Slide 155 text
IOPS & Latency
NVMe
MySQL
RDMA
NVMe
MySQL
Commodity
iSCSI
MySQL
100µs
1ms
150µs
Slide 156
Slide 156 text
IOPS & Latency
NVMe
MySQL
RDMA
NVMe
MySQL
AWS EBS
MySQL
Commodity
iSCSI
MySQL
100µs
1ms 2-5ms
150µs
Slide 157
Slide 157 text
IOPS & Latency
NVMe
MySQL
RDMA
NVMe
MySQL
AWS EBS
MySQL
Commodity
iSCSI
MySQL
100µs
1ms 2-5ms
150µs
Performance Analysis of NVMe SSDs
and their Implication on Real World Databases
https://www.cs.utah.edu/~manua/pubs/systor15.pdf
NVMe Performance Local vs. Remote
https://www.samsung.com/us/labs/pdfs/2016-fms-forum-f-22-nvmf-local-remote-v05.pdf
Slide 158
Slide 158 text
IOPS & Latency
NVMe
MySQL
RDMA
NVMe
MySQL
AWS EBS
MySQL
Commodity
iSCSI
MySQL
Допустим, базе данных требуется ~50K
IOPS
100µs
1ms 2-5ms
150µs
Performance Analysis of NVMe SSDs
and their Implication on Real World Databases
https://www.cs.utah.edu/~manua/pubs/systor15.pdf
NVMe Performance Local vs. Remote
https://www.samsung.com/us/labs/pdfs/2016-fms-forum-f-22-nvmf-local-remote-v05.pdf
Slide 159
Slide 159 text
IOPS & Latency
NVMe
MySQL
100µs
5 threads
RDMA
NVMe
MySQL
150µs
AWS EBS
MySQL
2-5ms
Commodity
iSCSI
MySQL
1ms
Performance Analysis of NVMe SSDs
and their Implication on Real World Databases
https://www.cs.utah.edu/~manua/pubs/systor15.pdf
NVMe Performance Local vs. Remote
https://www.samsung.com/us/labs/pdfs/2016-fms-forum-f-22-nvmf-local-remote-v05.pdf
Допустим, базе данных требуется ~50K
IOPS
Slide 160
Slide 160 text
IOPS & Latency
Performance Analysis of NVMe SSDs
and their Implication on Real World Databases
https://www.cs.utah.edu/~manua/pubs/systor15.pdf
NVMe Performance Local vs. Remote
https://www.samsung.com/us/labs/pdfs/2016-fms-forum-f-22-nvmf-local-remote-v05.pdf
NVMe
MySQL
100µs
5 threads
RDMA
NVMe
MySQL
150µs
7.5 threads
AWS EBS
MySQL
2-5ms
Commodity
iSCSI
MySQL
1ms
Допустим, базе данных требуется ~50K
IOPS
Slide 161
Slide 161 text
IOPS & Latency
Performance Analysis of NVMe SSDs
and their Implication on Real World Databases
https://www.cs.utah.edu/~manua/pubs/systor15.pdf
NVMe Performance Local vs. Remote
https://www.samsung.com/us/labs/pdfs/2016-fms-forum-f-22-nvmf-local-remote-v05.pdf
NVMe
MySQL
100µs
5 threads
RDMA
NVMe
MySQL
150µs
7.5 threads
AWS EBS
MySQL
2-5ms
Commodity
iSCSI
MySQL
1ms
50 threads
Допустим, базе данных требуется ~50K
IOPS
Slide 162
Slide 162 text
IOPS & Latency
Performance Analysis of NVMe SSDs
and their Implication on Real World Databases
https://www.cs.utah.edu/~manua/pubs/systor15.pdf
NVMe Performance Local vs. Remote
https://www.samsung.com/us/labs/pdfs/2016-fms-forum-f-22-nvmf-local-remote-v05.pdf
NVMe
MySQL
100µs
5 threads
RDMA
NVMe
MySQL
150µs
7.5 threads
AWS EBS
MySQL
2-5ms
100-250 threads
Commodity
iSCSI
MySQL
1ms
50 threads
Допустим, базе данных требуется ~50K
IOPS
Slide 163
Slide 163 text
Network Block Device
aka SAN
ReadWriteOnce
Kubernetes Access Mode:
Public cloud:
AzureDisk
AWSElasticBlockStore
GCEPersistentDisk
Private cloud:
Cinder
VsphereVolume
Bare metal or private cloud:
Ceph RBD
Fibre Channel
iSCSI
Local Storage
ReadWriteOnce
Kubernetes Access Mode:
Это просто локальный диск:
SATA / SAS / SSD / NVME
Slide 164
Slide 164 text
db-0
Local storage
Slide 165
Slide 165 text
db-0
Local storage
Slide 166
Slide 166 text
db-0
Local storage
Slide 167
Slide 167 text
db-0
Local storage
Slide 168
Slide 168 text
Network Block Device
aka SAN
ReadWriteOnce
Kubernetes Access Mode:
Public cloud:
AzureDisk
AWSElasticBlockStore
GCEPersistentDisk
Private cloud:
Cinder
VsphereVolume
Bare metal or private cloud:
Ceph RBD
Fibre Channel
iSCSI
Local Storage
ReadWriteOnce
Kubernetes Access Mode:
Это просто локальный диск:
SATA / SAS / SSD / NVME
Slide 169
Slide 169 text
Network Block Device
aka SAN
ReadWriteOnce
Kubernetes Access Mode:
Public cloud:
AzureDisk
AWSElasticBlockStore
GCEPersistentDisk
Private cloud:
Cinder
VsphereVolume
Bare metal or private cloud:
Ceph RBD
Fibre Channel
iSCSI
Local Storage
ReadWriteOnce
Kubernetes Access Mode:
Это просто локальный диск:
SATA / SAS / SSD / NVME
Slide 170
Slide 170 text
ReadWriteOnce
db-0
Slide 171
Slide 171 text
ReadWriteOnce
db-0
db-1
Slide 172
Slide 172 text
ReadWriteOnce
db-0
db-1
Slide 173
Slide 173 text
ReadWriteOnce
db-0
db-1
Slide 174
Slide 174 text
ReadWriteOnce
db-0
db-1
pv-1 pv-0
Slide 175
Slide 175 text
ReadWriteOnce и Stateful Set
db-0
pv-1 pv-0
db-1
Slide 176
Slide 176 text
ReadWriteOnce и Stateful Set
db-0
db-1
pv-1 pv-0
Slide 177
Slide 177 text
ReadWriteOnce и Stateful Set
db-0
db-1 db-2
pv-1 pv-0 pv-2
Slide 178
Slide 178 text
Network Block Device Network File System
aka SAN aka NAS
ReadWriteOnce
Kubernetes Access Mode:
ReadWriteMany
Kubernetes Access Mode:
Public cloud:
AzureDisk
AWSElasticBlockStore
GCEPersistentDisk
Private cloud:
Cinder
VsphereVolume
Bare metal or private cloud:
Ceph RBD
Fibre Channel
iSCSI
Local Storage
ReadWriteOnce
Kubernetes Access Mode:
Это просто локальный диск:
SATA / SAS / SSD / NVME
Slide 179
Slide 179 text
Network Block Device Network File System
aka SAN aka NAS
ReadWriteOnce
Kubernetes Access Mode:
ReadWriteMany
Kubernetes Access Mode:
Public cloud:
AzureDisk
AWSElasticBlockStore
GCEPersistentDisk
Private cloud:
Cinder
VsphereVolume
Bare metal or private cloud:
Ceph RBD
Fibre Channel
iSCSI
Public cloud:
AzureFile
AWSElasticFileSystem
Local Storage
ReadWriteOnce
Kubernetes Access Mode:
Это просто локальный диск:
SATA / SAS / SSD / NVME
Slide 180
Slide 180 text
Network Block Device Network File System
aka SAN aka NAS
ReadWriteOnce
Kubernetes Access Mode:
ReadWriteMany
Kubernetes Access Mode:
Public cloud:
AzureDisk
AWSElasticBlockStore
GCEPersistentDisk
Private cloud:
Cinder
VsphereVolume
Bare metal or private cloud:
Ceph RBD
Fibre Channel
iSCSI
Public cloud:
AzureFile
AWSElasticFileSystem
Bare metal or private cloud:
CephFS
Glusterfs
NFS
Local Storage
ReadWriteOnce
Kubernetes Access Mode:
Это просто локальный диск:
SATA / SAS / SSD / NVME
Slide 181
Slide 181 text
Network File System
PHP PHP
PHP
Slide 182
Slide 182 text
Network Block Device Network File System
aka SAN aka NAS
ReadWriteOnce
Kubernetes Access Mode:
ReadWriteMany
Kubernetes Access Mode:
Public cloud:
AzureDisk
AWSElasticBlockStore
GCEPersistentDisk
Private cloud:
Cinder
VsphereVolume
Bare metal or private cloud:
Ceph RBD
Fibre Channel
iSCSI
Public cloud:
AzureFile
AWSElasticFileSystem
Bare metal or private cloud:
CephFS
Glusterfs
NFS
Local Storage
ReadWriteOnce
Kubernetes Access Mode:
Это просто локальный диск:
SATA / SAS / SSD / NVME
Slide 183
Slide 183 text
Итог
Slide 184
Slide 184 text
Итог
С ReadWriteOnce
всё отлично!
Slide 185
Slide 185 text
Итог
С ReadWriteOnce
всё отлично!
Старайтесь избегать
Network File System
Slide 186
Slide 186 text
Всё, освоено!
Slide 187
Slide 187 text
Философия
высокой доступности
Гарантии
согласованности
Хранение данных
Практика!
Slide 188
Slide 188 text
#1
Standalone
Slide 189
Slide 189 text
postgres-0
Slide 190
Slide 190 text
postgres-0
Slide 191
Slide 191 text
postgres-0
mysql-0
Slide 192
Slide 192 text
postgres-0
mysql-0
Slide 193
Slide 193 text
postgres-0
mysql-0
Slide 194
Slide 194 text
postgres-0
mysql-0
Slide 195
Slide 195 text
mysql-0
StatefulSet
Slide 196
Slide 196 text
mysql-0
mysql
StatefulSet
Slide 197
Slide 197 text
app
mysql-0
mysql
StatefulSet
Slide 198
Slide 198 text
#2
Реплицированная пара
с ручным переключением
Slide 199
Slide 199 text
mysql-a-0
StatefulSet
Slide 200
Slide 200 text
mysql-a-0
mysql-a
StatefulSet
Slide 201
Slide 201 text
mysql-b-0
StatefulSet
mysql-a-0
mysql-a
StatefulSet
Slide 202
Slide 202 text
mysql-b-0
StatefulSet
mysql-a-0
mysql-a
StatefulSet
репликация
Slide 203
Slide 203 text
mysql-b-0
mysql-b
StatefulSet
mysql-a-0
mysql-a
StatefulSet
репликация
Slide 204
Slide 204 text
mysql-b-0
mysql-b
StatefulSet
mysql-a-0
mysql-a
StatefulSet
mysql
репликация
Slide 205
Slide 205 text
mysql-b-0
mysql-b
StatefulSet
mysql-a-0
mysql-a
StatefulSet
mysql
репликация
Slide 206
Slide 206 text
app
mysql-b-0
mysql-b
StatefulSet
mysql-a-0
mysql-a
StatefulSet
mysql
репликация
Slide 207
Slide 207 text
app
mysql-b-0
mysql-b
StatefulSet
mysql-a-0
mysql-a
StatefulSet
mysql
репликация
Slide 208
Slide 208 text
app
mysql-b-0
mysql-b
StatefulSet
mysql-a-0
mysql-a
StatefulSet
mysql
репликация
Slide 209
Slide 209 text
app
mysql-b-0
mysql-b
StatefulSet
mysql-a-0
mysql-a
StatefulSet
mysql
репликация
Slide 210
Slide 210 text
app
mysql-b-0
mysql-b
StatefulSet
mysql-a-0
mysql-a
StatefulSet
mysql
репликация
Slide 211
Slide 211 text
app
mysql-b-0
mysql-b
StatefulSet
mysql-a-0
mysql-a
StatefulSet
mysql
репликация
Slide 212
Slide 212 text
app
mysql-b-0
mysql-b
StatefulSet
mysql-a-0
mysql-a
StatefulSet
mysql
репликация
Slide 213
Slide 213 text
#3
Масштабирование
нагрузки на чтение
Slide 214
Slide 214 text
MySQL
Slide 215
Slide 215 text
MySQL
mysql-ro-0
StatefulSet
mysql-ro-1 mysql-ro-2
...
репликация
Slide 216
Slide 216 text
MySQL
mysql-ro-0
StatefulSet
mysql-ro-1 mysql-ro-2
...
репликация
mysql-ro
Slide 217
Slide 217 text
MySQL
mysql-ro-0
StatefulSet
mysql-ro-1 mysql-ro-2
...
репликация
mysql-ro
app
read
Slide 218
Slide 218 text
MySQL
mysql-ro-0
StatefulSet
mysql-ro-1 mysql-ro-2
...
репликация
mysql-ro
app
write
read
Slide 219
Slide 219 text
#4
Умный клиент
Slide 220
Slide 220 text
mc-0
StatefulSet
mc-1 mc-2
Slide 221
Slide 221 text
mc-0
StatefulSet
memcached
mc-1 mc-2
Slide 222
Slide 222 text
mc-1
mc-0
StatefulSet
memcached
mc-2
mc-0.memcached mc-1.memcached mc-2.memcached
Slide 223
Slide 223 text
mc-1
mc-0
StatefulSet
memcached
app
mc-2
mc-0.memcached mc-1.memcached mc-2.memcached
репликация
шардинг
Slide 224
Slide 224 text
mc-1
mc-0
StatefulSet
memcached
app
mc-2
mc-0.memcached mc-1.memcached mc-2.memcached
репликация
шардинг
Slide 225
Slide 225 text
mc-1
mc-0
StatefulSet
memcached
app
mc-2
mc-0.memcached mc-1.memcached mc-2.memcached
репликация
шардинг
Slide 226
Slide 226 text
#5
Cloud Native-решения
Slide 227
Slide 227 text
No content
Slide 228
Slide 228 text
Умеют сами
Failover
Recovery
Consistency
Slide 229
Slide 229 text
db-1
db-0
StatefulSet
db-2
Slide 230
Slide 230 text
1. Как узлы узнают друг о друге?
Slide 231
Slide 231 text
1. Как узлы узнают друг о друге?
Kubernetes API
Slide 232
Slide 232 text
1. Как узлы узнают друг о друге?
Kubernetes API
DNS (несколько A-записей)
Slide 233
Slide 233 text
1. Как узлы узнают друг о друге?
Kubernetes API
DNS (несколько A-записей)
Статически
Slide 234
Slide 234 text
1. Как узлы узнают друг о друге?
Kubernetes API
DNS (несколько A-записей)
Статически
Seed-узлы
Slide 235
Slide 235 text
1. Как узлы узнают друг о друге?
Kubernetes API
DNS (несколько A-записей)
Статически
Seed-узлы
Сторонний Service Discovery
Slide 236
Slide 236 text
2. Как подключается клиент?
Slide 237
Slide 237 text
rabbitmq-1
rabbitmq-0
StatefulSet
rabbitmq-2
app
rabbitmq
2. Как подключается клиент?
Slide 238
Slide 238 text
app
mg-1
mg-0
StatefulSet
mongo
mg-2
mc-0.mongo mc-1.mongo mc-2.mongo
2. Как подключается клиент?
Slide 239
Slide 239 text
3. Горизонтальное масштабирование?
Slide 240
Slide 240 text
minio-1
minio-0
StatefulSet
minio-2
3. Горизонтальное масштабирование?
Нет
Slide 241
Slide 241 text
minio-1
minio-0
StatefulSet
minio-2
cassandra-1
cassandra-0
StatefulSet
cassandra-2
...
3. Горизонтальное масштабирование
Нет
Есть
Slide 242
Slide 242 text
minio-1
minio-0
StatefulSet
minio-2
cassandra-1
cassandra-0
StatefulSet
cassandra-2
...
mongo-a-1
mongo-a-0
StatefulSet
mongo-a-2 mongo-a-1
mongo-a-0
StatefulSet
mongo-a-2
...
3. Горизонтальное масштабирование
Нет
Есть
Есть #2
Slide 243
Slide 243 text
No content
Slide 244
Slide 244 text
Отлично сочетаются с Kubernetes
Slide 245
Slide 245 text
cattle
pets
Отлично сочетаются с Kubernetes
Slide 246
Slide 246 text
cattle
pets
Отлично сочетаются с Kubernetes
Slide 247
Slide 247 text
#6
Stolon PostgreSQL
Slide 248
Slide 248 text
PostgreSQL cloud native High Availability
Slide 249
Slide 249 text
cattle
pets
PostgreSQL cloud native High Availability
Slide 250
Slide 250 text
cattle
pets
PostgreSQL cloud native High Availability
Slide 251
Slide 251 text
etcd-0
StatefulSet
etcd-1 etcd-2
etcd
Slide 252
Slide 252 text
etcd-0
StatefulSet
etcd-1 etcd-2
etcd
keeper-0
StatefulSet
keeper-1 keeper-2
postgresql postgresql postgresql
Slide 253
Slide 253 text
etcd-0
StatefulSet
etcd-1 etcd-2
etcd
keeper-0
StatefulSet
keeper-1 keeper-2
postgresql postgresql postgresql
Slide 254
Slide 254 text
etcd-0
StatefulSet
etcd-1 etcd-2
etcd
sentinel
Deployment
sentinel
keeper-0
StatefulSet
keeper-1 keeper-2
postgresql postgresql postgresql
Slide 255
Slide 255 text
etcd-0
StatefulSet
etcd-1 etcd-2
etcd
sentinel
Deployment
sentinel
keeper-0
StatefulSet
keeper-1 keeper-2
master
postgresql postgresql postgresql
Slide 256
Slide 256 text
etcd-0
StatefulSet
etcd-1 etcd-2
etcd
sentinel
Deployment
sentinel
keeper-0
StatefulSet
keeper-1 keeper-2
master standby
standby
postgresql postgresql postgresql
Slide 257
Slide 257 text
etcd-0
StatefulSet
etcd-1 etcd-2
etcd
sentinel
Deployment
sentinel
keeper-0
StatefulSet
keeper-1 keeper-2
master standby
standby
postgresql postgresql postgresql
Slide 258
Slide 258 text
etcd-0
StatefulSet
etcd-1 etcd-2
etcd
sentinel
Deployment
sentinel
keeper-0
StatefulSet
keeper-1 keeper-2
master standby
standby
postgresql postgresql postgresql
репликация
репликация
Slide 259
Slide 259 text
etcd-0
StatefulSet
etcd-1 etcd-2
etcd
sentinel
Deployment
sentinel
proxy
Deployment
proxy
keeper-0
StatefulSet
keeper-1 keeper-2
master standby
standby
postgresql postgresql postgresql
репликация
репликация
Slide 260
Slide 260 text
etcd-0
StatefulSet
etcd-1 etcd-2
etcd
sentinel
Deployment
sentinel
proxy
Deployment
proxy
keeper-0
StatefulSet
keeper-1 keeper-2
master standby
standby
postgresql postgresql postgresql
репликация
репликация
Consistent
Switchover
Slide 261
Slide 261 text
etcd-0
StatefulSet
etcd-1 etcd-2
etcd
sentinel
Deployment
sentinel
proxy
Deployment
proxy
keeper-0
StatefulSet
keeper-1 keeper-2
master standby
standby
postgresql postgresql postgresql
репликация
репликация
Consistent
Switchover
Slide 262
Slide 262 text
etcd-0
StatefulSet
etcd-1 etcd-2
etcd
sentinel
Deployment
sentinel
proxy
Deployment
proxy
keeper-0
StatefulSet
keeper-1 keeper-2
master standby
standby
postgresql postgresql postgresql
репликация
репликация
Consistent
Switchover
Slide 263
Slide 263 text
etcd-0
StatefulSet
etcd-1 etcd-2
etcd
sentinel
Deployment
sentinel
proxy
Deployment
proxy
postgresql
keeper-0
StatefulSet
keeper-1 keeper-2
master standby
standby
postgresql postgresql postgresql
репликация
репликация
Consistent
Switchover
Slide 264
Slide 264 text
app
etcd-0
StatefulSet
etcd-1 etcd-2
etcd
sentinel
Deployment
sentinel
proxy
Deployment
proxy
postgresql
keeper-0
StatefulSet
keeper-1 keeper-2
master standby
standby
postgresql postgresql postgresql
репликация
репликация
Consistent
Switchover
Slide 265
Slide 265 text
app
etcd-0
StatefulSet
etcd-1 etcd-2
etcd
sentinel
Deployment
sentinel
proxy
Deployment
proxy
postgresql
keeper-0
StatefulSet
keeper-1 keeper-2
master standby
standby
postgresql postgresql postgresql
репликация
репликация
Consistent
Switchover
cattle
pets
Slide 266
Slide 266 text
app
etcd-0
StatefulSet
etcd-1 etcd-2
etcd
sentinel
Deployment
sentinel
proxy
Deployment
proxy
postgresql
keeper-0
StatefulSet
keeper-1 keeper-2
master standby
standby
postgresql postgresql postgresql
репликация
репликация
Consistent
Switchover
Slide 267
Slide 267 text
No content
Slide 268
Slide 268 text
RAID MySQL
keepalived
IP
RAID MySQL
keepalived
IP
app
etcd-0
StatefulSet
etcd-1 etcd-2
etcd
sentinel
Deployment
sentinel
proxy
Deployment
proxy
postgresql
kpr-0
StatefulSet
kpr-1 kpr-2
master stndby
stndby
репликация репликация
Slide 269
Slide 269 text
github.com/flant/grafana-statusmap
grafana-statusmap
Grafana panel plugin to visualize status
of multiple objects over time.
Звезданите нас, пожалуйста!
Slide 270
Slide 270 text
github.com/flant/dapp
dapp
И тут тоже звезданите, плиз!
95% документации
переведено на английский
Перевод на GO
готов на 60%
Добавлена поддержка
yaml и ansible
Увеличена скорость работы с git
и размер repo больше не влияет на потребление памяти
Увеличена скорость работы с docker-registry
и добавлена поддержка Google Container Registry
Была придумана before-hook-creation hook delete policy,
которая была впоследствии добавлена в helm
Slide 271
Slide 271 text
Наши Docker/K8s-проекты
github.com/flant/dapp
github.com/flant/loghouse
Дмитрий Столяров
[email protected]
linkedin.com/in/distol
github.com/distol
Всем спасибо!
Наш блог на Хабре
habr.com/company/flant/
Наш YouTube-канал
youtube.com/c/flant