Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Базы данных и Kubernetes

flant
December 04, 2018

Базы данных и Kubernetes

Доклад технического директора компании «Флант» (http://flant.ru/) Дмитрия Столярова на конференции Highload++ 2018 в Москве.

* Текстовый обзор доклада: https://habr.com/company/flant/blog/431500/
* Видео с выступления: https://youtu.be/BnegHj53pW4

flant

December 04, 2018
Tweet

More Decks by flant

Other Decks in Technology

Transcript

  1. А можно базу в Kubernetes? ? ? ? ? ?

    ? В целом да, но... !
  2. ? ? ? ? ? ? В целом да, но...

    ! А можно базу в Kubernetes?
  3. ? ? ? ? ? ? В целом да, но...

    ! А можно базу в Kubernetes? stateful
  4. ? ? ? ? ? ? А можно базу в

    Kubernetes? stateful Зачем он нам в Kubernetes?
  5. ? ? ? ? ? ? А можно базу в

    Kubernetes? stateful Зачем он нам в Kubernetes? О каком stateful идет речь?
  6. О каком stateful идет речь? Search Engine Object Storage Relational

    Database NoSQL Database In-memory Database Message Queue
  7. О каком stateful идет речь? Search Engine Object Storage Relational

    Database NoSQL Database In-memory Database Message Queue Configuration & Service Discovery
  8. О каком stateful идет речь? ... Search Engine Object Storage

    Relational Database NoSQL Database In-memory Database Message Queue Configuration & Service Discovery
  9. RAID MySQL keepalived IP RAID MySQL keepalived IP Механизмы high

    availability Deployment и StatefulSet PHP PHP ... PHP PHP
  10. RAID MySQL keepalived IP RAID MySQL keepalived IP Механизмы high

    availability Deployment и StatefulSet PodAntiAffinity PHP PHP ... PHP PHP
  11. RAID MySQL keepalived IP RAID MySQL keepalived IP Deployment и

    StatefulSet PodAntiAffinity PodDisruptionBudgets Механизмы high availability PHP PHP ... PHP PHP
  12. RAID MySQL keepalived IP RAID MySQL keepalived IP Deployment и

    StatefulSet PodAntiAffinity PodDisruptionBudgets Механизмы high availability PHP PHP ... PHP PHP
  13. RAID MySQL keepalived IP RAID MySQL keepalived IP Deployment и

    StatefulSet PodAntiAffinity PodDisruptionBudgets Механизмы high availability PHP PHP ... PHP PHP
  14. RAID MySQL keepalived IP RAID MySQL keepalived IP Deployment и

    StatefulSet PodAntiAffinity PodDisruptionBudgets Механизмы high availability PHP PHP ... PHP
  15. RAID MySQL keepalived IP RAID MySQL keepalived IP Deployment и

    StatefulSet PodAntiAffinity PodDisruptionBudgets Механизмы high availability PHP PHP ... PHP PHP
  16. RAID MySQL keepalived IP RAID MySQL keepalived IP Deployment и

    StatefulSet PodAntiAffinity PodDisruptionBudgets Механизмы high availability PHP PHP ... PHP PHP
  17. App App App App App DB IP DB IP App

    App App App App keepalived keepalived Как оно было раньше?
  18. App App App App App DB IP DB IP App

    App App App App Balancer Balancer keepalived keepalived Как оно было раньше?
  19. App App App App App DB IP DB IP App

    App App App App Balancer Balancer keepalived keepalived Как оно было раньше?
  20. App App App App App DB IP DB IP App

    App App App App Balancer Balancer keepalived keepalived Как оно было раньше?
  21. App App App App App DB IP DB IP App

    App App App App Balancer Balancer keepalived keepalived Как оно было раньше?
  22. App App App App App DB IP DB IP App

    App App App App Balancer Balancer keepalived keepalived Как оно было раньше?
  23. App App App App App DB IP DB IP App

    App App App App Balancer Balancer keepalived keepalived Как оно было раньше?
  24. App App App App App DB IP DB IP App

    App App App App Balancer Balancer keepalived keepalived IP Как оно было раньше?
  25. App App App App App DB IP DB IP App

    App App App App Balancer Balancer keepalived keepalived IP Как оно было раньше?
  26. App App App App App DB IP DB IP App

    App App App App Balancer Balancer keepalived keepalived Как оно было раньше?
  27. App App App App App DB IP DB IP App

    App App App App Balancer Balancer Как оно было раньше?
  28. App App App App App DB IP DB IP App

    App App App App Balancer Balancer corosync corosync corosync Как оно было раньше?
  29. App App App App App DB IP DB IP App

    App App App App Balancer Balancer corosync corosync corosync Как оно было раньше?
  30. App App App App App DB IP DB IP App

    App App App App Balancer Balancer corosync corosync corosync Как оно было раньше?
  31. App App App App App DB IP DB IP App

    App App App App Balancer Balancer corosync corosync corosync Как оно было раньше?
  32. App App App App App DB IP DB IP App

    App App App App Balancer Balancer corosync corosync corosync Как оно было раньше?
  33. App App App App App DB IP DB IP App

    App App App App Balancer Balancer corosync corosync corosync Как оно было раньше?
  34. App App App App App DB IP App App App

    App App Balancer Balancer corosync corosync DB IP corosync Как оно было раньше?
  35. App App App App App App App App App App

    Balancer Balancer DB IP corosync Как оно было раньше? DB IP corosync corosync
  36. App App App App App App App App App App

    Balancer Balancer DB IP corosync Как оно было раньше? DB IP corosync corosync 1
  37. App App App App App App App App App App

    Balancer Balancer DB IP corosync Как оно было раньше? DB IP corosync IP 2 corosync 2 1
  38. App App App App App App App App App App

    Balancer Balancer corosync DB IP corosync Как оно было раньше? DB IP corosync IP
  39. App App App App App App App App App App

    Balancer Balancer corosync DB IP corosync Как оно было раньше? DB IP corosync IP
  40. App App App App App App App App App App

    Balancer Balancer corosync DB IP corosync Как оно было раньше? DB IP corosync IP At-most-once guarantee
  41. app-59ptg app-xfvr7 app-ck9cm Deployment StatefulSet А как там в Kubernetes?

    db-1 app-csm52 Может продолжить писать в базу! db-0
  42. app-csm52 app-59ptg app-xfvr7 app-ck9cm Deployment StatefulSet А как там в

    Kubernetes? At-most-once guarantee но также требуется fencing db-0 db-1
  43. app-csm52 app-59ptg app-xfvr7 app-ck9cm Deployment StatefulSet А как там в

    Kubernetes? At-most-once guarantee но так же требуется fencing db-0 db-1 Все еще требуется fencing!!
  44. app-csm52 app-59ptg app-xfvr7 app-ck9cm Deployment StatefulSet А как там в

    Kubernetes? At-most-once guarantee но так же требуется fencing db-0 db-1 Все еще требуется fencing!! ...или?
  45. DB IP DB IP App App App App App App

    App App App App Balancer Balancer Другой подход
  46. DB IP DB IP App App App App App App

    App App App App Balancer Balancer Другой подход
  47. DB IP DB IP App App App App App App

    App App App App Balancer Balancer Другой подход LB
  48. DB IP DB IP App App App App App App

    App App App App Balancer Balancer Другой подход LB
  49. DB IP DB IP App App App App App App

    App App App App Balancer Balancer Другой подход LB
  50. DB IP DB IP App App App App App App

    App App App App Balancer Balancer Другой подход LB
  51. DB IP DB IP App App App App App App

    App App App App Balancer Balancer Другой подход LB
  52. DB IP DB IP App App App App App App

    App App App App Balancer Balancer Другой подход LB
  53. DB IP DB IP App App App App App App

    App App App App Balancer Balancer Другой подход LB “Consistent Switchover”
  54. DB IP DB IP App App App App App App

    App App App App Balancer Balancer Другой подход LB Но как сделать generic?
  55. DB IP DB IP App App App App App App

    App App App App Balancer Balancer Другой подход
  56. App App App App App DB IP DB IP App

    P App P App P App P App P Balancer Balancer Другой подход
  57. App App App App App DB IP DB IP App

    P App P App P App P App P Balancer Balancer Другой подход
  58. App App App App App DB IP DB IP App

    P App P App P App P App P Balancer Balancer Другой подход
  59. App App App App App DB IP DB IP App

    P App P App P App P App P Balancer Balancer Другой подход
  60. App App App App App DB IP DB IP App

    P App P App P App P App P Balancer Balancer Другой подход
  61. App App App App App DB IP DB IP App

    P App P App P App P App P Balancer Balancer Другой подход
  62. App App App App App DB IP DB IP App

    P App P App P App P App P Balancer Balancer Другой подход
  63. App App App App App DB IP DB IP App

    P App P App P App P App P Balancer Balancer Другой подход
  64. App App App App App DB IP DB IP App

    P App P App P App P App P Balancer Balancer Другой подход
  65. App App App App App DB IP DB IP App

    P App P App P App P App P Balancer Balancer Другой подход
  66. App App App App App DB IP DB IP App

    P App P App P App P App P Balancer Balancer Другой подход
  67. App App App App App DB IP DB IP App

    P App P App P App P App P Balancer Balancer Другой подход
  68. App App App App App DB IP DB IP App

    P App P App P App P App P Balancer Balancer “Consistent Switchover” Другой подход
  69. Network Block Device aka SAN ReadWriteOnce Kubernetes Access Mode: Public

    cloud: AzureDisk AWSElasticBlockStore GCEPersistentDisk
  70. Network Block Device aka SAN ReadWriteOnce Kubernetes Access Mode: Public

    cloud: AzureDisk AWSElasticBlockStore GCEPersistentDisk Private cloud: Cinder VsphereVolume
  71. Network Block Device aka SAN ReadWriteOnce Kubernetes Access Mode: Public

    cloud: AzureDisk AWSElasticBlockStore GCEPersistentDisk Private cloud: Cinder VsphereVolume Bare metal or private cloud: Ceph RBD Fibre Channel iSCSI
  72. IOPS & Latency NVMe MySQL RDMA NVMe MySQL AWS EBS

    MySQL Commodity iSCSI MySQL 100µs 1ms 2-5ms 150µs
  73. IOPS & Latency NVMe MySQL RDMA NVMe MySQL AWS EBS

    MySQL Commodity iSCSI MySQL 100µs 1ms 2-5ms 150µs Performance Analysis of NVMe SSDs and their Implication on Real World Databases https://www.cs.utah.edu/~manua/pubs/systor15.pdf NVMe Performance Local vs. Remote https://www.samsung.com/us/labs/pdfs/2016-fms-forum-f-22-nvmf-local-remote-v05.pdf
  74. IOPS & Latency NVMe MySQL RDMA NVMe MySQL AWS EBS

    MySQL Commodity iSCSI MySQL Допустим, базе данных требуется ~50K IOPS 100µs 1ms 2-5ms 150µs Performance Analysis of NVMe SSDs and their Implication on Real World Databases https://www.cs.utah.edu/~manua/pubs/systor15.pdf NVMe Performance Local vs. Remote https://www.samsung.com/us/labs/pdfs/2016-fms-forum-f-22-nvmf-local-remote-v05.pdf
  75. IOPS & Latency NVMe MySQL 100µs 5 threads RDMA NVMe

    MySQL 150µs AWS EBS MySQL 2-5ms Commodity iSCSI MySQL 1ms Performance Analysis of NVMe SSDs and their Implication on Real World Databases https://www.cs.utah.edu/~manua/pubs/systor15.pdf NVMe Performance Local vs. Remote https://www.samsung.com/us/labs/pdfs/2016-fms-forum-f-22-nvmf-local-remote-v05.pdf Допустим, базе данных требуется ~50K IOPS
  76. IOPS & Latency Performance Analysis of NVMe SSDs and their

    Implication on Real World Databases https://www.cs.utah.edu/~manua/pubs/systor15.pdf NVMe Performance Local vs. Remote https://www.samsung.com/us/labs/pdfs/2016-fms-forum-f-22-nvmf-local-remote-v05.pdf NVMe MySQL 100µs 5 threads RDMA NVMe MySQL 150µs 7.5 threads AWS EBS MySQL 2-5ms Commodity iSCSI MySQL 1ms Допустим, базе данных требуется ~50K IOPS
  77. IOPS & Latency Performance Analysis of NVMe SSDs and their

    Implication on Real World Databases https://www.cs.utah.edu/~manua/pubs/systor15.pdf NVMe Performance Local vs. Remote https://www.samsung.com/us/labs/pdfs/2016-fms-forum-f-22-nvmf-local-remote-v05.pdf NVMe MySQL 100µs 5 threads RDMA NVMe MySQL 150µs 7.5 threads AWS EBS MySQL 2-5ms Commodity iSCSI MySQL 1ms 50 threads Допустим, базе данных требуется ~50K IOPS
  78. IOPS & Latency Performance Analysis of NVMe SSDs and their

    Implication on Real World Databases https://www.cs.utah.edu/~manua/pubs/systor15.pdf NVMe Performance Local vs. Remote https://www.samsung.com/us/labs/pdfs/2016-fms-forum-f-22-nvmf-local-remote-v05.pdf NVMe MySQL 100µs 5 threads RDMA NVMe MySQL 150µs 7.5 threads AWS EBS MySQL 2-5ms 100-250 threads Commodity iSCSI MySQL 1ms 50 threads Допустим, базе данных требуется ~50K IOPS
  79. Network Block Device aka SAN ReadWriteOnce Kubernetes Access Mode: Public

    cloud: AzureDisk AWSElasticBlockStore GCEPersistentDisk Private cloud: Cinder VsphereVolume Bare metal or private cloud: Ceph RBD Fibre Channel iSCSI Local Storage ReadWriteOnce Kubernetes Access Mode: Это просто локальный диск: SATA / SAS / SSD / NVME
  80. Network Block Device aka SAN ReadWriteOnce Kubernetes Access Mode: Public

    cloud: AzureDisk AWSElasticBlockStore GCEPersistentDisk Private cloud: Cinder VsphereVolume Bare metal or private cloud: Ceph RBD Fibre Channel iSCSI Local Storage ReadWriteOnce Kubernetes Access Mode: Это просто локальный диск: SATA / SAS / SSD / NVME
  81. Network Block Device aka SAN ReadWriteOnce Kubernetes Access Mode: Public

    cloud: AzureDisk AWSElasticBlockStore GCEPersistentDisk Private cloud: Cinder VsphereVolume Bare metal or private cloud: Ceph RBD Fibre Channel iSCSI Local Storage ReadWriteOnce Kubernetes Access Mode: Это просто локальный диск: SATA / SAS / SSD / NVME
  82. Network Block Device Network File System aka SAN aka NAS

    ReadWriteOnce Kubernetes Access Mode: ReadWriteMany Kubernetes Access Mode: Public cloud: AzureDisk AWSElasticBlockStore GCEPersistentDisk Private cloud: Cinder VsphereVolume Bare metal or private cloud: Ceph RBD Fibre Channel iSCSI Local Storage ReadWriteOnce Kubernetes Access Mode: Это просто локальный диск: SATA / SAS / SSD / NVME
  83. Network Block Device Network File System aka SAN aka NAS

    ReadWriteOnce Kubernetes Access Mode: ReadWriteMany Kubernetes Access Mode: Public cloud: AzureDisk AWSElasticBlockStore GCEPersistentDisk Private cloud: Cinder VsphereVolume Bare metal or private cloud: Ceph RBD Fibre Channel iSCSI Public cloud: AzureFile AWSElasticFileSystem Local Storage ReadWriteOnce Kubernetes Access Mode: Это просто локальный диск: SATA / SAS / SSD / NVME
  84. Network Block Device Network File System aka SAN aka NAS

    ReadWriteOnce Kubernetes Access Mode: ReadWriteMany Kubernetes Access Mode: Public cloud: AzureDisk AWSElasticBlockStore GCEPersistentDisk Private cloud: Cinder VsphereVolume Bare metal or private cloud: Ceph RBD Fibre Channel iSCSI Public cloud: AzureFile AWSElasticFileSystem Bare metal or private cloud: CephFS Glusterfs NFS Local Storage ReadWriteOnce Kubernetes Access Mode: Это просто локальный диск: SATA / SAS / SSD / NVME
  85. Network Block Device Network File System aka SAN aka NAS

    ReadWriteOnce Kubernetes Access Mode: ReadWriteMany Kubernetes Access Mode: Public cloud: AzureDisk AWSElasticBlockStore GCEPersistentDisk Private cloud: Cinder VsphereVolume Bare metal or private cloud: Ceph RBD Fibre Channel iSCSI Public cloud: AzureFile AWSElasticFileSystem Bare metal or private cloud: CephFS Glusterfs NFS Local Storage ReadWriteOnce Kubernetes Access Mode: Это просто локальный диск: SATA / SAS / SSD / NVME
  86. 1. Как узлы узнают друг о друге? Kubernetes API DNS

    (несколько A-записей) Статически
  87. 1. Как узлы узнают друг о друге? Kubernetes API DNS

    (несколько A-записей) Статически Seed-узлы
  88. 1. Как узлы узнают друг о друге? Kubernetes API DNS

    (несколько A-записей) Статически Seed-узлы Сторонний Service Discovery
  89. app mg-1 mg-0 StatefulSet mongo mg-2 mc-0.mongo mc-1.mongo mc-2.mongo 2.

    Как подключается клиент?
  90. minio-1 minio-0 StatefulSet minio-2 cassandra-1 cassandra-0 StatefulSet cassandra-2 ... 3.

    Горизонтальное масштабирование Нет Есть
  91. minio-1 minio-0 StatefulSet minio-2 cassandra-1 cassandra-0 StatefulSet cassandra-2 ... mongo-a-1

    mongo-a-0 StatefulSet mongo-a-2 mongo-a-1 mongo-a-0 StatefulSet mongo-a-2 ... 3. Горизонтальное масштабирование Нет Есть Есть #2
  92. etcd-0 StatefulSet etcd-1 etcd-2 etcd sentinel Deployment sentinel keeper-0 StatefulSet

    keeper-1 keeper-2 master postgresql postgresql postgresql
  93. etcd-0 StatefulSet etcd-1 etcd-2 etcd sentinel Deployment sentinel keeper-0 StatefulSet

    keeper-1 keeper-2 master standby standby postgresql postgresql postgresql
  94. etcd-0 StatefulSet etcd-1 etcd-2 etcd sentinel Deployment sentinel keeper-0 StatefulSet

    keeper-1 keeper-2 master standby standby postgresql postgresql postgresql
  95. etcd-0 StatefulSet etcd-1 etcd-2 etcd sentinel Deployment sentinel keeper-0 StatefulSet

    keeper-1 keeper-2 master standby standby postgresql postgresql postgresql репликация репликация
  96. etcd-0 StatefulSet etcd-1 etcd-2 etcd sentinel Deployment sentinel proxy Deployment

    proxy keeper-0 StatefulSet keeper-1 keeper-2 master standby standby postgresql postgresql postgresql репликация репликация
  97. etcd-0 StatefulSet etcd-1 etcd-2 etcd sentinel Deployment sentinel proxy Deployment

    proxy keeper-0 StatefulSet keeper-1 keeper-2 master standby standby postgresql postgresql postgresql репликация репликация Consistent Switchover
  98. etcd-0 StatefulSet etcd-1 etcd-2 etcd sentinel Deployment sentinel proxy Deployment

    proxy keeper-0 StatefulSet keeper-1 keeper-2 master standby standby postgresql postgresql postgresql репликация репликация Consistent Switchover
  99. etcd-0 StatefulSet etcd-1 etcd-2 etcd sentinel Deployment sentinel proxy Deployment

    proxy keeper-0 StatefulSet keeper-1 keeper-2 master standby standby postgresql postgresql postgresql репликация репликация Consistent Switchover
  100. etcd-0 StatefulSet etcd-1 etcd-2 etcd sentinel Deployment sentinel proxy Deployment

    proxy postgresql keeper-0 StatefulSet keeper-1 keeper-2 master standby standby postgresql postgresql postgresql репликация репликация Consistent Switchover
  101. app etcd-0 StatefulSet etcd-1 etcd-2 etcd sentinel Deployment sentinel proxy

    Deployment proxy postgresql keeper-0 StatefulSet keeper-1 keeper-2 master standby standby postgresql postgresql postgresql репликация репликация Consistent Switchover
  102. app etcd-0 StatefulSet etcd-1 etcd-2 etcd sentinel Deployment sentinel proxy

    Deployment proxy postgresql keeper-0 StatefulSet keeper-1 keeper-2 master standby standby postgresql postgresql postgresql репликация репликация Consistent Switchover cattle pets
  103. app etcd-0 StatefulSet etcd-1 etcd-2 etcd sentinel Deployment sentinel proxy

    Deployment proxy postgresql keeper-0 StatefulSet keeper-1 keeper-2 master standby standby postgresql postgresql postgresql репликация репликация Consistent Switchover
  104. RAID MySQL keepalived IP RAID MySQL keepalived IP app etcd-0

    StatefulSet etcd-1 etcd-2 etcd sentinel Deployment sentinel proxy Deployment proxy postgresql kpr-0 StatefulSet kpr-1 kpr-2 master stndby stndby репликация репликация
  105. github.com/flant/grafana-statusmap grafana-statusmap Grafana panel plugin to visualize status of multiple

    objects over time. Звезданите нас, пожалуйста!
  106. github.com/flant/dapp dapp И тут тоже звезданите, плиз! 95% документации переведено

    на английский Перевод на GO готов на 60% Добавлена поддержка yaml и ansible Увеличена скорость работы с git и размер repo больше не влияет на потребление памяти Увеличена скорость работы с docker-registry и добавлена поддержка Google Container Registry Была придумана before-hook-creation hook delete policy, которая была впоследствии добавлена в helm