Автомасштабирование и управление ресурсами в Kubernetes

93aef1d166a8a3536538eff713f80307?s=47 flant
April 27, 2019

Автомасштабирование и управление ресурсами в Kubernetes

Доклад технического директора компании «Флант» (https://flant.ru/) Дмитрия Столярова на секции «DevOps» конференции «Стачка-2019» в Ульяновске.

* Текстовый обзор доклада: https://habr.com/company/flant/blog/459326/
* Видео с выступления: https://youtu.be/10ZR-fbyuSY

93aef1d166a8a3536538eff713f80307?s=128

flant

April 27, 2019
Tweet

Transcript

  1. Автомасштабирование и управление ресурсами в Kubernetes Дмитрий Столяров И Ленин

    такой молодой, И юный — Октябрь впереди!
  2. Дмитрий Столяров CTO и соучредитель

  3. Андрей Климентьев Андрей Сидоров

  4. Андрей Климентьев Андрей Половов Андрей Сидоров

  5. Антон Климов Андрей Климентьев Андрей Половов Андрей Сидоров

  6. flant.ru habr.com/ru/company/flant youtube.com/c/Флант github.com/flant DevOps-профессионалы t.me/flant_ru

  7. Автомасштабирование и управление ресурсами в Kubernetes

  8. Автомасштабирование и управление ресурсами в Kubernetes

  9. None
  10. None
  11. Повторяемость

  12. Повторяемость Изоляция

  13. None
  14. None
  15. None
  16. ...

  17. ...

  18. ...

  19. ...

  20. ...

  21. ...

  22. ...

  23. ...

  24. ...

  25. ...

  26. ... Кластер контейнеров

  27. ... Кластер контейнеров

  28. ... Кластер контейнеров

  29. ... 4 Кластер контейнеров Высокоуровневые примитивы

  30. ... API 4 Кластер контейнеров Высокоуровневые примитивы Декларативное API

  31. ... API YAML 4 Кластер контейнеров Высокоуровневые примитивы Декларативное API

  32. Лучшие практики CI/CD с Docker youtu.be/FVFhqb1CbRc Наш опыт с Kubernetes

    в небольших проектах youtu.be/CgCLPYJRxbU Лучшие практики CI/CD с Kubernetes и GitLab youtu.be/G3nELxmECd8 Мониторинг и Kubernetes youtu.be/zj6SlzzBRaA Расширяем и дополняем Kubernetes youtu.be/6VHk1R1TNgk youtube.com/c/Флант Подписывайтесь на наш YouTube-канал!
  33. ... API YAML 4 Кластер контейнеров Высокоуровневые примитивы Декларативное API

  34. Автомасштабирование и управление ресурсами в Kubernetes

  35. Автомасштабирование и управление ресурсами в Kubernetes

  36. CPU

  37. None
  38. CPU

  39. CPU $ htop ==============================

  40. CPU $ htop ============================== nginx

  41. CPU $ htop ============================== nginx php-fpm

  42. CPU $ htop ============================== nginx php-fpm mysql

  43. CPU $ htop ============================== nginx php-fpm mysql

  44. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    php-fpm mysql
  45. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    php-fpm mysql
  46. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker mysql
  47. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker mysql 5 71 74 6 205 217 213 209 460
  48. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker mysql 5 71 74 6 205 217 213 209 460
  49. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    5 71 74 $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 $ htop ============================== mysql 460
  50. А что такое Docker?

  51. А что такое Docker? контейнер

  52. А что такое контейнер?

  53. capabilities 2.2 / 1999 А что такое контейнер?

  54. capabilities 2.2 / 1999 namespaces 2.6.19 / nov 2006 А

    что такое контейнер?
  55. capabilities 2.2 / 1999 namespaces 2.6.19 / nov 2006 cgroups

    2.6.24 / jan 2008 А что такое контейнер?
  56. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov

    2006 cgroups 2.6.24 / jan 2008 aufs 2007 А что такое контейнер? overlay2 2015 zfs 2013 btrfs 2013
  57. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov

    2006 cgroups 2.6.24 / jan 2008 aufs 2007 А что такое контейнер? overlay2 2015 zfs 2013 btrfs 2013 veth 2007 ipvlan 2014 macvlan 2007
  58. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov

    2006 cgroups 2.6.24 / jan 2008 aufs 2007 kernel А что такое контейнер? overlay2 2015 zfs 2013 btrfs 2013 veth 2007 ipvlan 2014 macvlan 2007
  59. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov

    2006 cgroups 2.6.24 / jan 2008 aufs 2007 kernel А что такое контейнер? docker 2013 overlay2 2015 zfs 2013 btrfs 2013 veth 2007 ipvlan 2014 macvlan 2007
  60. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov

    2006 cgroups 2.6.24 / jan 2008 aufs 2007 kernel А что такое контейнер? docker 2013 rkt 2014 overlay2 2015 zfs 2013 btrfs 2013 veth 2007 ipvlan 2014 macvlan 2007
  61. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov

    2006 cgroups 2.6.24 / jan 2008 aufs 2007 kernel А что такое контейнер? docker 2013 rkt 2014 cri-o 2017 overlay2 2015 zfs 2013 btrfs 2013 veth 2007 ipvlan 2014 macvlan 2007
  62. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov

    2006 cgroups 2.6.24 / jan 2008 aufs 2007 kernel А что такое контейнер? docker 2013 rkt 2014 cri-o 2017 podman 2018 overlay2 2015 zfs 2013 btrfs 2013 veth 2007 ipvlan 2014 macvlan 2007
  63. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov

    2006 cgroups 2.6.24 / jan 2008 aufs 2007 kernel А что такое контейнер? docker 2013 rkt 2014 cri-o 2017 podman 2018 overlay2 2015 zfs 2013 btrfs 2013 veth 2007 ipvlan 2014 macvlan 2007
  64. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    5 71 74 $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 $ htop ============================== mysql 460
  65. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    5 71 74 $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 $ htop ============================== mysql 460
  66. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    5 71 74 $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 $ htop ============================== mysql 460 1000
  67. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 1000 150 850 460 5 71 74 6 205 217 213 209 460
  68. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 150 850 460 5 71 74 6 205 217 213 209 460 1000
  69. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 150 850 460 5 71 74 6 205 217 213 209 460 1000 333 333 333
  70. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 333 333 333
  71. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 333 333 333
  72. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 183 333 150 - = 333 333
  73. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 183 333 150 - = 333 333
  74. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 183 333 150 - = 333 91 + 333 91 +
  75. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 183 333 150 - = 333 91 424 + = 333 91 424 + =
  76. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 2 3 5
  77. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 150 850 460 2 3 5 300 200 500 1000
  78. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 2 3 5 300 200 500
  79. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 200 500 2 3 5 300
  80. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 50 200 150 - = 40 500 460 - = 2 3 5 300
  81. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 50 200 150 - = 300 40 500 460 - = 2 3 5
  82. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 50 200 150 - = 300 40 + 40 500 460 - = 2 3 5 50 +
  83. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 50 200 150 - = 300 40 390 + = 40 500 460 - = 2 3 5 50 +
  84. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== mysql 5 71 74 460 1000 150 460 50 200 150 - = 300 40 + = 40 500 460 - = 2 5 50 + $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 850 390 3
  85. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== mysql 5 71 74 460 1000 150 460 50 200 150 - = 300 40 + = 40 500 460 - = 2 5 50 + $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 850 390 3 78 78 78 78 78
  86. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== mysql 5 71 74 460 1000 150 460 50 200 150 - = 300 40 + = 40 500 460 - = 2 5 50 + $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 850 390 3 6 78 78 78 78 72 78 = -
  87. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== mysql 5 71 74 460 1000 150 460 50 200 150 - = 300 40 + = 40 500 460 - = 2 5 50 + $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 850 390 3 6 78 78 78 78 72 78 = -
  88. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== mysql 5 71 74 460 1000 150 460 50 200 150 - = 300 40 + = 40 500 460 - = 2 5 50 + $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 850 390 3 6 96 96 96 96 72 78 = -
  89. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== mysql 5 71 74 460 1000 150 460 50 200 150 - = 300 40 + = 40 500 460 - = 2 5 50 + $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 850 390 3 6 96 96 96 96 2.12 2.26 2.21 2.17 100%
  90. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 2 3 5 1000
  91. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 2 3 5 150 390 460 1000
  92. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 2 3 5 150 390 460 1000 6 96 96 96 96 5 71 74 460
  93. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 2 3 5 150 390 460 1000 6 96 96 96 96 2.12 2.26 2.21 2.17 100% 5 71 74 460 100% 100% 100% 100%
  94. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 150 390 460 1000 6 96 96 96 96 2.12 2.26 2.21 2.17 100% 5 71 74 460 100% 100% 100% 100% 2 3 5
  95. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 150 390 460 1000 6 96 96 96 96 2.12 2.26 2.21 2.17 100% 5 71 74 460 100% 100% 100% 100% 2 3 5 CFS = Completely Fair Scheduler
  96. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 150 390 460 1000 6 96 96 96 96 2.12 2.26 2.21 2.17 100% 5 71 74 460 100% 100% 100% 100% 2 3 5 Реквесты
  97. None
  98. None
  99. nginx mysql php

  100. nginx mysql php

  101. nginx mysql php

  102. 15 sec nginx mysql php

  103. 15 sec nginx mysql php

  104. 15 sec nginx mysql php

  105. nginx mysql php

  106. nginx mysql php

  107. 39 sec nginx mysql php

  108. 39 sec nginx mysql php

  109. 39 sec nginx mysql php

  110. 39 sec nginx mysql php

  111. 39 sec nginx mysql php

  112. 39 sec nginx mysql php

  113. 39 sec nginx mysql php

  114. 39 sec nginx mysql php

  115. 39 sec nginx mysql php

  116. 39 sec nginx mysql php

  117. nginx mysql php

  118. nginx mysql php

  119. nginx mysql php 46 sec

  120. nginx mysql php 46 sec

  121. nginx mysql php 46 sec

  122. nginx mysql php 46 sec

  123. nginx mysql php 46 sec

  124. nginx mysql php 46 sec

  125. nginx mysql php 46 sec

  126. nginx mysql php 46 sec

  127. nginx mysql php

  128. nginx mysql php

  129. nginx mysql php

  130. nginx mysql php Лимиты

  131. nginx mysql php

  132. nginx mysql php 46 sec

  133. nginx mysql php 46 sec

  134. nginx mysql php 46 sec

  135. nginx mysql php 46 sec

  136. nginx mysql php 46 sec

  137. nginx mysql php 46 sec

  138. nginx mysql php 46 sec Куплю машину ближе к светофору!

  139. nginx mysql php 46 sec Куплю машину поближе к светофору!

  140. CPU

  141. kernel CPU

  142. kernel CPU CFS

  143. kernel CPU CFS cgroup

  144. kernel CPU CFS cgroup

  145. kernel CPU CFS cgroup

  146. kernel CPU CFS cgroup quota

  147. kernel CPU CFS cgroup quota shares

  148. kernel CPU Крутилки CFS cgroup quota shares

  149. kernel CPU Крутилки CFS cgroup quota shares Реквест Лимит

  150. В чём мерить CPU?

  151. в попугаях В чём мерить CPU?

  152. в попугаях В чём мерить CPU? в процентах

  153. в попугаях в процентах В чём мерить CPU? в весе

  154. в попугаях в весе в процентах В чём мерить CPU?

    в секундах
  155. в секундах в попугаях в весе в процентах В чём

    мерить CPU? в ядрах! N
  156. ? N

  157. ? N

  158. ? N

  159. ? N

  160. ? N

  161. ? N

  162. ? N 500

  163. ? N 500 1000

  164. ? N 500 1000 1500

  165. ? N 500 1000 1500 ? ? ?

  166. ? N 500 1000 1500 ? ? ?

  167. ? N 1000 1500 ? ? ? 500

  168. ? N 1000 1500 ? ? ? 500 ( )

  169. ? N ? ? ? 500 ( ) 1000 1500

  170. ? N ? ? 1000 1500 ? 500 3000

  171. ? N ? 1000 1500 ? ? 500 3000

  172. ? N ? 1000 1500 ? ? 500 3000 3

    x
  173. ? N ? 1000 1500 ? 0.5 500 3000 3

    x
  174. ? N ? 1000 1500 ? 0.5 500

  175. ? N ? 1000 1500 ? 0.5 500

  176. ? N 1.5 1000 1500 1 0.5 500

  177. ? N 1.5 1000 1500 1 0.5 500

  178. ? N 1.5 1000 1500 1 0.5 500 ? 1500

    ? 500 1000 ?
  179. ? N 1000 1500 500 ? 1500 ? 500 1.5

    1 0.5 1000 ?
  180. ? N 1000 1500 500 3 1500 1 500 1.5

    1 0.5 1000 2
  181. ? N 1000 1500 500 1.5 1 0.5 3 1500

    1 500 1000 2
  182. ? N 1000 1500 500 1.5 1 0.5 3 1000

    1500 2 1 500 3000 ?
  183. ? N 1000 1500 500 1.5 1 0.5 3 1000

    1500 2 1 500 3000 3
  184. ? N 1000 1500 500 1.5 1 0.5 3 1000

    1500 2 1 500 3000 3
  185. ? N 1000 1500 500 1.5 1 0.5 1.5 1000

    1500 1 0.5 500 3000 3
  186. ? N 1000 1500 500 1.5 1000 1500 1 0.5

    500 1.5 1 0.5 3000 3
  187. ? N 1 миллиядро = 1 cpu share* *0 миллиядер

    = 2 cpu share *На самом деле 1000 миллиядер = 1024 cpu shares
  188. None
  189. None
  190. allocatable: cpu: "8"

  191. allocatable: cpu: "8"

  192. allocatable: cpu: "8"

  193. allocatable: cpu: "8"

  194. allocatable: cpu: "8"

  195. allocatable: cpu: "4" allocatable: cpu: "8"

  196. allocatable: cpu: "4" allocatable: cpu: "8"

  197. allocatable: cpu: "4" allocatable: cpu: "6" allocatable: cpu: "8"

  198. allocatable: cpu: "4" allocatable: cpu: "6" allocatable: cpu: "8"

  199. None
  200. None
  201. cpu: 2500m

  202. cpu: 2500m

  203. cpu: 2500m

  204. cpu: 2500m

  205. А что если мы не указали никакой request?

  206. 100% ресурсов

  207. Request = Limit Request = Limit Guaranteed

  208. Request = 100m Limit = 200m Request = Limit Request

    = Limit Burstable Guaranteed
  209. Request = 100m Limit = 200m Request = Limit Request

    = Limit Burstable Guaranteed
  210. Request = 100m Limit = 200m Limit = 200m Request

    = 100m Request = 100m Limit = 200m Limit = 200m Request = 100m Request = Limit Request = Limit Burstable Guaranteed
  211. Request = 100m Limit = 200m Limit = 200m Request

    = 100m Request = 100m Limit = 200m Limit = 200m Request = 100m Request = Limit Request = Limit Burstable Best Effort Guaranteed
  212. None
  213. Memory

  214. kernel CPU Крутилки CFS cgroup quota shares Реквест Лимит

  215. kernel CPU CFS cgroup quota shares Реквест Лимит

  216. kernel CPU CFS cgroup quota shares kernel Memory mm subsystem

    cgroup limit_in_bytes oom_score_adj Реквест Лимит Реквест Лимит
  217. kernel CPU cgroup quota shares kernel Memory cgroup limit_in_bytes oom_score_adj

    Реквест Лимит Реквест Лимит CFS mm subsystem
  218. kernel Memory Крутилки mm subsystem cgroup limit_in_bytes oom_score_adj Реквест Лимит

  219. kernel Memory Крутилки mm subsystem cgroup limit_in_bytes oom_score_adj Реквест Лимит

  220. None
  221. None
  222. None
  223. None
  224. None
  225. None
  226. None
  227. None
  228. None
  229. None
  230. None
  231. None
  232. None
  233. None
  234. None
  235. None
  236. None
  237. А как же вычисляется oom_score_adj?

  238. Request = 100m Limit = 200m Limit = 200m Request

    = 100m Request = 100m Limit = 200m Limit = 200m Request = 100m Request = Limit Request = Limit Burstable Best Effort Guaranteed
  239. Burstable Best Effort Guaranteed Request = Limit Request = 100m

    Limit = 200m QoS Class
  240. Burstable Best Effort Guaranteed Request = Limit Request = 100m

    Limit = 200m QoS Class oom_score_adj
  241. Burstable Best Effort Guaranteed Request = Limit Request = 100m

    Limit = 200m QoS Class oom_score_adj -998
  242. Burstable Best Effort Guaranteed Request = Limit Request = 100m

    Limit = 200m QoS Class oom_score_adj -998 1000
  243. Burstable Best Effort Guaranteed Request = Limit Request = 100m

    Limit = 200m QoS Class oom_score_adj -998 min(max(2, 1000 - (1000*memoryRequestBytes)/ machineMemoryCapacityBytes), 999) 1000
  244. Burstable Best Effort Guaranteed Request = Limit Request = 100m

    Limit = 200m QoS Class oom_score_adj -998 min(max(2, 1000 - (1000*memoryRequestBytes)/ machineMemoryCapacityBytes), 999) 1000 Обратно пропорционально request ÷ capacity
  245. Реквест Лимит kernel Memory mm subsystem cgroup limit_in_bytes oom_score_adj Крутилки

  246. Реквест Лимит kernel Memory mm subsystem cgroup limit_in_bytes oom_score_adj Крутилки

  247. None
  248. None
  249. None
  250. None
  251. None
  252. None
  253. None
  254. None
  255. None
  256. None
  257. None
  258. None
  259. Итого...

  260. requests: cpu: 250m memory: 5G limits: cpu: 500m memory: 10G

  261. requests: cpu: 250m memory: 5G limits: cpu: 500m memory: 10G

  262. requests: cpu: 250m memory: 5G limits: cpu: 500m memory: 10G

  263. requests: cpu: 250m memory: 5G limits: cpu: 500m memory: 10G

  264. requests: cpu: 250m memory: 5G limits: cpu: 500m memory: 10G

  265. requests: cpu: 250m memory: 5G limits: cpu: 500m memory: 10G

  266. requests: cpu: 250m memory: 5G limits: cpu: 500m memory: 10G

  267. requests: cpu: 250m memory: 5G limits: cpu: 500m memory: 10G

  268. requests: cpu: 250m memory: 5G limits: cpu: 500m memory: 10G

  269. Автомасштабирование и управление ресурсами в Kubernetes

  270. Автомасштабирование и управление ресурсами в Kubernetes

  271. Автомасштабирование и управление ресурсами в Kubernetes

  272. Автомасштабирование и управление ресурсами в Kubernetes

  273. Автомасштабирование и управление ресурсами в Kubernetes

  274. Автомасштабирование и управление ресурсами в Kubernetes

  275. None
  276. None
  277. None
  278. cpu: 2500m

  279. cpu: 2500m Pending...

  280. cpu: 2500m Pending...

  281. allocatable: cpu: "8" cpu: 2500m Pending...

  282. cpu: 2500m Pending...

  283. cpu: 2500m Pending...

  284. None
  285. None
  286. (Авто)масштабирование кластера

  287. None
  288. None
  289. None
  290. None
  291. None
  292. None
  293. None
  294. None
  295. None
  296. None
  297. None
  298. None
  299. None
  300. Но...

  301. None
  302. Нельзя так просто взять и...

  303. None
  304. None
  305. None
  306. Deployment 6

  307. app-59ptg app-xfvr7 app-ck9cm Deployment app-4ktms 6 app-d38nl app-vq1g5

  308. app-59ptg app-xfvr7 app-ck9cm Deployment app-4ktms 6 app-d38nl app-vq1g5

  309. app-59ptg app-xfvr7 app-ck9cm Deployment app-4ktms $ kubectl drain node-green 6

    app-d38nl app-vq1g5
  310. app-59ptg app-xfvr7 app-ck9cm Deployment app-4ktms $ kubectl drain node-green 6

    app-d38nl app-vq1g5
  311. app-59ptg app-xfvr7 app-ck9cm Deployment app-4ktms $ kubectl drain node-green 6

    app-d38nl app-vq1g5
  312. app-59ptg app-xfvr7 app-ck9cm Deployment app-4ktms $ kubectl drain node-green app-d38nl

    app-vq1g5 6
  313. app-59ptg app-xfvr7 app-ck9cm Deployment app-4ktms $ kubectl drain node-green app-d38nl

    app-vq1g5 6
  314. app-59ptg app-xfvr7 app-ck9cm Deployment app-4ktms $ kubectl drain node-green app-heb0y

    app-my9sq 6
  315. app-59ptg app-xfvr7 app-ck9cm Deployment app-4ktms $ kubectl drain node-green app-heb0y

    app-my9sq 6
  316. None
  317. None
  318. Statefulset 3

  319. Statefulset 3 mongo-1 mongo-3 mongo-2

  320. Statefulset 3 mongo-1 mongo-3 mongo-2

  321. Statefulset 3 mongo-1 mongo-3 mongo-2 CrashLoopBackOff

  322. Statefulset 3 mongo-1 mongo-3 mongo-2 CrashLoopBackOff

  323. Statefulset $ kubectl drain node-green 3 mongo-1 mongo-3 mongo-2 CrashLoopBackOff

  324. Statefulset $ kubectl drain node-green 3 mongo-1 mongo-3 mongo-2 CrashLoopBackOff

  325. Statefulset $ kubectl drain node-green 3 mongo-1 mongo-3 mongo-2 CrashLoopBackOff

  326. Statefulset $ kubectl drain node-green 3 mongo-1 mongo-3 mongo-2 CrashLoopBackOff

    PodDisruptionBudget minAvailable: 2
  327. Statefulset $ kubectl drain node-green 3 mongo-1 mongo-3 PodDisruptionBudget minAvailable:

    2 mongo-2 CrashLoopBackOff
  328. Statefulset $ kubectl drain node-green 3 mongo-1 mongo-3 PodDisruptionBudget minAvailable:

    2 mongo-2 CrashLoopBackOff
  329. Statefulset $ kubectl drain node-green 3 CrashLoopBackOff mongo-1 mongo-3 PodDisruptionBudget

    minAvailable: 2 mongo-2
  330. Statefulset $ kubectl drain node-green 3 mongo-1 mongo-3 PodDisruptionBudget minAvailable:

    2 mongo-2 CrashLoopBackOff
  331. Statefulset $ kubectl drain node-green 3 mongo-1 mongo-3 PodDisruptionBudget minAvailable:

    2 mongo-2 CrashLoopBackOff
  332. Statefulset $ kubectl drain node-green 3 mongo-1 mongo-3 PodDisruptionBudget minAvailable:

    2 mongo-2
  333. Statefulset $ kubectl drain node-green 3 mongo-1 mongo-3 PodDisruptionBudget minAvailable:

    2 mongo-2
  334. Statefulset $ kubectl drain node-green 3 mongo-1 mongo-3 PodDisruptionBudget minAvailable:

    2 mongo-2
  335. Statefulset $ kubectl drain node-green 3 mongo-2 mongo-1 mongo-3 PodDisruptionBudget

    minAvailable: 2
  336. Statefulset $ kubectl drain node-green 3 mongo-2 mongo-1 mongo-3 PodDisruptionBudget

    minAvailable: 2
  337. Statefulset $ kubectl drain node-green $ kubectl uncordon node-green 3

    mongo-2 mongo-1 mongo-3 PodDisruptionBudget minAvailable: 2
  338. Statefulset $ kubectl drain node-green $ kubectl uncordon node-green 3

    mongo-2 mongo-1 mongo-3 PodDisruptionBudget minAvailable: 2
  339. None
  340. 1. Пересоздание

  341. 1. Пересоздание 2. PodDisruptionBudget

  342. ... 3-10 backend

  343. ... 3-10 backend stateful

  344. ... 3-10 backend stateful frontend

  345. ... dev ... 3-10 backend stateful frontend 5-20

  346. ... dev ... 0-30 test ... 3-10 backend stateful frontend

    5-20
  347. ... dev ... 0-30 test ... 3-10 backend stateful frontend

    5-20
  348. (Авто)масштабирование кластера

  349. None
  350. None
  351. 3

  352. 3

  353. 3

  354. 3

  355. 3

  356. 3

  357. 3

  358. 3

  359. 3

  360. 3 Ручное масштабирование?

  361. 3 Ручное масштабирование?

  362. 3

  363. 3

  364. 3

  365. 3

  366. 3

  367. 6

  368. 6

  369. 6

  370. 6

  371. 6

  372. 6

  373. 6

  374. 6

  375. 6

  376. 6

  377. 6

  378. 3

  379. 3

  380. 3

  381. 3

  382. 3

  383. 3 Горизонтальное (авто) масштабирование

  384. Что именно измерять?

  385. Что именно измерять? Как интерпретировать полученные значения?

  386. CPU Usage CPU Throttling Memory Usage OOM Системные RPS Connections

    Worker Pool Usage Уровня приложения Queue Length Response Time
  387. USE Utilization – потребление Saturation – пресыщение Errors – ошибки

  388. (Авто)масштабирование кластера Горизонтальное (авто)масштабирование

  389. 3

  390. 3

  391. 3

  392. requests: cpu: 500m 3

  393. 3 requests: cpu: 500m

  394. 3 requests: cpu: 500m

  395. 3 requests: cpu: 500m

  396. 3 requests: cpu: 1000m

  397. requests: cpu: 1000m 3

  398. requests: cpu: 1000m 3

  399. requests: cpu: 1000m 3

  400. requests: cpu: 1000m 3

  401. requests: cpu: 1000m 3

  402. requests: cpu: 1000m 3

  403. requests: cpu: 1000m 3

  404. requests: cpu: 1000m 3

  405. 6 3 requests: cpu: 1000m Вертикальное (авто) масштабирование

  406. (Авто)масштабирование кластера Горизонтальное (авто)масштабирование Вертикальное (авто)масштабирование

  407. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 150 390 460 1000 6 96 96 96 96 2.12 2.26 2.21 2.17 100% 5 71 74 460 100% 100% 100% 100% 2 3 5
  408. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 150 390 460 1000 6 96 96 96 96 2.12 2.26 2.21 2.17 100% 5 71 74 460 100% 100% 100% 100% 2 3 5
  409. (Авто)масштабирование кластера Горизонтальное (авто)масштабирование Вертикальное (авто)масштабирование

  410. 6 3

  411. 6 3

  412. 6 3

  413. 6 3 requests: cpu: 1000m

  414. 6 3 requests: cpu: 1000m

  415. 6 3 requests: cpu: 1000m

  416. 6 3 requests: cpu: 1000m

  417. 6 3 requests: cpu: 500m

  418. 6 3 requests: cpu: 500m

  419. 6 3 requests: cpu: 500m

  420. 6 3 requests: cpu: 500m

  421. 6 3 requests: cpu: 500m

  422. 6 3 requests: cpu: 500m

  423. 6 3 requests: cpu: 500m

  424. 6 5 requests: cpu: 500m

  425. 6 5 requests: cpu: 500m

  426. 6 5 requests: cpu: 500m

  427. (Авто)масштабирование кластера Горизонтальное (авто)масштабирование Вертикальное (авто)масштабирование

  428. None
  429. max(utilisation) при min(saturation)

  430. Спасибо! Наш блог на Хабре habr.com/company/flant Наш YouTube-канал youtube.com/c/Флант shell-operator

    github.com/flant/shell-operator addon-operator github.com/flant/addon-operator werf github.com/flant/werf grafana-statusmap github.com/flant/grafana-statusmap kubedog github.com/flant/kubedog Дмитрий Столяров dmitry.stolyarov@flant.com CTO и соучредитель