サービスのパフォーマンス数値と
依存関係を用いたサービス同士の
協調スケール構想 / Web System Architecture #1

E4619fc2a039391a1677beeac58dd487?s=47 itkq
December 23, 2017
450

サービスのパフォーマンス数値と
依存関係を用いたサービス同士の
協調スケール構想 / Web System Architecture #1

Web System Architecture 研究会 #1

E4619fc2a039391a1677beeac58dd487?s=128

itkq

December 23, 2017
Tweet

Transcript

  1. 6.

    എܠɿΞϓϦέʔγϣϯίϯςφ • ίϯςφܕԾ૝Խ • খΦʔόʔϔου • Container as a Service

    • Production ready • Docker, Kubernetes • Managed services (GKE, ECS, EKS) 5
  2. 9.

    എܠɿαʔϏεϝογϡͷ༻్ • Envoy, Linkerd • Advanced load balancing • Circuit

    breaking • Rate limiting • Service discovery • Observation • Statistics • Logging • Tracing 8 ։ൃج൫ͷྖҬʹۙ͘ɺӡ༻΁ͷԠ༻͸ߟ͑Δ༨஍͋Γ
  3. 10.

    ίϯςφɾϚΠΫϩαʔϏεͷ࣮ӡ༻՝୊ 1. ϗετΩϟύγςΟϓϥϯχϯά • ίϯςφͷਫฏεέʔϧ͸༰қ • ͔͠͠ϗετΩϟύγςΟ͸ඞཁ 2. ίϯςφϦιʔεͷܾఆ๏ •

    “desired count” ͸୭͕Ͳ͏ܾΊΔͷ͔ 3. αʔϏεಉ͕࢜ڠௐͨ͠εέʔϧ • ґଘઌαʔϏεͷΩϟύγςΟ΋ߟྀ͢Δඞཁ 9 ͍ͣΕ΋ΩϟύγςΟͷ໰୊ͱͯ͠ू໿Ͱ͖Δ
  4. 16.

    ՝୊Πϝʔδ 1 13 A B C user facing internal ×

    2 × 1 × 1 D System × 1 desired count: 6 ex. ϐʔΫ࣌
 A ͷෛՙ 3 ഒ desired count: 3
  5. 17.

    ՝୊Πϝʔδ 1 13 A B C user facing internal ×

    2 × 1 × 1 D System × 1 desired count: 6 × 6 ex. ϐʔΫ࣌
 A ͷෛՙ 3 ഒ × 3 desired count: 3 C ͷߟྀ࿙Ε
  6. 18.

    ՝୊Πϝʔδ 1 13 A B C user facing internal ×

    2 × 1 × 1 D System × 1 desired count: 6 × 6 ex. ϐʔΫ࣌
 A ͷෛՙ 3 ഒ C × 3 desired count: 3 C ͷߟྀ࿙Ε
  7. 19.

    ՝୊Πϝʔδ 1 13 A B C user facing internal ×

    2 × 1 × 1 D System × 1 desired count: 6 × 6 ex. ϐʔΫ࣌
 A ͷෛՙ 3 ഒ C A × 3 desired count: 3 C ͷߟྀ࿙Ε
  8. 20.

    ՝୊Πϝʔδ 2 14 A B C user facing internal ×

    2 × 1 × 1 D System × 1 desired count: 6 desired count: 3 desired count: 3 ex. ϐʔΫ࣌
 A ͷෛՙ 3 ഒ
  9. 21.

    ՝୊Πϝʔδ 2 14 A B C user facing internal ×

    2 × 1 × 1 D System × 1 desired count: 6 × 6 × 3 × 3 desired count: 3 desired count: 3 ex. ϐʔΫ࣌
 A ͷෛՙ 3 ഒ D ͷߟྀ࿙Ε
  10. 22.

    ՝୊Πϝʔδ 2 14 A B C user facing internal ×

    2 × 1 × 1 D System × 1 desired count: 6 × 6 B × 3 × 3 desired count: 3 desired count: 3 ex. ϐʔΫ࣌
 A ͷෛՙ 3 ഒ D ͷߟྀ࿙Ε
  11. 23.

    ՝୊Πϝʔδ 2 14 A B C user facing internal ×

    2 × 1 × 1 D System × 1 desired count: 6 × 6 B A × 3 × 3 desired count: 3 desired count: 3 ex. ϐʔΫ࣌
 A ͷෛՙ 3 ഒ D ͷߟྀ࿙Ε
  12. 24.

    ղܾΠϝʔδɿ૝ఆঢ়گ 15 A 100 rps/container A B C user facing

    internal × 2 × 1 × 1 D System × 1 B 200 rps/container C 100 rps/container D 100 rps/container 0.4 0.2 0.7
  13. 25.

    ղܾΠϝʔδ 16 A 100 rps/container A B C user facing

    internal × 2 × 1 × 1 D System × 1 B 200 rps/container C 100 rps/container D 100 rps/container 0.4 0.2 0.7 ex. ϐʔΫ࣌
 A ͷෛՙ 3 ഒ
  14. 26.

    ղܾΠϝʔδ 16 A 100 rps/container A B C user facing

    internal × 2 × 1 × 1 D System × 1 B 200 rps/container C 100 rps/container D 100 rps/container 0.4 0.2 0.7 ex. ϐʔΫ࣌
 A ͷෛՙ 3 ഒ × 6
  15. 27.

    ղܾΠϝʔδ 16 A 100 rps/container A B C user facing

    internal × 2 × 1 × 1 D System × 1 B 200 rps/container C 100 rps/container D 100 rps/container 0.4 0.2 0.7 ex. ϐʔΫ࣌
 A ͷෛՙ 3 ഒ (100 * 6 * 0.4 + 100 * 0.7) 
 / 200 ~ 2 (100 * 6 * 0.2) / 100 ~ 2 × 6 × 2 × 2 ύϑΥʔϚϯε਺஋ + ґଘ౓ → ඞཁ࠷௿ݶͷίϯςφ਺
  16. 28.

    ໰୊ɿύϑΥʔϚϯεఆྔԽͱґଘ౓ • ఆྔԽʹؔͯ͠ • ͲͷΑ͏ʹଌఆ͢Δ͔ • ϝτϦΫε͸ rps ͰΑ͍͔ •

    ͲͷΤϯυϙΠϯτʹର͢Δ rps ͔ • ϦΫΤετύλʔϯ͸ prod ͱಉҰ͔ • αʔϏεͷґଘ౓͸Ͳ͏ܾ·Δ͔ 17 ύϑΥʔϚϯεࣗಈଌఆɾఆྔԽͷ࣮ྫ͕গͳ͍
  17. 29.

    ໰୊ɿύϑΥʔϚϯεఆྔԽͱґଘ౓ • ఆྔԽʹؔͯ͠ • ͲͷΑ͏ʹଌఆ͢Δ͔ • ϝτϦΫε͸ rps ͰΑ͍͔ •

    ͲͷΤϯυϙΠϯτʹର͢Δ rps ͔ • ϦΫΤετύλʔϯ͸ prod ͱಉҰ͔ • αʔϏεͷґଘ౓͸Ͳ͏ܾ·Δ͔ 17 ύϑΥʔϚϯεࣗಈଌఆɾఆྔԽͷ࣮ྫ͕গͳ͍ αʔϏεϝογϡ + shadowing
  18. 30.

    αʔϏεϝογϡߏ੒ྫ 18 front-proxy Request B proxy A proxy stats discovery

    • connection / request count • 1xx, 2xx, … 5xx count • etc. per proxy metrics:
  19. 31.

    shadowing 19 front-proxy Request B proxy A proxy Prod Shadow

    # routes { "virtual_hosts": [ { "name": "service_a", "domains": [ "*" ], "routes": [ { "prefix": "/", "cluster": "service_a", "shadow": { "cluster": "service_a_prime" } ... A’ B’ strage front->A: GET / front->A: GET /users A->B: GET /awesome_process record log shadow shadow
  20. 32.

    ύϑΥʔϚϯεଌఆҊ • ೖྗ • ϦΫΤετύεॏΈ෼෍ d • ಉ࣌઀ଓ਺ c •

    ϨεϙϯελΠϜ໨ඪ r • ग़ྗɿϝτϦΫε • d ʹैͬͨ WRR, c ฒྻͰϦΫΤετ 20 RTT ≦ r Ҏ಺͔ͭ 5xx Ҏ֎ͷϦΫΤετ਺ ୯Ґ࣌ؒ rps = GET / 0.3 GET /users 0.2 POST /users 0.2 GET /depends_b 0.1 …