サービスのパフォーマンス数値と 依存関係を用いたサービス同士の 協調スケール構想 / Web System Architecture #1

Slide 1

Slide 1 text

αʔϏεͷύϑΥʔϚϯε਺஋ͱ  ґଘؔ܎Λ༻͍ͨαʔϏεಉ࢜ͷ  ڠௐεέʔϧߏ૝ Web System Architecture ݚڀձ #1 @itkq

Slide 2

Slide 2 text

Me 2 @itkq Takuya Kosugiyama ౦޻େ ৘ใ௨৴ܥ M2 όΠτ (SRE) ͍ͨ͜ (͖Ύʔ)

Slide 3

Slide 3 text

ࢀՃ໨త • Web System ͷՄೳੑɺݶքɺֶज़తՁ஋ʹ  ڵຯ͕͋Δ • ༷ʑͳਓʑͱٞ࿦͍ͨ͠ • ए͍͏ͪ [ཁग़య] ʹੈքΛ޿͍͛ͨ 3

Slide 4

Slide 4 text

ࢀՃ໨త • Web System ͷՄೳੑɺݶքɺֶज़తՁ஋ʹ  ڵຯ͕͋Δ • ༷ʑͳਓʑͱٞ࿦͍ͨ͠ • ए͍͏ͪ [ཁग़య] ʹੈքΛ޿͍͛ͨ 3 • ϦΞϧ id:y_uuki ͞ΜΛݟʹ͖ͨ

Slide 5

Slide 5 text

ൃද಺༰ • ࠷ۙͷ Web ΞʔΩςΫνϟͷಈ޲ͱ SRE ຊΛ  ಡΜͰߟ͑ͨ͜ͱ • ίϯςφϕʔεͷαʔϏεϝογϡΛ׆༻ͨ͠  ӡ༻্ͷεέʔϧ໰୊ղܾͷͨΊͷ଍͕͔Γ 4

Slide 6

Slide 6 text

എܠɿΞϓϦέʔγϣϯίϯςφ • ίϯςφܕԾ૝Խ • খΦʔόʔϔου • Container as a Service • Production ready • Docker, Kubernetes • Managed services (GKE, ECS, EKS) 5

Slide 7

Slide 7 text

എܠɿαʔϏεࢦ޲ΞʔΩςΫνϟ • ϞϊϦεͷେن໛Խͱݶք • อकੑɺ։ൃޮ཰ɺ… • αʔϏεࢦ޲ΞʔΩςΫνϟ • ػೳΛαʔϏεͱͯ͠੾Γग़͢ • αʔϏεಉ͕࢜࿈ܞ 6 Q. αʔϏεςΟεΧόϦ? ϦτϥΠ? λΠϜΞ΢τ?

Slide 8

Slide 8 text

എܠɿαʔϏεϝογϡ • service-to-service ௨৴ͷࡍͰϓϩΩγΛհ͢Δ • αʔϏεϝογϡɿ  L7 ϓϩΩγʹΑΔωοτϫʔΫͷந৅ϨΠϠʔ 7 Service A Proxy Service B Proxy Controller Data Plane Control Plane

Slide 9

Slide 9 text

എܠɿαʔϏεϝογϡͷ༻్ • Envoy, Linkerd • Advanced load balancing • Circuit breaking • Rate limiting • Service discovery • Observation • Statistics • Logging • Tracing 8 ։ൃج൫ͷྖҬʹۙ͘ɺӡ༻΁ͷԠ༻͸ߟ͑Δ༨஍͋Γ

Slide 10

Slide 10 text

ίϯςφɾϚΠΫϩαʔϏεͷ࣮ӡ༻՝୊ 1. ϗετΩϟύγςΟϓϥϯχϯά • ίϯςφͷਫฏεέʔϧ͸༰қ • ͔͠͠ϗετΩϟύγςΟ͸ඞཁ 2. ίϯςφϦιʔεͷܾఆ๏ • “desired count” ͸୭͕Ͳ͏ܾΊΔͷ͔ 3. αʔϏεಉ͕࢜ڠௐͨ͠εέʔϧ • ґଘઌαʔϏεͷΩϟύγςΟ΋ߟྀ͢Δඞཁ 9 ͍ͣΕ΋ΩϟύγςΟͷ໰୊ͱͯ͠ू໿Ͱ͖Δ

Slide 11

Slide 11 text

ΩϟύγςΟϓϥϯχϯάͷ޲͔͏΂͖࢟ • Site Reliability Engineering 18.2 10 ΠϯςϯτϕʔεͷΩϟύγςΟ໨ඪ Ϧιʔε੍໿ɾྉۚ࠷খԽͷ࠷దԽ໰୊ ґଘؔ܎ɾύϑΥʔϚϯεϝτϦΫεͷ೺Ѳ

Slide 12

Slide 12 text

Slide 13

Slide 13 text

࠶ܝɿ࣮ӡ༻՝୊ 11 1. ϗετΩϟύγςΟϓϥϯχϯά 2. ίϯςφϦιʔεͷܾఆ๏ 3. αʔϏεಉ͕࢜ڠௐͨ͠εέʔϧ

Slide 14

Slide 14 text

࠶ܝɿ࣮ӡ༻՝୊ 11 1. ϗετΩϟύγςΟϓϥϯχϯά 2. ίϯςφϦιʔεͷܾఆ๏ 3. αʔϏεಉ͕࢜ڠௐͨ͠εέʔϧ جૅݕ౼ɿ • ύϑΥʔϚϯεఆྔԽ

Slide 15

Slide 15 text

՝୊Πϝʔδɿ૝ఆঢ়گ 12 A B C user facing internal × 2 × 1 × 1 D System × 1

Slide 16

Slide 16 text

՝୊Πϝʔδ 1 13 A B C user facing internal × 2 × 1 × 1 D System × 1 desired count: 6 ex. ϐʔΫ࣌  A ͷෛՙ 3 ഒ desired count: 3

Slide 17

Slide 17 text

՝୊Πϝʔδ 1 13 A B C user facing internal × 2 × 1 × 1 D System × 1 desired count: 6 × 6 ex. ϐʔΫ࣌  A ͷෛՙ 3 ഒ × 3 desired count: 3 C ͷߟྀ࿙Ε

Slide 18

Slide 18 text

՝୊Πϝʔδ 1 13 A B C user facing internal × 2 × 1 × 1 D System × 1 desired count: 6 × 6 ex. ϐʔΫ࣌  A ͷෛՙ 3 ഒ C × 3 desired count: 3 C ͷߟྀ࿙Ε

Slide 19

Slide 19 text

՝୊Πϝʔδ 1 13 A B C user facing internal × 2 × 1 × 1 D System × 1 desired count: 6 × 6 ex. ϐʔΫ࣌  A ͷෛՙ 3 ഒ C A × 3 desired count: 3 C ͷߟྀ࿙Ε

Slide 20

Slide 20 text

՝୊Πϝʔδ 2 14 A B C user facing internal × 2 × 1 × 1 D System × 1 desired count: 6 desired count: 3 desired count: 3 ex. ϐʔΫ࣌  A ͷෛՙ 3 ഒ

Slide 21

Slide 21 text

՝୊Πϝʔδ 2 14 A B C user facing internal × 2 × 1 × 1 D System × 1 desired count: 6 × 6 × 3 × 3 desired count: 3 desired count: 3 ex. ϐʔΫ࣌  A ͷෛՙ 3 ഒ D ͷߟྀ࿙Ε

Slide 22

Slide 22 text

՝୊Πϝʔδ 2 14 A B C user facing internal × 2 × 1 × 1 D System × 1 desired count: 6 × 6 B × 3 × 3 desired count: 3 desired count: 3 ex. ϐʔΫ࣌  A ͷෛՙ 3 ഒ D ͷߟྀ࿙Ε

Slide 23

Slide 23 text

՝୊Πϝʔδ 2 14 A B C user facing internal × 2 × 1 × 1 D System × 1 desired count: 6 × 6 B A × 3 × 3 desired count: 3 desired count: 3 ex. ϐʔΫ࣌  A ͷෛՙ 3 ഒ D ͷߟྀ࿙Ε

Slide 24

Slide 24 text

ղܾΠϝʔδɿ૝ఆঢ়گ 15 A 100 rps/container A B C user facing internal × 2 × 1 × 1 D System × 1 B 200 rps/container C 100 rps/container D 100 rps/container 0.4 0.2 0.7

Slide 25

Slide 25 text

ղܾΠϝʔδ 16 A 100 rps/container A B C user facing internal × 2 × 1 × 1 D System × 1 B 200 rps/container C 100 rps/container D 100 rps/container 0.4 0.2 0.7 ex. ϐʔΫ࣌  A ͷෛՙ 3 ഒ

Slide 26

Slide 26 text

ղܾΠϝʔδ 16 A 100 rps/container A B C user facing internal × 2 × 1 × 1 D System × 1 B 200 rps/container C 100 rps/container D 100 rps/container 0.4 0.2 0.7 ex. ϐʔΫ࣌  A ͷෛՙ 3 ഒ × 6

Slide 27

Slide 27 text

ղܾΠϝʔδ 16 A 100 rps/container A B C user facing internal × 2 × 1 × 1 D System × 1 B 200 rps/container C 100 rps/container D 100 rps/container 0.4 0.2 0.7 ex. ϐʔΫ࣌  A ͷෛՙ 3 ഒ (100 * 6 * 0.4 + 100 * 0.7)   / 200 ~ 2 (100 * 6 * 0.2) / 100 ~ 2 × 6 × 2 × 2 ύϑΥʔϚϯε਺஋ + ґଘ౓ → ඞཁ࠷௿ݶͷίϯςφ਺

Slide 28

Slide 28 text

໰୊ɿύϑΥʔϚϯεఆྔԽͱґଘ౓ • ఆྔԽʹؔͯ͠ • ͲͷΑ͏ʹଌఆ͢Δ͔ • ϝτϦΫε͸ rps ͰΑ͍͔ • ͲͷΤϯυϙΠϯτʹର͢Δ rps ͔ • ϦΫΤετύλʔϯ͸ prod ͱಉҰ͔ • αʔϏεͷґଘ౓͸Ͳ͏ܾ·Δ͔ 17 ύϑΥʔϚϯεࣗಈଌఆɾఆྔԽͷ࣮ྫ͕গͳ͍

Slide 29

Slide 29 text

໰୊ɿύϑΥʔϚϯεఆྔԽͱґଘ౓ • ఆྔԽʹؔͯ͠ • ͲͷΑ͏ʹଌఆ͢Δ͔ • ϝτϦΫε͸ rps ͰΑ͍͔ • ͲͷΤϯυϙΠϯτʹର͢Δ rps ͔ • ϦΫΤετύλʔϯ͸ prod ͱಉҰ͔ • αʔϏεͷґଘ౓͸Ͳ͏ܾ·Δ͔ 17 ύϑΥʔϚϯεࣗಈଌఆɾఆྔԽͷ࣮ྫ͕গͳ͍ αʔϏεϝογϡ + shadowing

Slide 30

Slide 30 text

αʔϏεϝογϡߏ੒ྫ 18 front-proxy Request B proxy A proxy stats discovery • connection / request count • 1xx, 2xx, … 5xx count • etc. per proxy metrics:

Slide 31

Slide 31 text

shadowing 19 front-proxy Request B proxy A proxy Prod Shadow # routes { "virtual_hosts": [ { "name": "service_a", "domains": [ "*" ], "routes": [ { "prefix": "/", "cluster": "service_a", "shadow": { "cluster": "service_a_prime" } ... A’ B’ strage front->A: GET / front->A: GET /users A->B: GET /awesome_process record log shadow shadow

Slide 32

Slide 32 text

ύϑΥʔϚϯεଌఆҊ • ೖྗ • ϦΫΤετύεॏΈ෼෍ d • ಉ࣌઀ଓ਺ c • ϨεϙϯελΠϜ໨ඪ r • ग़ྗɿϝτϦΫε • d ʹैͬͨ WRR, c ฒྻͰϦΫΤετ 20 RTT ≦ r Ҏ಺͔ͭ 5xx Ҏ֎ͷϦΫΤετ਺ ୯Ґ࣌ؒ rps = GET / 0.3 GET /users 0.2 POST /users 0.2 GET /depends_b 0.1 …

Slide 33

Slide 33 text

·ͱΊ • ϚΠΫϩαʔϏεʹ൐͏αʔϏεϝογϡͷଘࡏ • ࣮ӡ༻΁ͷԠ༻͕ݕ౼༨஍͋Γ • ࠷ߴͷΩϟύγςΟϓϥϯχϯά΁ͷ଍͕͔Γ  ͱͯ͠ɺύϑΥʔϚϯεఆྔԽʹண໨ • αʔϏεϝογϡͰϦΫΤετύεॏΈ෼෍  Λࢉग़͠ϝτϦΫεΛܭࢉ • ύϑΥʔϚϯε࣮ଌɺґଘ౓͸ࠓޙͷ՝୊ 21