サービスのパフォーマンス数値と
依存関係を用いたサービス同士の
協調スケール構想 / Web System Architecture #1

Transcript

1. αʔϏεͷύϑΥʔϚϯε਺஋ͱ
 ґଘؔ܎Λ༻͍ͨαʔϏεಉ࢜ͷ
 ڠௐεέʔϧߏ૝ Web System Architecture ݚڀձ #1 @itkq

2. Me 2 @itkq Takuya Kosugiyama ౦޻େ ৘ใ௨৴ܥ M2 όΠτ (SRE)

   ͍ͨ͜ (͖Ύʔ)

3. ࢀՃ໨త • Web System ͷՄೳੑɺݶքɺֶज़తՁ஋ʹ
 ڵຯ͕͋Δ • ༷ʑͳਓʑͱٞ࿦͍ͨ͠ • ए͍͏ͪ

   [ཁग़య] ʹੈքΛ޿͍͛ͨ 3

4. ࢀՃ໨త • Web System ͷՄೳੑɺݶքɺֶज़తՁ஋ʹ
 ڵຯ͕͋Δ • ༷ʑͳਓʑͱٞ࿦͍ͨ͠ • ए͍͏ͪ

   [ཁग़య] ʹੈքΛ޿͍͛ͨ 3 • ϦΞϧ id:y_uuki ͞ΜΛݟʹ͖ͨ

5. ൃද಺༰ • ࠷ۙͷ Web ΞʔΩςΫνϟͷಈ޲ͱ SRE ຊΛ
 ಡΜͰߟ͑ͨ͜ͱ • ίϯςφϕʔεͷαʔϏεϝογϡΛ׆༻ͨ͠


   ӡ༻্ͷεέʔϧ໰୊ղܾͷͨΊͷ଍͕͔Γ 4

6. എܠɿΞϓϦέʔγϣϯίϯςφ • ίϯςφܕԾ૝Խ • খΦʔόʔϔου • Container as a Service

   • Production ready • Docker, Kubernetes • Managed services (GKE, ECS, EKS) 5

7. എܠɿαʔϏεࢦ޲ΞʔΩςΫνϟ • ϞϊϦεͷେن໛Խͱݶք • อकੑɺ։ൃޮ཰ɺ… • αʔϏεࢦ޲ΞʔΩςΫνϟ • ػೳΛαʔϏεͱͯ͠੾Γग़͢ •

   αʔϏεಉ͕࢜࿈ܞ 6 Q. αʔϏεςΟεΧόϦ? ϦτϥΠ? λΠϜΞ΢τ?

8. എܠɿαʔϏεϝογϡ • service-to-service ௨৴ͷࡍͰϓϩΩγΛհ͢Δ • αʔϏεϝογϡɿ
 L7 ϓϩΩγʹΑΔωοτϫʔΫͷந৅ϨΠϠʔ 7 Service

   A Proxy Service B Proxy Controller Data Plane Control Plane

9. എܠɿαʔϏεϝογϡͷ༻్ • Envoy, Linkerd • Advanced load balancing • Circuit

   breaking • Rate limiting • Service discovery • Observation • Statistics • Logging • Tracing 8 ։ൃج൫ͷྖҬʹۙ͘ɺӡ༻΁ͷԠ༻͸ߟ͑Δ༨஍͋Γ

10. ίϯςφɾϚΠΫϩαʔϏεͷ࣮ӡ༻՝୊ 1. ϗετΩϟύγςΟϓϥϯχϯά • ίϯςφͷਫฏεέʔϧ͸༰қ • ͔͠͠ϗετΩϟύγςΟ͸ඞཁ 2. ίϯςφϦιʔεͷܾఆ๏ •

    “desired count” ͸୭͕Ͳ͏ܾΊΔͷ͔ 3. αʔϏεಉ͕࢜ڠௐͨ͠εέʔϧ • ґଘઌαʔϏεͷΩϟύγςΟ΋ߟྀ͢Δඞཁ 9 ͍ͣΕ΋ΩϟύγςΟͷ໰୊ͱͯ͠ू໿Ͱ͖Δ

11. ΩϟύγςΟϓϥϯχϯάͷ޲͔͏΂͖࢟ • Site Reliability Engineering 18.2 10 ΠϯςϯτϕʔεͷΩϟύγςΟ໨ඪ Ϧιʔε੍໿ɾྉۚ࠷খԽͷ࠷దԽ໰୊ ґଘؔ܎ɾύϑΥʔϚϯεϝτϦΫεͷ೺Ѳ

12. ΩϟύγςΟϓϥϯχϯάͷ޲͔͏΂͖࢟ • Site Reliability Engineering 18.2 10 ΠϯςϯτϕʔεͷΩϟύγςΟ໨ඪ Ϧιʔε੍໿ɾྉۚ࠷খԽͷ࠷దԽ໰୊ ґଘؔ܎ɾύϑΥʔϚϯεϝτϦΫεͷ೺Ѳ

13. ࠶ܝɿ࣮ӡ༻՝୊ 11 1. ϗετΩϟύγςΟϓϥϯχϯά 2. ίϯςφϦιʔεͷܾఆ๏ 3. αʔϏεಉ͕࢜ڠௐͨ͠εέʔϧ

14. ࠶ܝɿ࣮ӡ༻՝୊ 11 1. ϗετΩϟύγςΟϓϥϯχϯά 2. ίϯςφϦιʔεͷܾఆ๏ 3. αʔϏεಉ͕࢜ڠௐͨ͠εέʔϧ جૅݕ౼ɿ •

    ύϑΥʔϚϯεఆྔԽ

15. ՝୊Πϝʔδɿ૝ఆঢ়گ 12 A B C user facing internal × 2

    × 1 × 1 D System × 1

16. ՝୊Πϝʔδ 1 13 A B C user facing internal ×

    2 × 1 × 1 D System × 1 desired count: 6 ex. ϐʔΫ࣌
 A ͷෛՙ 3 ഒ desired count: 3

17. ՝୊Πϝʔδ 1 13 A B C user facing internal ×

    2 × 1 × 1 D System × 1 desired count: 6 × 6 ex. ϐʔΫ࣌
 A ͷෛՙ 3 ഒ × 3 desired count: 3 C ͷߟྀ࿙Ε

18. ՝୊Πϝʔδ 1 13 A B C user facing internal ×

    2 × 1 × 1 D System × 1 desired count: 6 × 6 ex. ϐʔΫ࣌
 A ͷෛՙ 3 ഒ C × 3 desired count: 3 C ͷߟྀ࿙Ε

19. ՝୊Πϝʔδ 1 13 A B C user facing internal ×

    2 × 1 × 1 D System × 1 desired count: 6 × 6 ex. ϐʔΫ࣌
 A ͷෛՙ 3 ഒ C A × 3 desired count: 3 C ͷߟྀ࿙Ε

20. ՝୊Πϝʔδ 2 14 A B C user facing internal ×

    2 × 1 × 1 D System × 1 desired count: 6 desired count: 3 desired count: 3 ex. ϐʔΫ࣌
 A ͷෛՙ 3 ഒ

21. ՝୊Πϝʔδ 2 14 A B C user facing internal ×

    2 × 1 × 1 D System × 1 desired count: 6 × 6 × 3 × 3 desired count: 3 desired count: 3 ex. ϐʔΫ࣌
 A ͷෛՙ 3 ഒ D ͷߟྀ࿙Ε

22. ՝୊Πϝʔδ 2 14 A B C user facing internal ×

    2 × 1 × 1 D System × 1 desired count: 6 × 6 B × 3 × 3 desired count: 3 desired count: 3 ex. ϐʔΫ࣌
 A ͷෛՙ 3 ഒ D ͷߟྀ࿙Ε

23. ՝୊Πϝʔδ 2 14 A B C user facing internal ×

    2 × 1 × 1 D System × 1 desired count: 6 × 6 B A × 3 × 3 desired count: 3 desired count: 3 ex. ϐʔΫ࣌
 A ͷෛՙ 3 ഒ D ͷߟྀ࿙Ε

24. ղܾΠϝʔδɿ૝ఆঢ়گ 15 A 100 rps/container A B C user facing

    internal × 2 × 1 × 1 D System × 1 B 200 rps/container C 100 rps/container D 100 rps/container 0.4 0.2 0.7

25. ղܾΠϝʔδ 16 A 100 rps/container A B C user facing

    internal × 2 × 1 × 1 D System × 1 B 200 rps/container C 100 rps/container D 100 rps/container 0.4 0.2 0.7 ex. ϐʔΫ࣌
 A ͷෛՙ 3 ഒ

26. ղܾΠϝʔδ 16 A 100 rps/container A B C user facing

    internal × 2 × 1 × 1 D System × 1 B 200 rps/container C 100 rps/container D 100 rps/container 0.4 0.2 0.7 ex. ϐʔΫ࣌
 A ͷෛՙ 3 ഒ × 6

27. ղܾΠϝʔδ 16 A 100 rps/container A B C user facing

    internal × 2 × 1 × 1 D System × 1 B 200 rps/container C 100 rps/container D 100 rps/container 0.4 0.2 0.7 ex. ϐʔΫ࣌
 A ͷෛՙ 3 ഒ (100 * 6 * 0.4 + 100 * 0.7) 
 / 200 ~ 2 (100 * 6 * 0.2) / 100 ~ 2 × 6 × 2 × 2 ύϑΥʔϚϯε਺஋ + ґଘ౓ → ඞཁ࠷௿ݶͷίϯςφ਺

28. ໰୊ɿύϑΥʔϚϯεఆྔԽͱґଘ౓ • ఆྔԽʹؔͯ͠ • ͲͷΑ͏ʹଌఆ͢Δ͔ • ϝτϦΫε͸ rps ͰΑ͍͔ •

    ͲͷΤϯυϙΠϯτʹର͢Δ rps ͔ • ϦΫΤετύλʔϯ͸ prod ͱಉҰ͔ • αʔϏεͷґଘ౓͸Ͳ͏ܾ·Δ͔ 17 ύϑΥʔϚϯεࣗಈଌఆɾఆྔԽͷ࣮ྫ͕গͳ͍

29. ໰୊ɿύϑΥʔϚϯεఆྔԽͱґଘ౓ • ఆྔԽʹؔͯ͠ • ͲͷΑ͏ʹଌఆ͢Δ͔ • ϝτϦΫε͸ rps ͰΑ͍͔ •

    ͲͷΤϯυϙΠϯτʹର͢Δ rps ͔ • ϦΫΤετύλʔϯ͸ prod ͱಉҰ͔ • αʔϏεͷґଘ౓͸Ͳ͏ܾ·Δ͔ 17 ύϑΥʔϚϯεࣗಈଌఆɾఆྔԽͷ࣮ྫ͕গͳ͍ αʔϏεϝογϡ + shadowing

30. αʔϏεϝογϡߏ੒ྫ 18 front-proxy Request B proxy A proxy stats discovery

    • connection / request count • 1xx, 2xx, … 5xx count • etc. per proxy metrics:

31. shadowing 19 front-proxy Request B proxy A proxy Prod Shadow

    # routes { "virtual_hosts": [ { "name": "service_a", "domains": [ "*" ], "routes": [ { "prefix": "/", "cluster": "service_a", "shadow": { "cluster": "service_a_prime" } ... A’ B’ strage front->A: GET / front->A: GET /users A->B: GET /awesome_process record log shadow shadow

32. ύϑΥʔϚϯεଌఆҊ • ೖྗ • ϦΫΤετύεॏΈ෼෍ d • ಉ࣌઀ଓ਺ c •

    ϨεϙϯελΠϜ໨ඪ r • ग़ྗɿϝτϦΫε • d ʹैͬͨ WRR, c ฒྻͰϦΫΤετ 20 RTT ≦ r Ҏ಺͔ͭ 5xx Ҏ֎ͷϦΫΤετ਺ ୯Ґ࣌ؒ rps = GET / 0.3 GET /users 0.2 POST /users 0.2 GET /depends_b 0.1 …

33. ·ͱΊ • ϚΠΫϩαʔϏεʹ൐͏αʔϏεϝογϡͷଘࡏ • ࣮ӡ༻΁ͷԠ༻͕ݕ౼༨஍͋Γ • ࠷ߴͷΩϟύγςΟϓϥϯχϯά΁ͷ଍͕͔Γ
 ͱͯ͠ɺύϑΥʔϚϯεఆྔԽʹண໨ • αʔϏεϝογϡͰϦΫΤετύεॏΈ෼෍


    Λࢉग़͠ϝτϦΫεΛܭࢉ • ύϑΥʔϚϯε࣮ଌɺґଘ౓͸ࠓޙͷ՝୊ 21