xDS を活用したサービスディスカバリーで実現するブランチ別 QA 環境の構築手法

Transcript

1. xDS Λ׆༻ͨ͠ αʔϏεσΟεΧόϦʔͰ࣮ݱ͢Δ ϒϥϯνผ QA ؀ڥͷߏஙख๏ ʲGoಛूʳόοΫΤϯυGo։ൃͷཪଆ — Findy
2. ࣗݾ঺հ Takahashi Kenta Χ΢γΣגࣜձࣾ όοΫΤϯυΤϯδχΞ ࠓ೔࿩͢͜ͱ ҎԼͷΑ͏ͳ՝୊ΛɺxDSΛ࢖ͬͯղܾ͍ͯ͠ΔΑɺͱ͍͏࿩Λ͠·͢ - ϒϥϯν͝ͱʹ؀ڥΛͭͬͯ͘ಈ࡞֬ೝΛ͍ͨ͠ - αʔϏεΛ·͍ͨͰ΋ϒϥϯν͝ͱͷ؀ڥͰಈ࡞֬ೝ͍ͨ͠
3. લఏ Χ΢γΣͷΞʔΩςΫνϟ • Everything runs on Cloud Run • Everything runs as an API (gRPC) • API Gateway = Envoy (ԣஅతؔ৺ࣄ) Mobile App API Gateway (Envoy) grpc-gateway Service A Service B + Pub/Sub / Cloud Tasks / Scheduler Envoy ͕୲͏ԣஅతؔ৺ࣄ: JWT ೝূ / RBAC / Transcoding TLS / CDN / HTTP Header ૢ࡞ (Wasm)
4. ՝୊ͷมભ
5. Phase 1 ࠷ॳ͸͏·͍͍ͬͯͨ͘ ϒϥϯν͝ͱʹ Cloud Run ΛσϓϩΠ͠ Mobile ͔Β௚઀ϒϥϯν؀ڥʹϦΫΤετ Mobile App branch ؀ڥ Service A (branch) Service B (main) γϯϓϧͰ໰୊ͳ͠ ୯ҰͷαʔϏεΛมߋ͢Δ͚ͩͷ PR Ͱ͸ ͜ΕͰे෼ʹݕূͰ͖͍ͯͨ
6. Phase 2 αʔϏεΛލ͍Ͱ΋ͳ͓ϒϥϯν؀ڥΛར༻͍ͨ͠ Mobile → Service A → Service B (branch) ʹϦΫΤετΛ౤͍͛ͨ Service A ͷ޲͖ઌΛຖճखಈͰมߋʁ → ։ൃऀɾϒϥϯν͕૿͑Δ΄Ͳݱ࣮తͰͳ͍ Mobile App Service A B (main) B (br-1) B (br-2) खಈͰ޲͖ઌΛมߋ͢Δͷ͸ෆՄೳ
7. Phase 3 Job API Ͱ΋ϒϥϯν؀ڥΛར༻͍ͨ͠ Χ΢γΣͰ͸ Pub/Sub ͔Βݺͼग़͞ΕΔ Job API ΋ gRPC Ͱ࣮૷͞Ε͍ͯΔ Pub/Sub → Job API (gRPC) ͱ͍͏ྲྀΕ ͜ͷ gRPC ݺͼग़͠΋ϒϥϯν୯ҐͰ੍ޚ͍ͨ͠ Pub/Sub + Tasks / Scheduler Job API (main) Job API (branch) gRPC ? Job API ΋ gRPC → xDS ͰղܾͰ͖Δ͸ͣ
8. ٻΊΒΕͨཁ݅ • αʔϏεΛލ͍Ͱ΋ϒϥϯν؀ڥʹϦΫΤετΛ޲͚Δ • Pub/Sub ΍ TasksɺScheduler ͔ΒͷϦΫΤετ΋ ϒϥϯν؀ڥʹ޲͚Δ → xDS ϕʔεͷαʔϏεσΟεΧόϦʔͰղܾ
9. xDS ͷجૅ஌ࣝ
10. xDS ͱ͸ ΋ͱ΋ͱ Envoy ͷͨΊʹઃܭ͞ΕͨɺϧʔςΟ ϯά΍Ѽઌ৘ใͳͲͷ઀ଓؔ࿈ઃఆΛಈతʹ഑ ৴͢ΔͨΊͷ API ܈ɻ "x" ͸ Listener / Route / Cluster / Endpoint ͳ Ͳ༷ʑͳϦιʔεΛࢦ͠ɺͦΕΒͷ Discovery Service Ͱ xDS grpc-go ͸ xDS ΛωΠςΟϒαϙʔτ͓ͯ͠Γ ࠶ىಈͳ͠ʹϧʔςΟϯάΛϦΞϧλΠϜߋ৽Ͱ͖Δ Control Plane (Go / go-control-plane) xDS Data Plane (grpc-gateway / grpc-go)
11. xDS ϓϩτίϧͷߏ੒ LDS Listener Discovery ͲͷϙʔτͰ listen ͢Δ͔ RDS Route Discovery ͲͷϧʔτͰϚονͤ͞Δ͔ CDS Cluster Discovery ͲͷΫϥελʹৼΓ෼͚Δ͔ EDS Endpoint Discovery ۩ମతͳΤϯυϙΠϯτ͸Կ͔ ϒϥϯν QA Ͱͷ׆༻: ϔομʔͷ஋ʹԠͯ͡ ϒϥϯν؀ڥͷ Cloud Run ʹ ಈతʹϧʔςΟϯάΛ੾Γସ͑Δ → LDS ͱ CDS Λ׆༻
12. Proxyless gRPC ͷ xDS ωΠςΟϒαϙʔτ Envoy ͳͲͷαΠυΧʔϓϩΩγΛܦ༝ͤͣ gRPC ΫϥΠΞϯτࣗମ͕ xDS Λ௚઀ղऍ grpc-go ͸ xds:/// εΩʔϜͰ Control Plane ͔ΒϧʔςΟϯά৘ใΛऔಘ͠ ࣗ෼ࣗ਎ͰϩʔυόϥϯγϯάɾϧʔςΟϯάΛߦ͏ αΠυΧʔෆཁ = ӡ༻ίετ࡟ݮ ैདྷ (Sidecar) App → Envoy → upstream Sidecar ؅ཧ͕ඞཁ Proxyless (ࠓճͷํࣜ) App + grpc-go (xDS) → upstream Sidecar ෆཁ grpc-go v1.40+ Ͱ xDS αϙʔτ͕҆ఆ
13. Why xDS? Χ΢γΣʹϑΟοτͨ͠ཧ༝ • grpc-go ͕ xDS ΛωΠςΟϒαϙʔτ طଘͷ gRPC αʔϏεʹͦͷ··૊ΈࠐΊΔɻ௥ՃϓϩΩγෆཁ • Dynamic Configurations αʔϏεͷ࠶ىಈͳ͠ʹϧʔςΟϯάΛϦΞϧλΠϜߋ৽ • Go Ͱ Control-Plane ͷ࣮૷͕͠΍͍͢ Go Ͱ Control Plane Λ࣮૷ɻ https://github.com/envoyproxy/go-control-plane Λ࢖ͬͯ؆୯ʹ࣮૷͕Ͱ͖Δ
14. OSS cloud-run-service-router-xds Χ΢γΣ͕։ൃɾެ։ͨ͠ xDS Control Plane Cloud Run ͷϦϏδϣϯ৘ใΛݩʹ xDS Snapshot Λࣗಈੜ੒͢Δαʔό • go-control-plane ϕʔεͷ Control Plane • Cloud Run Admin API ͰϦϏδϣϯ؂ࢹ • ϒϥϯν → Snapshot ͷࣗಈϚοϐϯά • seed.yaml ͰαʔϏεఆٛΛએݴత؅ཧ kauche/ cloud-run-service-router-xds Go 98.9% ˒ 4 v0.0.6 github.com/kauche/cloud-run-service-router-xds
15. ΞʔΩςΫνϟશମ૾
16. શମΞʔΩςΫνϟ 1. ϒϥϯν؀ڥͷσϓϩΠ Cloud Run αʔϏε໊: svc-a-main svc-a-feat-payment svc-b-main svc-b-feat-payment {prefix}-{branch} 2. Control Plane ͕αʔϏεҰཡΛऔಘ cloud-run-service-router-xds Cloud Run Admin API Λఆظతʹݺͼग़͠ σϓϩΠࡁΈαʔϏεɾϦϏδϣϯΛݕग़ ϒϥϯν໊Λநग़ͯ͠ xDS Snapshot Λੜ੒ Cloud Run API 3. xDS ͰϧʔςΟϯά৘ใΛ഑৴ Control Plane (xDS Server) xDS stream grpc-gateway Service A Service B Service C 4. x-branch ϔομʔʹج͍ͮͯϧʔςΟϯά Mobile (x-branch: feat/payment) → grpc-gateway → Service A (branch) → Service B (branch)
17. ౷Ұݪଇ: ಛघϔομʔʹΑΔϧʔςΟϯά x-branch: feature/xxx (Ծ໊Ͱ͢) ͜ͷϔομʔ͕෇͍͍ͯΕ͹ → xDS ͕ϒϥϯν؀ڥʹϧʔςΟϯά ϔομʔ͕ͳ͚Ε͹ → main ʹϑΥʔϧόοΫ ͢΂ͯͷϒϥϯνϧʔςΟϯά͕͜ͷݪଇʹ౷Ұ͞ΕΔ • ಉظ௨৴ (gRPC): ϔομʔΛαʔϏεؒͰ఻ൖ • ඇಉظ௨৴ (Pub/Sub): attributes → ϔομʔʹม׵ͯ͠߹ྲྀ
18. ಉظ௨৴ gRPC ͷέʔε Mobile App x-branch: feat/xxx grpc-gateway ← xDS ϧʔςΟϯά Service A (branch) ϔομʔ఻ൖ → Service B (branch) ← grpc-go xDS खಈͷ޲͖ઌมߋ͕ෆཁɻϔομʔ͕ࣗಈ఻ൖ͞ΕΔ
19. ࣮૷ grpc-go Ͱͷ xDS ઀ଓ xds:/// εΩʔϜ + Bootstrap ઃఆͰ xDS ϧʔςΟϯά͕༗ޮʹ // xDS resolver Λ import _ "google.golang.org/grpc/xds" // ઀ଓઌΛ xds:/// εΩʔϜͰࢦఆ conn, err := grpc.Dial( "xds:///service-a.example.com:443", grpc.WithTransportCredentials(...), ) ඞཁͳઃఆ: • GRPC_XDS_BOOTSTRAP ؀ڥม਺Ͱ bootstrap ઃఆΛ஫ೖ • ઀ଓઌ URI Λ xds:/// εΩʔϜʹมߋ • xds ύοέʔδΛ import (resolver / balancer ͷొ࿥) → grpc-go ͕ Control Plane ͱ௨৴͠ɺϧʔςΟϯά৘ใΛࣗಈऔಘ
20. ඇಉظ௨৴ Pub/Sub → Job API (gRPC) ͷέʔε Pub/Sub attr: branch=feat/ xxx attr → x-branch ϔομʔʹม׵ (api-gateway) Job API (gRPC) + xDS ϧʔςΟϯά gRPC ͳͷͰಉظ௨৴ͱಉ͡࢓૊Έ ϙΠϯτ: Pub/Sub (+ Tasks / Scheduler) ͸ Job API Λ௚઀ݺͼग़͢ Job API ΋ gRPC Ͱ࣮૷͞Ε͍ͯΔͨΊ Pub/Sub ͷ attributes Λ x-branch ϔομʔʹม׵͢Ε͹ ಉ͡ xDS ϧʔςΟϯάʹ৐Δ
21. ઃܭͷΩϞ ಉظ΋ඇಉظ΋ ʮಛघϔομʔ ʯ ͱ͍͏୯ҰͷϧʔςΟϯάػߏʹू໿ ಉظ: gRPC ϔομʔ఻ൖ ඇಉظ: attr → header ม׵ xDS ϧʔςΟϯά
22. ࣮૷ xDS Control Plane go-control-plane ϥΠϒϥϦͰ Go ࣮૷ 1 ϒϥϯνͷ Cloud Run σϓϩΠ 2 Control Plane ͕ݕ஌ 3 xDS Snapshot ߋ৽ 4 grpc-go ͕ ࣗಈ൓ө Cloud Run ͷσϓϩΠ৘ใΛ؂ࢹ͠ɺSnapshot Λಈతʹੜ੒ grpc-gateway / grpc-go ͸ gRPC stream ͰมߋΛड৴ → ଈ࣌൓ө https://github.com/envoyproxy/go-control-plane
23. ࣮૷ xDS Bootstrap ઃఆ grpc-go ͕ Control Plane ʹ઀ଓ͢ΔͨΊͷઃఆ { "xds_servers": [{ "server_uri": "xds-server:18000", "channel_creds": [ {"type": "google_default"} ], "server_features": ["xds_v3"] }], "node": { "id": "service-a" } } ઃఆͷϙΠϯτ: server_uri xDS Control Plane ͷΞυϨε channel_creds TLS ೝূ৘ใͷઃఆ node.id Control Plane ͕ઃఆΛৼΓ෼͚Δ ͨΊͷϊʔυࣝผࢠ GRPC_XDS_BOOTSTRAP ؀ڥม਺ͰϑΝΠϧύεΛࢦఆͯ͠஫ೖ
24. ࣮૷ Control Plane ΁ͷ TLS ઀ଓ grpc-go-xds-tls-credentials Control Plane ΁ͷ઀ଓʹඞཁͳ TLS ΫϨσϯγϟϧΛఏڙ͢Δ Go ϥΠϒϥϦ (OSS) ͳͥඞཁ͔: Cloud Run ্ͷαʔϏε͕ Control Plane ʹ҆શʹ઀ଓ͢Δʹ͸ Google Cloud ͷೝূ৘ใΛ࢖ͬͨ TLS ͕ඞཁ ఏڙ͢Δػೳ: • google_default credentials ͷࣗಈऔಘ • Bootstrap ͷ channel_creds ͱ࿈ܞ • Cloud Run ͷαʔϏεΞΧ΢ϯτೝূ github.com/kauche/grpc-go-xds-tls-credentials
25. ։ൃ؀ڥ Cloud Run API Emulator ϩʔΧϧ։ൃͰ xDS ͷಈ࡞ݕূΛՄೳʹ͢ΔΤϛϡϨʔλ cloud-run-service-router-xds ͸Cloud Run Admin API ʹґଘ → ϩʔΧϧͰಈ͔͢ʹ͸ΤϛϡϨʔλ͕ඞཁ $ docker run --publish 8000:8000 ghcr.io/kauche/cloud-run-api-emulator:0.0.3 docker compose ͰҰൃىಈ: compose.yaml ʹ Control Plane + ΤϛϡϨʔλ + ֤αʔϏεΛఆٛ → ϩʔΧϧͰϒϥϯνϧʔςΟϯάͷ E2E ςετ͕Մೳ github.com/kauche/cloud-run-api-emulator
26. ࣮૷ ϔομʔ఻ൖͷ࢓૊Έ x-branch ϔομʔ͕ͲͷΑ͏ʹ఻ൖ͞ΕΔ͔ Mobile App x-branch: feature/payment Λ෇༩ grpc-gateway xDS Ͱ Service A (branch) ʹϧʔςΟϯάɻϔομʔ͸ಁա Service A (branch) gRPC metadata ͔Β x-branch Λऔಘ͠ɺoutgoing context ʹ఻ൖ Service B (branch) grpc-go xDS ͕ x-branch ʹج͖ͮ branch endpoint ʹ઀ଓ gRPC metadata Λ࢖ͬͨϔομʔ఻ൖ͸ Go ͷ grpc.UnaryInterceptor Ͱڞ௨Խ
27. Before / After Before • खಈͰ޲͖ઌมߋ͕ඞཁ • αʔϏεؒ௨৴ͷςετෆՄ • ඇಉظ௨৴ͷϒϥϯνςετෆՄ • QA ͷฒߦݕূ͕Ͱ͖ͳ͍ • ։ൃऀ͕ΠϯϑϥΛҙࣝ͢Δඞཁ After • ϔομʔ෇༩͚ͩͰࣗಈϧʔςΟϯά • αʔϏεؒ௨৴΋ࣗಈ௥ै • Pub/Sub ΋ϒϥϯν؀ڥʹ౸ୡ • ෳ਺ϒϥϯνΛಉ࣌ݕূ • Platform Team ͕ج൫ఏڙ
28. OSS ެ։͍ͯ͠Δπʔϧ܈ kauche/cloud-run-service-router-xds xDS Control Plane ຊମɻCloud Run ϦϏδϣϯΛ؂ࢹ͠ Snapshot Λੜ੒ ˒ 4 kauche/grpc-go-xds-tls-credentials Control Plane ઀ଓ༻ͷ TLS ΫϨσϯγϟϧϥΠϒϥϦ ˒ 4 kauche/cloud-run-api-emulator ϩʔΧϧ։ൃ༻ͷ Cloud Run Admin API ΤϛϡϨʔλ ˒ 12 ͢΂ͯ github.com/kauche Ͱެ։த
29. ·ͱΊ 1. طଘͷ grpc-gateway / grpc-go ʹ xDS Λ૊ΈࠐΉ͜ͱͰ Πϯϑϥͷ࠶ߏஙͳ͠ʹϒϥϯνผ QA ؀ڥΛ࣮ݱ 2. ಉظɾඇಉظΛʮಛघϔομʔ + xDSʯʹ౷Ұ ୯ҰͷϧʔςΟϯάػߏͰ֦ுੑΛ֬อ 3. Go (go-control-plane) Ͱ Control Plane Λ࣮૷ طଘͷٕज़ελοΫͱͷࣗવͳ౷߹ 4. OSS ͱͯ͠ެ։ → ஌ݟΛίϛϡχςΟʹؐݩ github.com/kauche
30. Thank you! xDS Λ׆༻ͨ͠αʔϏεσΟεΧόϦʔͰ࣮ݱ͢Δ ϒϥϯνผ QA ؀ڥͷߏஙख๏