production_ready_envoy

本番環境でEnvoyを導入するためにやったこと
Envoy Meetup Tokyo #1 の発表資料
https://envoytokyo.connpass.com/event/157711/

Transcript

1. ຊ൪؀ڥͰEnvoyΛಋೖ͢ΔͨΊʹ΍ͬͨ͜ͱ
2. Outline 1. Ͳ͏ͯ͠EnvoyΛಋೖͨ͠ͷ? 2. ͜Μͳײ͡Ͱಋೖ͠·ͨ͠ 3. Configͷڞ௨Խ 4. ϩά΍ϝτϦΫεपΓͷઃఆ 5. ࣮ࡍʹӡ༻ͯ͜͠·ͬͨ͜ͱ 6. ϝτϦΫεΛऔΓ͜΅͞ͳ͍ҝʹ
3. Ͳ͏ͯ͠EnvoyΛಋೖ͠ ͨͷ?
4. gRPCͷBalancingͷͨΊʹಋೖ αʔϏεσΟεΧόϦ͸Headless Services
5. ͜Μͳײ͡Ͱಋೖ
6. Sidecarύλʔϯ
7. Configͷڞ௨Խ
8. ϚΠΫϩαʔϏεຖʹ։ൃऀ͕ҧ͏ EnvoyͷΩϟονΞοϓͷίετΛݮΒ͍ͨ͠ͷͰɺ શϚΠΫϩαʔϏεڞ௨ͷconfigΛ࡞੒͠ɺ ͦΕΛ಺แͨ͠envoyΠϝʔδΛར༻͢Δ͜ͱʹͨ͠ɻ 4 ݸผʹConfigMapΛ࡞Βͳ͍͍ͯ͘ͷͱɺઃఆͷϨϕϧײ Λ౷Ұ͢Δ͜ͱ͕Ͱ͖ͨɻ
9. 4 DockerfileΛॻ͍ͯΠϝʔδΛ༻ҙ 4 YAMLͷΞϯΧʔͱΤΠϦΞεͰهड़ྔΛݮΒ͢ type: STRICT_DNS lb_policy: ROUND_ROBIN connect_timeout: 0.25s drain_connections_on_host_removal: true http2_protocol_options: {} health_checks: *health_checks outlier_detection: *outlier_detection circuit_breakers: *circuit_breakers
10. ϩά΍ϝτϦΫεपΓͷ ઃఆ
11. ΞΫηεϩάͷઃఆ 4 %RESPONSE_FLAGS%Ͱresponceͷ৘ใΛΈΔ͙Β͍ access_log: - name: envoy.file_access_log config: path: "/dev/stdout" json_format: start_time: "%START_TIME%" method: "%REQ(:METHOD)%" path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%" protocol: "%PROTOCOL%" response_code: "%RESPONSE_CODE%" response_flags: "%RESPONSE_FLAGS%" bytes_rcvd: "%BYTES_RECEIVED%" bytes_snt: "%BYTES_SENT%" duration: "%DURATION%" x-envoy-upstream-svc-time: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%" x-forwarded-for: "%REQ(X-FORWARDED-FOR)%" useragent: "%REQ(USER-AGENT)%" x-request-id: "%REQ(X-REQUEST-ID)%" backend_address: "%UPSTREAM_HOST%" client: "%DOWNSTREAM_REMOTE_ADDRESS%" referer: "%REQ(REFERER)%" response_duration: "%RESPONSE_DURATION%" upstream_transport_failure_reason: "%UPSTREAM_TRANSPORT_FAILURE_REASON%"
12. Circuit Breaking ա৒ͳϦΫΤετͱ͔ίωΫγϣϯ͕͖ͨͱ͖ʹɺΞϓϦ͕Ԡ ౴ෆՄʹͳΔͷΛ๷͙ circuit_breakers: &circuit_breakers thresholds: - priority: DEFAULT max_connections: 1024 max_pending_requests: 1024 max_requests: 1024 max_retries: 3
13. outlier_detection pod΁ͷ500ܥ΍200ܥͷճ਺ΛΈͯΫϥελ͔ΒऔΓআ͘ ੍͔ޚ͍ͯ͠Δ consecutive_5xx: 5 interval: 5s base_ejection_time: 30s max_ejection_percent: 10 enforcing_consecutive_5xx: 100 enforcing_success_rate: 100 success_rate_minimum_hosts: 5 success_rate_request_volume: 100 success_rate_stdev_factor: 1900 consecutive_gateway_failure: 5 enforcing_consecutive_gateway_failure: 0 split_external_local_origin_errors: true consecutive_local_origin_failure: 5 enforcing_consecutive_local_origin_failure: 100 enforcing_local_origin_success_rate: 100 failure_percentage_threshold: 85 enforcing_failure_percentage: 0 enforcing_failure_percentage_local_origin: 0 failure_percentage_minimum_hosts: 5 failure_percentage_request_volume: 50
14. healthcheck appଆͰHTTPͷΤϯυϙΠϯτΛੜ΍ͯ͠Readiness/ Liveness Prove ͰͷhealthcheckΛͨ͠ gRPCͷhealthcheck͸αΠυΧʔͷenvoy͔ΒͷΈୟ͘Α͏ ʹ͍ͯ͠Δ
15. EnvoyͷϝτϦΫε ࣮ࡍ͸Datadog APMͰऔಘͨ͠ϝτϦΫεΛΈ͍ͯΔ... annotations: ad.datadoghq.com/envoy.check_names: '["envoy"]' ad.datadoghq.com/envoy.init_configs: '[{}]' ad.datadoghq.com/envoy.instances: | [ { "stats_url": "http://%%host%%:8001/stats" } ]
16. ࣮ࡍʹӡ༻ͯ͠ࠔͬͨࣄ
17. pod ਺͕૿͑ΔͱϔϧενΣοΫͷgRPCΞΫηε͕ܶతʹ૿ ͑ͯ͠·͍API؂ࢹͰUNKNOWNͷΞϥʔτΛൃใ pass_through_mode: false ʹͯ͠ϔϧενΣοΫͷঢ়ଶΛ อ͓͍࣋ͯͯ͠ฦ͢Α͏ʹઃఆͨ͠ http_filters: - name: envoy.health_check typed_config: "@type": type.googleapis.com/envoy.config.filter.http.health_check.v2.HealthCheck pass_through_mode: false cluster_min_healthy_percentages: self-grpc: value: 100 headers: - name: ":path" exact_match: /healthz no_traffic_interval ΛσϑΥϧτ஋ 60s ʹͯ͠େྔͷϔϧ ενΣοΫΛૹ৴͠ͳ͍Α͏ʹ͍ͯ͠Δ
18. ϝτϦΫεΛऔΓ͜΅͞ ͳ͍ҝʹ
19. 1. drain_connections_on_host_removal Λtrueʹͯ͠ healthcheckͷࣦഊΛ଴ͨͣʹservice discovery͔Βআ֎ ͤ͞Δ 2. ΞϓϦέʔγϣϯΛىಈ͢ΔલʹenvoyΛىಈͤ͞Δ 4 http://localhost:8001/ready Λୟ͍ͯ200εςʔλ ε͕ฦ͖͔ͬͯͯΒΞϓϦΛىಈͤ͞Δ 3. envoy͕ऴྃ͢ΔલʹΞϓϦέʔγϣϯΛऴྃͤ͞Δ 4 ΞϓϦέʔγϣϯίϯςφ͔Βͷશͯͷ઀ଓ͕੾ΕΔ· Ͱ଴ͭγΣϧܳΛ͍ͯ͠Δ
20. ͓͠·͍