A Developer's Introduction to Service Mesh

In the ideal development practice, we secure, shape, and observe traffic between services with a single line of code. However, most environments have multiple types of applications running many versions across diverse workloads and platforms, from containers to public cloud to private datacenter. With so many platforms and application frameworks, you cannot use the same code libraries across all services to shape traffic, secure communications, or enhance observability. How can we reduce the development and operational complexity?

In this session, you dive into why and how a service mesh can alleviate the management complexity of shaping, securing, and observing traffic across multiple platforms and environments. First, I’ll provide a short introduction to the session’s setup, which uses HashiCorp Consul and Envoy proxy on Kubernetes. Then, you will learn how to implement and debug traffic shaping and certificate management in the mesh. Finally, you will configure tracing and metrics collection for your service mesh application and examine the telemetry in Prometheus and Jaeger. We’ll compare the experience of using a service mesh to various programming language implementations and discuss how to extend the mesh across different workloads.

Transcript

1. Copyright © 2020 HashiCorp April 2022 A Developer’s Introduction to Service Mesh 1
2. We must have a service mesh. 2
3. We want to secure service-to-service communication with mTLS. 3
4. Service Discovery DNS Load Balancing Round-robin Weighted Security mTLS Authorization Telemetry Metrics Tracing Traffic Management Circuit Breaking Retries 4
5. 5 PROXY EXPENSE V1 (.NET) PROXY REPORT V2 (.NET) REPORT V3 (.NET) EXPENSE V2 (JAVA) PROXY PROXY REPORT SERVICE EXPENSE SERVICE Service Mesh github.com/joatmon08/expense-report
6. CODE EDITOR -- - apiVersion: apps/v 1 kind: Deploymen t metadata : name: expens e labels : app: expens e release: v 1 spec : replicas: 1 selector : matchLabels : app: expens e release: v 1 template : metadata : annotations : prometheus.io/scrape: "true " consul.hashicorp.com/connect-inject: "true" https://github.com/sqshq/piggymetrics 6
7. Service Discovery Service Mesh Load Balancing Service Mesh Security Service Mesh 7 Telemetry Service Mesh?? Traffic Management Service Mesh
8. Service Discovery 8
9. Choosing an abstraction. Service Mesh. Proxy registration Application-Side Options. Libraries (e.g., Eureka) DNS Kubernetes services 9
10. CODE EDITOR @SpringBootApplicatio n @EnableDiscoveryClien t ## omitte d public class AccountApplication { public static void main(String[] args) { SpringApplication.run(AccountApplication.class, args) ; } } https://github.com/sqshq/piggymetrics 10
11. 11 PROXY EXPENSE V1 (.NET) PROXY REPORT V2 (.NET) REPORT V3 (.NET) EXPENSE V2 (JAVA) PROXY PROXY REPORT SERVICE EXPENSE SERVICE Service Mesh
12. TERMINAL > wget -qO- 127.0.0.1:19000/cluster s ## omitted for clarit y expense.default.eastus.internal.***.consul::10.244.1.7:20000: :cx_active:: 1 jaeger_9411::10.0.244.168:9411::cx_active:: 1 expense- v2.default.eastus.internal.***.consul::10.244.0.16:20000::cx_ active::1
13. Load Balancing 13
14. Choosing an abstraction. Service Mesh. Proxy configuration Application-Side Options. Libraries (e.g., Feign) Load balancers DNS 14
15. CODE EDITOR @SpringBootApplicatio n @EnableFeignClient s ## omitte d public class AccountApplication { public static void main(String[] args) { SpringApplication.run(AccountApplication.class, args) ; } } https://github.com/sqshq/piggymetrics 15
16. 16 PROXY EXPENSE V1 (.NET) PROXY REPORT V2 (.NET) REPORT V3 (.NET) EXPENSE V2 (JAVA) PROXY PROXY REPORT SERVICE EXPENSE SERVICE Service Mesh 50% TO V1 50% TO V2
17. TERMINAL > wget localhost:19000/config_dum p ## omitted for clarit y "dynamic_route_configs": [ { "route_config": { "@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration" , "name": "expense" , ## omitted for clarit y "route": { "weighted_clusters": { "clusters": [ { "name": “expense.default.eastus.internal.***.consul” , "weight": 500 0 } , { "name": “expense-v2.default.eastus.internal.***.consul” , "weight": 500 0 } ] , "total_weight": 1000 0 }
18. Security 18
19. Choosing an abstraction. Service Mesh. mTLS between proxies Proxy filters (TCP & HTTP) Application-Side Options. Libraries Write your own API authorization servers OIDC/JWT 19
20. CODE EDITOR builder.Services.AddAuthentication ( CertificateAuthenticationDefaults.AuthenticationScheme ) .AddCertificate(options = > { options.Events = new CertificateAuthenticationEvent s { OnCertificateValidated = context = > { var validationService = context.HttpContext.RequestService s .GetRequiredService<ICertificateValidationService>() ; if (validationServic e .ValidateCertificate(context.ClientCertificate) ) { ## omitte d } return Task.CompletedTask ; } } ; }); https://docs.microsoft.com/en-us/aspnet/core/security/authentication/certauth?view=aspnetcore-6.0 20
21. 21 PROXY EXPENSE V1 (.NET) PROXY REPORT V2 (.NET) REPORT V3 (.NET) EXPENSE V2 (JAVA) PROXY PROXY REPORT SERVICE EXPENSE SERVICE Service Mesh MTLS MTLS MTLS MTLS MTLS
22. TERMINAL > wget localhost:19000/config_dum p ## omitted for clarit y "transport_socket": { "name": "tls" , "typed_config": { "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext" , "common_tls_context": { "tls_params": {} , "tls_certificates": [ { "certificate_chain": { "inline_string": "-----BEGIN CERTIFICATE——\n***\n——END CERTIFICATE-----\n " } , "private_key": { "inline_string": "[redacted] " } } ] , "validation_context": { "trusted_ca": { "inline_string": "-----BEGIN CERTIFICATE——\n***\n——END CERTIFICATE-----\n " } } } , "require_client_certificate": tru e
23. CODE EDITOR @SpringBootApplicatio n @EnableOAuth2Clien t @EnableGlobalMethodSecurity(prePostEnabled = true ) ## omitte d public class AccountApplication { public static void main(String[] args) { SpringApplication.run(AccountApplication.class, args) ; } } https://github.com/sqshq/piggymetrics 23
24. 24 PROXY EXPENSE V1 (.NET) PROXY REPORT V2 (.NET) REPORT V3 (.NET) EXPENSE V2 (JAVA) PROXY PROXY REPORT SERVICE EXPENSE SERVICE Service Mesh ALLOW REPORT TO ACCESS /API/EXPENSE/TRIP
25. TERMINAL > wget localhost:19000/config_dum p ## omitted for clarit y { "@type": "type.googleapis.com/envoy.admin.v3.ListenersConfigDump" , "dynamic_listeners": [ "active_state": { "filter_chains": [ { "filters": [ { "http_filters": [ { "name": "envoy.filters.http.rbac" , "typed_config": { "@type": "type.googleapis.com/envoy.extensions.filters.http.rbac.v3.RBAC" , "rules": { "policies": { "consul-intentions-layer7-0": { "permissions": [ { "and_rules": { "rules": [ { "url_path": { "path": { "prefix": "/api/expense/trip " } } } , "principals": [ { "authenticated": { "principal_name": { "safe_regex": { "regex": "^spiffe://[^/]+/ns/default/dc/[^/]+/svc/report$ " }
26. Traffic Management 26
27. Choosing an abstraction. Service Mesh. Proxy configuration Envoy circuit breaker: set maximum, pending, and current connections for upstream service instances Envoy outlier detection: eject service instance if failures reach threshold Application-Side Options. Libraries Write your own 27
28. CODE EDITOR @SpringBootApplicatio n @EnableCircuitBreake r ## omitte d public class AccountApplication { public static void main(String[] args) { SpringApplication.run(AccountApplication.class, args) ; } } https://github.com/sqshq/piggymetrics 28
29. CODE EDITOR var retryPolicy = GetRetryPolicy() ; var circuitBreakerPolicy = GetCircuitBreakerPolicy() ; services.AddHttpClient<IBasketService, BasketService>( ) .SetHandlerLifetime(TimeSpan.FromMinutes(5) ) .AddHttpMessageHandler<HttpClientAuthorizationDelegatingHandler>( ) .AddPolicyHandler(retryPolicy ) .AddPolicyHandler(circuitBreakerPolicy) ; static IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicy( ) { return HttpPolicyExtension s .HandleTransientHttpError( ) .CircuitBreakerAsync(5, TimeSpan.FromSeconds(30)) ; } https://docs.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/implement-circuit-breaker-pattern 29
30. 30 PROXY EXPENSE V1 (.NET) PROXY REPORT V2 (.NET) REPORT V3 (.NET) EXPENSE V2 (JAVA) PROXY PROXY REPORT SERVICE EXPENSE SERVICE Service Mesh IF HTTP 5XX > 3 ERRORS, EJECT SERVICE. DIVERT TRAFFIC TO ANOTHER SERVICE VERSION
31. CODE EDITOR apiVersion: consul.hashicorp.com/v1alpha 1 kind: ServiceDefault s metadata : name: repor t spec : protocol: htt p upstreamConfig : overrides : - name: expens e passiveHealthCheck : interval: "10s " maxFailures: 3 https://www.consul.io/docs/connect/proxies/envoy#passive_health_check 31
32. Telemetry 32
33. Two sources of telemetry. Service Mesh. Proxy metrics (merge with application) Proxy traces (only work if you have application traces) Application-Side Options. Libraries (OpenTelemetry, Prometheus exporters) Write your own 33
34. You still need instrumentation for your application.
35. CODE EDITOR builder.Services.AddOpenTelemetryMetrics(b = > { b.AddPrometheusExporter(o = > { o.StartHttpListener = true ; o.HttpListenerPrefixes = new string[] { metricsEndpoint } ; } ) .AddHttpClientInstrumentation( ) .AddAspNetCoreInstrumentation() ; }) ; builder.Services.AddOpenTelemetryTracing(b = > { b.AddSource(serviceName ) .SetResourceBuilder ( ResourceBuilder.CreateDefault( ) .AddService(serviceName: serviceName, serviceVersion: serviceVersion) ) .AddSqlClientInstrumentation(o = > { o.SetDbStatementForText = true ; } ) .AddHttpClientInstrumentation( ) .AddAspNetCoreInstrumentation( ) .AddZipkinExporter(o = > { o.Endpoint = new Uri(tracingUri) ; }) ; }); https://github.com/joatmon08/expense-report/tree/main/expense/dotnet 35
36. TERMINAL > java \ -javaagent:/app/agent/opentelemetry-javaagent.jar \ -Dotel.traces.exporter=zipkin \ -Dotel.metrics.exporter=prometheus \ -Dotel.resource.attributes=service.name=expense \ -jar /app/spring-boot-application.jar
37. 37 PROXY EXPENSE V1 (.NET) PROXY REPORT V2 (.NET) REPORT V3 (.NET) EXPENSE V2 (JAVA) PROXY PROXY REPORT SERVICE EXPENSE SERVICE Service Mesh CONFIGURE SERVICE MESH TO EXPOSE PROXY TRACES
38. TERMINAL > wget localhost:19000/config_dum p "tracing": { "http": { "name": "envoy.tracers.zipkin" , "typed_config": { "@type": "type.googleapis.com/ envoy.config.trace.v3.ZipkinConfig" , "collector_cluster": "jaeger_9411" , "collector_endpoint": "/api/v2/spans" , "shared_span_context": true , "collector_endpoint_version": "HTTP_JSON " } } }
39. 39 PROXY EXPENSE V1 (.NET) PROXY REPORT V2 (.NET) REPORT V3 (.NET) EXPENSE V2 (JAVA) PROXY PROXY REPORT SERVICE EXPENSE SERVICE Service Mesh CONFIGURE SERVICE MESH TO EXPOSE PROXY METRICS
40. CODE EDITOR apiVersion: consul.hashicorp.com/v1alpha 1 kind: ProxyDefault s metadata : name: globa l spec : config : protocol: htt p envoy_prometheus_bind_addr: "0.0.0.0:20200 " https://www.consul.io/docs/connect/proxies/envoy#passive_health_check 40
41. 41 PROXY EXPENSE V1 (.NET) PROXY REPORT V2 (.NET) REPORT V3 (.NET) EXPENSE V2 (JAVA) PROXY PROXY REPORT SERVICE EXPENSE SERVICE Service Mesh MERGE METRICS FROM APPLICATION TO PROXY METRICS ENDPOINT
42. CODE EDITOR apiVersion: apps/v 1 kind: Deploymen t metadata : name: expens e labels : app: expens e release: v 1 spec : replicas: 1 selector : matchLabels : app: expens e release: v 1 template : metadata : annotations : prometheus.io/scrape: "true " consul.hashicorp.com/connect-inject: "true " consul.hashicorp.com/enable-metrics-merging: "true " consul.hashicorp.com/service-metrics-port: "9464 " https://www.consul.io/docs/connect/proxies/envoy#passive_health_check 42
43. TERMINAL > wget localhost:20200/metric s # TYPE envoy_cluster_upstream_rq_time histogra m # TYPE runtime_jvm_gc_count_total counte r # HELP runtime_jvm_gc_count_total The number of collections that have occurred for a given JVM garbage collect
44. Service Discovery Service Mesh Load Balancing Service Mesh Security Service Mesh 44 Telemetry Service Mesh?? Traffic Management Service Mesh
45. github.com/joatmon08 
 /expense-report
46. Rosemary Wang HashiCorp 
 she/her 
 
 @joatmon08 Thank you!