Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Developer's Introduction to Service Mesh

A Developer's Introduction to Service Mesh

In the ideal development practice, we secure, shape, and observe traffic between services with a single line of code. However, most environments have multiple types of applications running many versions across diverse workloads and platforms, from containers to public cloud to private datacenter. With so many platforms and application frameworks, you cannot use the same code libraries across all services to shape traffic, secure communications, or enhance observability. How can we reduce the development and operational complexity?

In this session, you dive into why and how a service mesh can alleviate the management complexity of shaping, securing, and observing traffic across multiple platforms and environments. First, I’ll provide a short introduction to the session’s setup, which uses HashiCorp Consul and Envoy proxy on Kubernetes. Then, you will learn how to implement and debug traffic shaping and certificate management in the mesh. Finally, you will configure tracing and metrics collection for your service mesh application and examine the telemetry in Prometheus and Jaeger. We’ll compare the experience of using a service mesh to various programming language implementations and discuss how to extend the mesh across different workloads.

Be8b596c46f4c9a1aec6a7586af33134?s=128

Rosemary Wang

April 28, 2022
Tweet

More Decks by Rosemary Wang

Other Decks in Technology

Transcript

  1. Copyright © 2020 HashiCorp April 2022 A Developer’s Introduction to

    Service Mesh 1
  2. We must have a service mesh. 2

  3. We want to secure service-to-service communication with mTLS. 3

  4. Service Discovery DNS Load Balancing Round-robin Weighted Security mTLS Authorization

    Telemetry Metrics Tracing Traffic Management Circuit Breaking Retries 4
  5. 5 PROXY EXPENSE V1 (.NET) PROXY REPORT V2 (.NET) REPORT

    V3 (.NET) EXPENSE V2 (JAVA) PROXY PROXY REPORT SERVICE EXPENSE SERVICE Service Mesh github.com/joatmon08/expense-report
  6. CODE EDITOR -- - apiVersion: apps/v 1 kind: Deploymen t

    metadata : name: expens e labels : app: expens e release: v 1 spec : replicas: 1 selector : matchLabels : app: expens e release: v 1 template : metadata : annotations : prometheus.io/scrape: "true " consul.hashicorp.com/connect-inject: "true" https://github.com/sqshq/piggymetrics 6
  7. Service Discovery Service Mesh Load Balancing Service Mesh Security Service

    Mesh 7 Telemetry Service Mesh?? Traffic Management Service Mesh
  8. Service Discovery 8

  9. Choosing an abstraction. Service Mesh. Proxy registration Application-Side Options. Libraries

    (e.g., Eureka) DNS Kubernetes services 9
  10. CODE EDITOR @SpringBootApplicatio n @EnableDiscoveryClien t ## omitte d public

    class AccountApplication { public static void main(String[] args) { SpringApplication.run(AccountApplication.class, args) ; } } https://github.com/sqshq/piggymetrics 10
  11. 11 PROXY EXPENSE V1 (.NET) PROXY REPORT V2 (.NET) REPORT

    V3 (.NET) EXPENSE V2 (JAVA) PROXY PROXY REPORT SERVICE EXPENSE SERVICE Service Mesh
  12. TERMINAL > wget -qO- 127.0.0.1:19000/cluster s ## omitted for clarit

    y expense.default.eastus.internal.***.consul::10.244.1.7:20000: :cx_active:: 1 jaeger_9411::10.0.244.168:9411::cx_active:: 1 expense- v2.default.eastus.internal.***.consul::10.244.0.16:20000::cx_ active::1
  13. Load Balancing 13

  14. Choosing an abstraction. Service Mesh. Proxy configuration Application-Side Options. Libraries

    (e.g., Feign) Load balancers DNS 14
  15. CODE EDITOR @SpringBootApplicatio n @EnableFeignClient s ## omitte d public

    class AccountApplication { public static void main(String[] args) { SpringApplication.run(AccountApplication.class, args) ; } } https://github.com/sqshq/piggymetrics 15
  16. 16 PROXY EXPENSE V1 (.NET) PROXY REPORT V2 (.NET) REPORT

    V3 (.NET) EXPENSE V2 (JAVA) PROXY PROXY REPORT SERVICE EXPENSE SERVICE Service Mesh 50% TO V1 50% TO V2
  17. TERMINAL > wget localhost:19000/config_dum p ## omitted for clarit y

    "dynamic_route_configs": [ { "route_config": { "@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration" , "name": "expense" , ## omitted for clarit y "route": { "weighted_clusters": { "clusters": [ { "name": “expense.default.eastus.internal.***.consul” , "weight": 500 0 } , { "name": “expense-v2.default.eastus.internal.***.consul” , "weight": 500 0 } ] , "total_weight": 1000 0 }
  18. Security 18

  19. Choosing an abstraction. Service Mesh. mTLS between proxies Proxy filters

    (TCP & HTTP) Application-Side Options. Libraries Write your own API authorization servers OIDC/JWT 19
  20. CODE EDITOR builder.Services.AddAuthentication ( CertificateAuthenticationDefaults.AuthenticationScheme ) .AddCertificate(options = > {

    options.Events = new CertificateAuthenticationEvent s { OnCertificateValidated = context = > { var validationService = context.HttpContext.RequestService s .GetRequiredService<ICertificateValidationService>() ; if (validationServic e .ValidateCertificate(context.ClientCertificate) ) { ## omitte d } return Task.CompletedTask ; } } ; }); https://docs.microsoft.com/en-us/aspnet/core/security/authentication/certauth?view=aspnetcore-6.0 20
  21. 21 PROXY EXPENSE V1 (.NET) PROXY REPORT V2 (.NET) REPORT

    V3 (.NET) EXPENSE V2 (JAVA) PROXY PROXY REPORT SERVICE EXPENSE SERVICE Service Mesh MTLS MTLS MTLS MTLS MTLS
  22. TERMINAL > wget localhost:19000/config_dum p ## omitted for clarit y

    "transport_socket": { "name": "tls" , "typed_config": { "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext" , "common_tls_context": { "tls_params": {} , "tls_certificates": [ { "certificate_chain": { "inline_string": "-----BEGIN CERTIFICATE——\n***\n——END CERTIFICATE-----\n " } , "private_key": { "inline_string": "[redacted] " } } ] , "validation_context": { "trusted_ca": { "inline_string": "-----BEGIN CERTIFICATE——\n***\n——END CERTIFICATE-----\n " } } } , "require_client_certificate": tru e
  23. CODE EDITOR @SpringBootApplicatio n @EnableOAuth2Clien t @EnableGlobalMethodSecurity(prePostEnabled = true )

    ## omitte d public class AccountApplication { public static void main(String[] args) { SpringApplication.run(AccountApplication.class, args) ; } } https://github.com/sqshq/piggymetrics 23
  24. 24 PROXY EXPENSE V1 (.NET) PROXY REPORT V2 (.NET) REPORT

    V3 (.NET) EXPENSE V2 (JAVA) PROXY PROXY REPORT SERVICE EXPENSE SERVICE Service Mesh ALLOW REPORT TO ACCESS /API/EXPENSE/TRIP
  25. TERMINAL > wget localhost:19000/config_dum p ## omitted for clarit y

    { "@type": "type.googleapis.com/envoy.admin.v3.ListenersConfigDump" , "dynamic_listeners": [ "active_state": { "filter_chains": [ { "filters": [ { "http_filters": [ { "name": "envoy.filters.http.rbac" , "typed_config": { "@type": "type.googleapis.com/envoy.extensions.filters.http.rbac.v3.RBAC" , "rules": { "policies": { "consul-intentions-layer7-0": { "permissions": [ { "and_rules": { "rules": [ { "url_path": { "path": { "prefix": "/api/expense/trip " } } } , "principals": [ { "authenticated": { "principal_name": { "safe_regex": { "regex": "^spiffe://[^/]+/ns/default/dc/[^/]+/svc/report$ " }
  26. Traffic Management 26

  27. Choosing an abstraction. Service Mesh. Proxy configuration Envoy circuit breaker:

    set maximum, pending, and current connections for upstream service instances Envoy outlier detection: eject service instance if failures reach threshold Application-Side Options. Libraries Write your own 27
  28. CODE EDITOR @SpringBootApplicatio n @EnableCircuitBreake r ## omitte d public

    class AccountApplication { public static void main(String[] args) { SpringApplication.run(AccountApplication.class, args) ; } } https://github.com/sqshq/piggymetrics 28
  29. CODE EDITOR var retryPolicy = GetRetryPolicy() ; var circuitBreakerPolicy =

    GetCircuitBreakerPolicy() ; services.AddHttpClient<IBasketService, BasketService>( ) .SetHandlerLifetime(TimeSpan.FromMinutes(5) ) .AddHttpMessageHandler<HttpClientAuthorizationDelegatingHandler>( ) .AddPolicyHandler(retryPolicy ) .AddPolicyHandler(circuitBreakerPolicy) ; static IAsyncPolicy<HttpResponseMessage> GetCircuitBreakerPolicy( ) { return HttpPolicyExtension s .HandleTransientHttpError( ) .CircuitBreakerAsync(5, TimeSpan.FromSeconds(30)) ; } https://docs.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/implement-circuit-breaker-pattern 29
  30. 30 PROXY EXPENSE V1 (.NET) PROXY REPORT V2 (.NET) REPORT

    V3 (.NET) EXPENSE V2 (JAVA) PROXY PROXY REPORT SERVICE EXPENSE SERVICE Service Mesh IF HTTP 5XX > 3 ERRORS, EJECT SERVICE. DIVERT TRAFFIC TO ANOTHER SERVICE VERSION
  31. CODE EDITOR apiVersion: consul.hashicorp.com/v1alpha 1 kind: ServiceDefault s metadata :

    name: repor t spec : protocol: htt p upstreamConfig : overrides : - name: expens e passiveHealthCheck : interval: "10s " maxFailures: 3 https://www.consul.io/docs/connect/proxies/envoy#passive_health_check 31
  32. Telemetry 32

  33. Two sources of telemetry. Service Mesh. Proxy metrics (merge with

    application) Proxy traces (only work if you have application traces) Application-Side Options. Libraries (OpenTelemetry, Prometheus exporters) Write your own 33
  34. You still need instrumentation for your application.

  35. CODE EDITOR builder.Services.AddOpenTelemetryMetrics(b = > { b.AddPrometheusExporter(o = > {

    o.StartHttpListener = true ; o.HttpListenerPrefixes = new string[] { metricsEndpoint } ; } ) .AddHttpClientInstrumentation( ) .AddAspNetCoreInstrumentation() ; }) ; builder.Services.AddOpenTelemetryTracing(b = > { b.AddSource(serviceName ) .SetResourceBuilder ( ResourceBuilder.CreateDefault( ) .AddService(serviceName: serviceName, serviceVersion: serviceVersion) ) .AddSqlClientInstrumentation(o = > { o.SetDbStatementForText = true ; } ) .AddHttpClientInstrumentation( ) .AddAspNetCoreInstrumentation( ) .AddZipkinExporter(o = > { o.Endpoint = new Uri(tracingUri) ; }) ; }); https://github.com/joatmon08/expense-report/tree/main/expense/dotnet 35
  36. TERMINAL > java \ -javaagent:/app/agent/opentelemetry-javaagent.jar \ -Dotel.traces.exporter=zipkin \ -Dotel.metrics.exporter=prometheus \

    -Dotel.resource.attributes=service.name=expense \ -jar /app/spring-boot-application.jar
  37. 37 PROXY EXPENSE V1 (.NET) PROXY REPORT V2 (.NET) REPORT

    V3 (.NET) EXPENSE V2 (JAVA) PROXY PROXY REPORT SERVICE EXPENSE SERVICE Service Mesh CONFIGURE SERVICE MESH TO EXPOSE PROXY TRACES
  38. TERMINAL > wget localhost:19000/config_dum p "tracing": { "http": { "name":

    "envoy.tracers.zipkin" , "typed_config": { "@type": "type.googleapis.com/ envoy.config.trace.v3.ZipkinConfig" , "collector_cluster": "jaeger_9411" , "collector_endpoint": "/api/v2/spans" , "shared_span_context": true , "collector_endpoint_version": "HTTP_JSON " } } }
  39. 39 PROXY EXPENSE V1 (.NET) PROXY REPORT V2 (.NET) REPORT

    V3 (.NET) EXPENSE V2 (JAVA) PROXY PROXY REPORT SERVICE EXPENSE SERVICE Service Mesh CONFIGURE SERVICE MESH TO EXPOSE PROXY METRICS
  40. CODE EDITOR apiVersion: consul.hashicorp.com/v1alpha 1 kind: ProxyDefault s metadata :

    name: globa l spec : config : protocol: htt p envoy_prometheus_bind_addr: "0.0.0.0:20200 " https://www.consul.io/docs/connect/proxies/envoy#passive_health_check 40
  41. 41 PROXY EXPENSE V1 (.NET) PROXY REPORT V2 (.NET) REPORT

    V3 (.NET) EXPENSE V2 (JAVA) PROXY PROXY REPORT SERVICE EXPENSE SERVICE Service Mesh MERGE METRICS FROM APPLICATION TO PROXY METRICS ENDPOINT
  42. CODE EDITOR apiVersion: apps/v 1 kind: Deploymen t metadata :

    name: expens e labels : app: expens e release: v 1 spec : replicas: 1 selector : matchLabels : app: expens e release: v 1 template : metadata : annotations : prometheus.io/scrape: "true " consul.hashicorp.com/connect-inject: "true " consul.hashicorp.com/enable-metrics-merging: "true " consul.hashicorp.com/service-metrics-port: "9464 " https://www.consul.io/docs/connect/proxies/envoy#passive_health_check 42
  43. TERMINAL > wget localhost:20200/metric s # TYPE envoy_cluster_upstream_rq_time histogra m

    # TYPE runtime_jvm_gc_count_total counte r # HELP runtime_jvm_gc_count_total The number of collections that have occurred for a given JVM garbage collect
  44. Service Discovery Service Mesh Load Balancing Service Mesh Security Service

    Mesh 44 Telemetry Service Mesh?? Traffic Management Service Mesh
  45. github.com/joatmon08 
 /expense-report

  46. Rosemary Wang HashiCorp 
 she/her 
 
 @joatmon08 Thank you!