Protocol-Independent Context Propagation for Sharing Microservices in Multiple Environments IC2E 2023 Hiroya ONOE, Daisuke KOTANI, Yasuo OKABE 1 Slide URL: https://onoe.dev/picop-en

Background: Production-Like Environment Require Production-Like Environments in CI/CD For testing, staging, debugging, and previewing →Resources are wasted due to low access Microservice Architecture A single application consists of several microservices →A large number of microservices can easily lead to a high waste 2

Background: Production-Like Environment A large number of microservices lead to a high waste 3 https://github.com/microservices-demo/microservices-demo/blob/master/internal-docs/design.md Microservice Demo: Sock Shop

Background: Sharing Microservices Reduce resource usage by sharing stateless and non-updated services in multiple environments (proposed by Wantedly, Mercari, Lyft, and Ambassador Labs) 1. Propagate contexts that identify environments by adding them to requests 2. Route requests based on the contexts 4

Add contexts to HTTP header or gRPC metadata 1. Propagate contexts with OpenTelemetry OpenTelemetry Provide specifications and libraries for distributed tracing, including context propagation Background: Sharing Microservices 5 func handler(w http.ResponseWriter, r *http.Request) { propagator := otelprop.TextMapPropagator // Extract envID := h.propagator.Extract(r.Context(), “EnvID”) // Propagate log.Print(envID) // Inject ctx := propagator.Inject(“EnvID”, envID) http.Post(ctx, “https://service2.local/api”) ... }

Background: Sharing Microservices Add contexts to HTTP header or gRPC metadata 2. Route requests with Envoy and Istio 6 Istio Build service mesh with Envoy in Kubernetes Envoy A cloud-native proxy deployed as a sidecar Provide observability, telemetry, and traffic management with cloud-native applications

Background: Sharing Microservices Problem Propagating and Routing depend on specific protocols • Not available for other protocols, especially those that cannot have optional data (MySQL, Redis, MQTT, etc.) • Require instrumentation for each protocol 7 ?

Related Works: Context Propagation Distributed Tracing Trace distributed systems by adding contexts to requests and propagating them (Dapper, Pinpoint, X-Trace, Pip, and Canopy) Context Propagation Frameworks Reduce the effort in instrumenting by standardizing the context propagation mechanisms (Tracing Plane, Canopy, Pivot Tracing, and Pythia) →All methods depend on specific protocols and platforms 8

Proposal: PiCoP A framework to propagate contexts and route requests independently of application layer protocols Protocol Propagate contexts without interpreting application layer protocols Proxy Route requests to the appropriate environments by using the protocol 9

Proposal: Why Protocol Independence? • Enable to propagate contexts in any application layer protocol, including those that cannot have optional data • Enable to use the same proxy to route requests of any application layer protocol 10

Proposal: PiCoP Protocol Add contexts to the front of the TCP byte stream (as with PROXY Protocol) →Available regardless of the type of application layer protocols PROXY Protocol Protocol-independently propagate source client information through a reverse proxy server the concept is similar, but the content and format of propagated information is different 11 PiCoP Protocol

Proposal: PiCoP Protocol Extend the signature of PROXY Protocol version 2 →No conflicts with application layer protocols 12 Byte Description 1-12 The same signature as PROXY Protocol version 2 13 The upper 4 bits: the signature The lower 4 bits: the version

Proposal: PiCoP Protocol Satisfy Propagators API defined by OpenTelemetry →Easy to instrument into applications →Easy to use for context propagation for other purposes Propagate environment ID by using PiCoP Protocol 13 Byte Description 14,15 The byte length of the context 16- The context in the form of key-value pairs (Satisfied Propagators API)

Proposal: PiCoP Proxy Route requests of any application layer protocol to the appropriate environment based on environment ID Receive the correspondence between an environment ID and a destination service in advance(route information) 14 Environment ID Destination Service EnvA Service[EnvA] EnvB Service[Base](default route) Other Service[Base](default route)

Proposal: PiCoP Proxy Some services do not accept PiCoP Protocol (e.g., MySQL server) Services at the end of request processing in an application do not need to propagate environment ID →Enable to choose whether or not to propagate PiCoP Protocol 15

Implementation Proxy Controller Manage proxy and route information in DB Receive information from administrator Send information to each proxy Support scaling out of proxies →For large cloud-scale clusters 16

Evaluation: Proxy Communication Delay • Send HTTP requests with 1kB payload to NGINX server • Measure response time of 10000 requests in 10 seconds • Set the number of simultaneous connections from 1 to 64 • Add ‘Env-Id:Main’ context to HTTP header or PiCoP Protocol Four Conditions With vs. Without PiCoP Proxy PiCoP Proxy vs. Istio Proxy 17

Evaluation: Proxy Communication Delay Comparison with no PiCoP Proxy (base vs. base+picop) 18 3.2ms-12.3ms slower 6.7ms-13.3ms slower

Comparison with Istio Proxy (base+gw+istio vs. base+gw+picop) Almost equivalent to Istio, widely used in existing methods →The delay of PiCoP Proxy is within practical range Evaluation: Proxy Communication Delay 19 almost equal 0.3ms-2.7ms slower

Evaluation: Resource Reduction • Load PiCoP Proxy and NGINX server by continuously sending HTTP requests • Scale out to 80% or less CPU utilization for all containers • Compare the number of containers of proxy and NGINX server • Each container’s resource is limited • Change the number of environments from 1 to 100 • Send 100 or 1000 requests per second in total for all environments from clients equal to the number of environments 20

Evaluation: Resource Reduction PiCoP reduce resource usage when the number of environments exceeds a specific value The more environments there are, the more significant the reduction is 21

Discussion: Protocol Independence Inter-Service Propagation Intra-Service Propagation 22

Discussion: Protocol Independence Inter-Service Propagation PiCoP can propagate contexts between services in the same mechanism and proxy for all protocols, including those that cannot have optional data 23 Cannot have optional data Can have optional data MySQL, PostgreSQL, Memcached, Redis, MongoDB Wire Protocol, MQTT, Kafka Wire Protocol, TDS(Microsoft SQL Server) HTTP, gRPC, AMQP, Cassandra Native Protocol

Discussion: Protocol Independence Inter-Service Propagation Restriction on connection reuse (e.g., persistent HTTP connections) Overhead of establishing a TCP connection each time 24

Discussion: Protocol Independence Inter-Service Propagation Need to implement a proxy for each transport layer protocol Available for UDP, QUIC, etc. in addition to TCP Add PiCoP Protocol to the front of the datagram or stream Unidirectional Communication Deploy PiCoP Proxy for services receiving requests Bidirectional Communication Deploy PiCoP Proxy for both services 25

Discussion: Protocol Independence Intra-Service Propagation In services in the middle of request processing in an application (e.g., a web server), instrumentation required (as with existing methods) 26 Need instrumentation libraries per each language and protocol Enable to use instrumentation libraries provided by OpenTelemetry

Discussion: Protocol Independence Intra-Service Propagation In services at the end of request processing in an application (e.g., a database), no instrumentation required The services do not need environment ID 27

Discussion: Protocol Independence Intra-Service Propagation In asynchronous processing (e.g., a message queue), need to extend PiCoP Proxy Unable to propagate contexts in one direction through asynchronous processing 28 Extended Proxy

Conclusion • Propose PiCoP for protocol-independent microservice sharing • PiCoP protocol propagates contexts without interpreting application layer protocols • PiCoP proxy routes requests to the appropriate environments by using the protocol • Show practicality in terms of communication delay and resource reduction • Show constraints on connection reuse and asynchronous processing • Available as OSS ( https://github.com/picop-rd ) Future Work • Resolve the above constraints • Evaluate at large cloud-scale clusters • Analysis the applicability of PiCoP protocol to context propagation for other purposes 29 Slide URL: https://onoe.dev/picop-en