Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How Many Proxies Do You Need

Liz Rice
October 25, 2022

How Many Proxies Do You Need

To provide app-level network functionality like L7 load balancing and TLS termination, Service Meshes use a proxy component that terminates L7 connections on behalf of applications. Traditionally the proxy has been co-located in each application pod as a sidecar container, but Cilium Service Mesh changed this with the innovation of sidecarless service mesh. Istio have also now adopted a sidecarless approach that was recently announced as Istio Ambient Mesh.

But “sidecarless” doesn’t mean “proxyless”! It’s a question of where you deploy the proxies, and how you create the relationship between apps and proxies. In this talk we’ll explore the pro’s and con’s of different models, and explain where eBPF makes a difference (and where it doesn’t) in not only network performance, but also to provide observability and security capabilities.

Liz Rice

October 25, 2022

More Decks by Liz Rice

Other Decks in Technology


  1. How many proxies do you need? Liz Rice | @lizrice

    Chief Open Source Officer, Isovalent Emeritus Chair, CNCF Technical Oversight Committee Thomas Graf | @tgraf_ CTO & Co-founder, Isovalent Chair, eBPF Foundation GB
  2. - Cilium Service Mesh beta tester, Jan 2022 “ While

    we're big fans of Envoy we're not hugely fond of the sidecar model and the extra latency & complexity involved ”
  3. userspace kernel Reduce resource usage Sidecars are a little bit

    inefficient [...] you have to allocate the RAM and CPU for that sidecar for kind of the worst-case usage that you expect for that pod. – Ethan Jackson, Ambient Mesh, Google, Kubernetes Podcast #189
  4. userspace pod containe r sidecar container userspace pod container sidecar

    container my-app.yaml containers: - name: my-app ... - name: my-app-init … - name: my-sidecar ... The operational cost of sidecars
  5. Delegating responsibility to user space userspace kernel Cilium eBPF delegates

    - L7 termination to Envoy proxy - L7 observability to Envoy proxy - L7 network policy to Envoy proxy - L7 identity to SPIFFE or cert-manager
  6. Traffic Management - L3/L4 forwarding & Load-balancing - Canary, Topology

    Aware Routing - Multi-Cluster Routing Security - Network Policy - mTLS Observability - Tracing, OpenTelemetry, & Metrics - HTTP, TLS, DNS, TCP, UDP, … eBPF Native (no proxy needed) Proxy Traffic Management - L7 Load-balancing & Ingress Resilience - Retries, L7 Rate Limiting Security - TLS Termination & Origination - L7 Network Policy* *Roadmap for eBPF Native When eBPF can’t support it Whenever possible
  7. Proxy per pod (sidecar model) userspace kernel - Share pod’s

    namespaces and cgroups → Resources for app + proxy - Proxy access to pod’s service account → Secrets / identity management directly from app Cilium Status: → Supported via Istio Integration on top of Cilium CNI
  8. Proxy on the node - Proxy is co-located on same

    node → No additional proxies needed on network → No ability to share proxy for single tenant across nodes Cilium Service Mesh Status: → Defaulting to Per-Node Model → Flexible deployment granularity on the roadmap userspace kernel
  9. Proxy on the network userspace kernel - Proxy is located

    on network → Requires additional network hops → Ability to share proxy for individual tenants across nodes Cilium Service Mesh Status: → Evaluating interest in ztunnel/HBONE to support Waypoint proxies userspace kernel
  10. Cilium network level encryption userspace kernel Encryption at L3 -

    no need to traverse proxy Uses node identity. Do you trust your nodes?
  11. Cilium Next-Gen Mutual Authentication - Works for any protocol (UDP,

    SCTP, …) - IPsec/Wireguard can use TLS negoiated service-specific keys - User space mTLS authentication - Proxy-free in-kernel datapath - Keeps secrets out of L7 proxies More information: https://isovalent.com/blog/post/2022-05-03-servicemesh-security
  12. Data plane Control plane Configuration Ingress Gateway API Services EnvoyConfig

    SPIFFE Network Policy Kubernetes cert-manager Cilium Service Mesh mTLS Traffic Management Identity Management Observability Envoy Secrets Service Discovery Stable Available in Dev Branch WIP / Roadmap +
  13. Original L7 Load-balancing standard in K8s Simple Supported since Cilium

    1.12 Services Ingress Layer 7 Traffic Management Options EnvoyConfig Use of K8s services with annotations Simple Support coming In Cilium 1.13 Pull Request: cilium/cilium#21244 Raw Envoy Config via CustomResource Advanced Users & Integrations Supported since Cilium 1.12 Gateway API Originally labelled Ingress v2. Richer in features. Simple Support for v0.5.1 coming in Cilium 1.13 Pull Request: cilium/cilium#21749