Upgrade to Pro — share decks privately, control downloads, hide ads and more …

IPv6 First Container Network Design Practices

IPv6 First Container Network Design Practices

2026.11.21
JAWS-UG山梨 【第14回】- 最凶のエージェント、上陸!!
https://jaws-ug-yamanashi.connpass.com/event/411061/

GitHub
https://github.com/MASAKIOKUDA-eng/ContainerNetwork-Research-handson

モブエンジニア

November 19, 2026

Transcript

30 slides
  1. IPv6 First Container Network Design Practices Measuring PrivateLink, VPC endpoints,

    and VPC Lattice for speed and security Mob Engineer (@mob_engineer) JAWS-UG Yamanashi #14 | November 21, 2026
  2. Who This Talk Is For / What You Will Take

    Away For engineers who connect containers across VPCs and plan for IPv6. Who this talk is for What you will take away 1 You run containers on Amazon ECS across several VPCs or accounts 1 How to design IPv6 per subnet tier, not per VPC 2 You want to cut public IPv4 costs and are planning for IPv6 2 Measured speed and security differences of three private paths 3 You are choosing between PrivateLink, VPC endpoints, and VPC Lattice 3 A simple way to choose the right path for each service Icons: AWS Architecture Icons https://aws.amazon.com/architecture/icons/ 2
  3. What I Will Cover / What I Will Not Cover

    I focus on private connectivity between services, with measured numbers. I will cover I will not cover ✓ Service to service paths across VPCs in an IPv6 first design ✗ IPv6 basics such as address format and subnetting ✓ Numbers measured in my own lab (Tokyo Region, November 19, 2026) ✗ Internet egress design (NAT64, DNS64, egress only gateway) ✓ Design practices and a guide to choose a path ✗ Transit Gateway, EKS specifics, and detailed pricing 3
  4. About Me Pen name:  Mob Engineer (@mob_engineer) Job: 

    SRE / Project Lead at a system integrator Favorite tech areas:  Cloud (AWS, Azure, OCI)  Containers (k8s), networking (virtual)  Observability operations (Dynatrace) Hobbies:  Running communities and conferences  Speaking and blogging (less often lately) In one line:  An engineer having fun growing communities outside work! 4
  5. Agenda From the background to the lab, then the results

    and how to choose. 1 2 3 4 5 Why IPv6 First Now The Lab Speed Security How to Choose Background and the real problem Two VPCs, three paths, no internet Latency, connections, IPv6 vs IPv4 Layers, matrix, what the provider sees Decision guide and takeaways 5
  6. 01 Section 1 of 5 1 2 3 4 5

    Why IPv6 First Now Background and the real problem
  7. Why IPv6 First Now Three changes made IPv6 first a

    realistic choice for containers. February 1, 2024 Public IPv4 addresses are charged $0.005 per IP per hour September 29, 2025 Amazon ECS supports IPv6 only tasks and services No IPv4 address needed November 10, 2025 Amazon S3 supports IPv6 on VPC endpoints No additional cost The question is no longer whether we can use IPv6, but how we design for it. IPv4: https://aws.amazon.com/blogs/aws/new-aws-public-ipv4-address-charge-public-ip-insights/ ECS: https://aws.amazon.com/about-aws/whats-new/2025/09/amazon-ecs-ipv6-only-support S3: https://aws.amazon.com/about-aws/whats-new/2025/11/ipv6-amazon-s3-gateway-interface-vpc-endpoints Icons: AWS Architecture Icons https://aws.amazon.com/architecture/icons/ 7
  8. The Real Problem Turning on IPv6 is easy. Connecting services

    across VPCs and accounts is the hard part. Team A account 10.0.0.0/16 Team B account 10.0.0.0/16 Same IPv4 range: no simple routing Three building blocks that work even when IPv4 ranges overlap AWS PrivateLink VPC endpoints Amazon VPC Lattice Icons: AWS Architecture Icons https://aws.amazon.com/architecture/icons/ 8
  9. 02 Section 2 of 5 1 2 3 4 5

    The Lab Two VPCs, three paths, no internet
  10. The Lab Same container behind each path, same IPv4 range,

    and no internet path. AWS Cloud 1 Region: ap-northeast-1 (Tokyo) Consumer VPC: 10.0.0.0/16, dual stack Endpoint service with acceptance, internal NLB Provider VPC: 10.0.0.0/16, dual stack Private subnets 2 Private subnets AWS PrivateLink Internal NLB Task: svc-pl 2 EC2 (k6) VPC Lattice 3 Task: baseline Task: svc-lattice Amazon S3 S3 interface endpoint S3 gateway endpoint Lab and results: ContainerNetwork-Research-handson (verification report 2026-11-19) https://github.com/MASAKIOKUDA-eng/ContainerNetwork-Research-handson Icons: AWS Architecture Icons https://aws.amazon.com/architecture/icons/ VPC Lattice VPC association, IAM auth policy 0 1 Interface endpoint AWS PrivateLink 3 VPC endpoints S3 gateway and interface, endpoint policies 0 Baseline k6 task calls the NLB directly No IGW, no NAT AWS CDK + cdkd 10
  11. Practice 1: Decide IPv6 per Subnet Tier IPv6 first does

    not mean IPv6 only everywhere. I decide it per subnet tier. Pitfall I hit Edge tier Dual stack NLB In ap-northeast-1, the ec2messages endpoint supports IPv4 only, so my first deploy failed. I created only that endpoint as IPv4. SSM Agent 3.3.40 or later works with ssm and ssmmessages only. Endpoints Workload tier Dual stack, then IPv6 only ECS tasks Dependencies Reachable over IPv6 (dual stack endpoints) ECR Logs SSM S3 Lab and results: ContainerNetwork-Research-handson (verification report 2026-11-19) https://github.com/MASAKIOKUDA-eng/ContainerNetwork-Research-handson Icons: AWS Architecture Icons https://aws.amazon.com/architecture/icons/ 11
  12. Three Paths in One Picture Keep one question in mind:

    does the provider know who is calling? AWS PrivateLink Layer What you expose Access control Caller identity at the provider VPC endpoints Amazon VPC Lattice L4 L4 or route table L7 One NLB port An AWS service API Services and paths Acceptance, security group No Icons: AWS Architecture Icons https://aws.amazon.com/architecture/icons/ Endpoint policy, security group Only aws:SourceVpce Auth policy (IAM), security group Yes: source VPC and IAM principal 12
  13. 03 Section 3 of 5 1 2 3 4 5

    Speed Latency, connections, IPv6 vs IPv4
  14. Speed 1: Latency PrivateLink adds almost nothing. VPC Lattice adds

    about 2 to 3 ms as an L7 proxy. p50 (ms) p99 (ms) 7 6.44 6.27 PrivateLink, p50 vs baseline 6 5 3 2 3.80 3.65 4 1.84 4.27 +0.13 to 0.30 ms 4.36 +2.3 to 2.9 ms 1.97 VPC Lattice, p50 vs baseline 1 0 Baseline (NLB) PrivateLink Lattice HTTP Lattice HTTPS p99 did not get worse up to 1000 rps on any path Conditions: 500 rps, 1 KiB, IPv6, connection reuse, 3 runs of 1 minute (baseline 1 run), ap-northeast-1 Lab and results: ContainerNetwork-Research-handson (verification report 2026-11-19) https://github.com/MASAKIOKUDA-eng/ContainerNetwork-Research-handson 14
  15. Speed 2: New Connections Without connection reuse, each request costs

    2.3 to 2.6 ms more. That is as large as the gap between the paths. TCP connect 0 1 TLS handshake 2 3 4 Request 5 6 PrivateLink 7 8 TCP connect 2.5 to 2.8 ms PrivateLink The handshake travels all the way to the task. Lattice HTTP VPC Lattice TCP connect 0.5 to 0.7 ms TCP ends near the client, but TLS adds 1.7 to 1.9 ms. Lattice HTTPS Total p50: 4.87 ms / 5.37 ms / 7.21 ms Fix connection reuse first, then choose the path. Conditions: 100 rps, new connection per request, IPv6, 1 run of 1 minute, ap-northeast-1 Lab and results: ContainerNetwork-Research-handson (verification report 2026-11-19) https://github.com/MASAKIOKUDA-eng/ContainerNetwork-Research-handson 15
  16. Speed 3: IPv6 vs IPv4 IPv6 is not the slow

    option. The gaps stayed within run to run variation. PrivateLink Lattice HTTP IPv6 p50 IPv4 p50 IPv6 p50 IPv4 p50 2.01 2.04 4.24 4.34 ms at 500 rps S3 gateway IPv6 p50 IPv4 p50 24.96 24.75 ms at 500 rps ms at 500 rps VPC Flow Logs: about 3.16 GB of test traffic over IPv6 vs about 36 MB over IPv4. Conditions: 500 rps, 1 KiB, connection reuse, 1 run of 1 minute each, ap-northeast-1 Lab and results: ContainerNetwork-Research-handson (verification report 2026-11-19) https://github.com/MASAKIOKUDA-eng/ContainerNetwork-Research-handson Icons: AWS Architecture Icons https://aws.amazon.com/architecture/icons/ 16
  17. Speed 4: Throughput and S3 For S3, start with the

    gateway endpoint. It is free, supports IPv6, and had a better p99. Throughput in MiB/s at 64 VUs (1 MiB objects) 1600 1475 1472 1460 1400 1200 1136 Throughput PrivateLink hit the client limit (12.5 Gbps). Lattice stayed near 1.1 GiB/s, and only latency doubled. Cause not isolated yet. 1000 800 600 S3 gateway vs interface p50: 24.8 ms vs 25.9 ms p99: 41 ms vs 57 ms 400 200 0 Gateway endpoints have no extra charge. PrivateLink Lattice HTTP S3 gateway S3 interface Conditions: 1 MiB objects, 64 VUs, 1 run of 1 minute, c7i.xlarge client, ap-northeast-1 Lab and results: ContainerNetwork-Research-handson (verification report 2026-11-19) https://github.com/MASAKIOKUDA-eng/ContainerNetwork-Research-handson 17
  18. Speed 5: Time to First Call Speed also means how

    fast a new team can make its first call. S3 gateway endpoint 4s Ready right away PrivateLink endpoint 6 s + 38 s after acceptance Plus a human approval Lattice VPC association 74 s No approval needed S3 interface endpoint 247 s Ready right away Pitfall: if you look up the PrivateLink DNS name before acceptance, the VPC resolver caches NXDOMAIN. In my lab, the name did not resolve for about 9.7 minutes after acceptance. Lab and results: ContainerNetwork-Research-handson (verification report 2026-11-19) https://github.com/MASAKIOKUDA-eng/ContainerNetwork-Research-handson Icons: AWS Architecture Icons https://aws.amazon.com/architecture/icons/ 18
  19. 04 Section 4 of 5 1 2 3 4 5

    Security Layers, matrix, what the provider sees
  20. Security 1: Think in Layers I tested security as three

    layers, so one mistake does not open everything. Network layer Security groups on the endpoint or on the Lattice association Connectivity layer PrivateLink acceptance, Lattice auth policy (IAM), endpoint policy Resource layer Bucket policy that only accepts aws:SourceVpce Icons: AWS Architecture Icons https://aws.amazon.com/architecture/icons/ 20
  21. Security 2: The Matrix I changed only one thing per

    column. The S3 gateway endpoint has no security group. Policies protect it. Path Trusted Untrusted SG Not accepted yet IAM required PrivateLink ✓ 200 ✗ timeout ✗ NXDOMAIN ✓ 200 Lattice HTTP (anonymous) ✓ 200 ✗ timeout ✓ 200 ✗ 403 Lattice HTTPS (SigV4) ✓ 200 ✗ timeout ✓ 200 ✓ 200 S3 gateway endpoint ✓ 200 ✓ 200 n/a ✓ 200 S3 interface endpoint ✓ 200 ✗ timeout n/a ✓ 200 Same results over IPv6 and IPv4. n/a: acceptance applies to PrivateLink only. Lab and results: ContainerNetwork-Research-handson (verification report 2026-11-19) https://github.com/MASAKIOKUDA-eng/ContainerNetwork-Research-handson 21
  22. Evidence: Where S3 Traffic Goes Gateway endpoint: public S3 addresses,

    reached through the route table. Interface endpoint: addresses inside my VPC, behind a security group. Gateway endpoint ENDPOINT S3 gateway S3 gateway S3 interface S3 interface FAMILY IPv6 IPv4 IPv6 IPv4 ANSWERS 8 AAAA 8 A 2 AAAA 2 A CONNECTED TO 2406:da61:4009::100f:3966 52.219.137.20 2406:da14:1153:4500:c60e:bec9:9805:4301 10.0.2.212 # VPC Flow Logs, consumer VPC, S3 traffic is_ipv6=1 path=7 ACCEPT total_bytes=664700145 is_ipv6=0 path=7 ACCEPT total_bytes=8750858 Public S3 addresses through the route table. No network interface, so no security group. Interface endpoint Addresses of a network interface in my VPC. The security group applies. Traffic path 7 Flow Logs record traffic through a gateway VPC endpoint as path 7. Raw data: reports/2026-11-19-ap-northeast-1/probe.csv (trusted), 31-s3-flowlogs.txt https://github.com/MASAKIOKUDA-eng/ContainerNetwork-Research-handson 22
  23. Evidence: The Lattice Access Log With IAM required, the Lattice

    access log shows who was denied and why. 403 for anonymous calls # VPC Lattice access log, latticeAuthMode=iam-required responseCode=403 authDeniedReason=Service callerPrincipal=Anonymous requests=4 responseCode=200 authDeniedReason=callerPrincipal=arn:aws:sts::<ACCOUNT_ID>: assumed-role/v6lab-consumer-LoadgenRole-xxxx/i-xxxx requests=2 Denied by the service auth policy, with the reason in the log. 200 for SigV4 calls The IAM role of the caller is recorded on every request. Before IAM required 1,340,654 anonymous requests, each logged with the source VPC. Raw data: reports/2026-11-19-ap-northeast-1/42-x3-lattice-log.txt, 30-x4-source-ip.txt https://github.com/MASAKIOKUDA-eng/ContainerNetwork-Research-handson 23
  24. Security 3: What the Provider Sees PrivateLink hides the caller.

    VPC Lattice tells the provider the source VPC and the IAM principal. The task sees PrivateLink Caller remote = NLB IPv4 address Nothing about the caller NLB Task The task sees VPC Lattice Caller Task remote = 169.254.171.x (link local) Source VPC in a request header IAM principal with SigV4 IPv6 finding: in dual stack mode, the task receives IPv4 even when the client uses IPv6. Lab and results: ContainerNetwork-Research-handson (verification report 2026-11-19) https://github.com/MASAKIOKUDA-eng/ContainerNetwork-Research-handson Icons: AWS Architecture Icons https://aws.amazon.com/architecture/icons/ 24
  25. Evidence: IPv6 In, IPv4 Out The client called over IPv6,

    but the task saw an IPv4 address. Lattice even carries an IPv4 address inside its IPv6 address. Client connected to (IPv6) Task saw 2406:da14:1153:4501:1d3c:4b65:fdf8:6122 remote=10.0.21.32 NLB IPv4 address fd00:ec2:80::a9fe:ab61 remote=169.254.171.197 Lattice link local AWS PrivateLink VPC Lattice a9fe:ab61, one byte at a time, in decimal a9 169 . fe 254 . ab 171 . 61 97 Raw data: reports/2026-11-19-ap-northeast-1/probe.csv (trusted after DNS) https://github.com/MASAKIOKUDA-eng/ContainerNetwork-Research-handson = 169.254.171.97 The same address the IPv4 lookup returned. 25
  26. Security 4: Data Perimeter Endpoint policy controls where data can

    go. Bucket policy controls where requests can come from. Endpoint policy lists the data bucket Data bucket ✓ 200 In the VPC Two policies, two directions Outside bucket (allows my endpoint) ✗ 403 A mistake in one is still caught by the other. Internet From the internet: ✗ 403 on the data bucket (bucket policy accepts aws:SourceVpce only) Lab and results: ContainerNetwork-Research-handson (verification report 2026-11-19) https://github.com/MASAKIOKUDA-eng/ContainerNetwork-Research-handson Icons: AWS Architecture Icons https://aws.amazon.com/architecture/icons/ 26
  27. 05 Section 5 of 5 1 2 3 4 5

    How to Choose Decision guide and takeaways
  28. How to Choose Choose the path by what the provider

    must know about the caller. Check the questions from the top. Q1. Calling an AWS service API? Yes Q2. Need each caller's identity, or L7 routing across many teams? Yes Q3. Publishing one TCP service to other VPCs? Yes VPC endpoints + endpoint policy Gateway endpoint first for S3 and DynamoDB Amazon VPC Lattice IAM auth policy and access logs AWS PrivateLink Lowest added latency, provider approval For every path: dual stack edge, IPv6 only workloads where possible, security group rules for both IPv4 and IPv6, and logs that show the address family. Icons: AWS Architecture Icons https://aws.amazon.com/architecture/icons/ 28
  29. Takeaways Three things I would like you to take home

    today. 1 Design IPv6 per subnet tier, and make every dependency reachable over IPv6 2 Choose the path by what the provider must know about the caller, not only by speed 3 Fix connection reuse before arguing about milliseconds Lab, code, and full report Numbers come from short runs: 1 to 3 runs of 1 minute, dual stack workloads, ap-northeast-1. Please measure with your own workload before deciding. Lab and results: ContainerNetwork-Research-handson (verification report 2026-11-19) https://github.com/MASAKIOKUDA-eng/ContainerNetwork-Research-handson 29
  30. Thank You! Try the lab, break it, and tell me

    what you find. @mob_engineer

More Decks by モブエンジニア

Other Decks in Technology