Networking in a Containerized Data Center - the Gotchas

This was presented at the Microservices for the Enterprise Meetup in March 2016. I present 10 "gotchas" to applying virtual networking concepts to containers, all of which were traps people were falling into at that time.

Transcript

1. Project Calico is sponsored by @projectcalico Sponsored by Networking in a Containerized Data Center: the Gotchas! MICROSERVICES FOR ENTERPRISES MEETUP Andy Randall | @andrew_randall Palo Alto, March 31, 2016
2. Project Calico is sponsored by @projectcalico (n) North American. “an instance of publicly tricking someone or exposing them to ridicule, especially by means of an elaborate deception.”
3. Project Calico is sponsored by @projectcalico
4. Project Calico is sponsored by @projectcalico Run anywhere Simple Lightweight Standard Speed Cloud Efficient
5. Project Calico is sponsored by @projectcalico
6. Project Calico is sponsored by @projectcalico The original “container approach” to networking § All containers on a machine share the same IP address § Gotcha #1: WWW1 WWW2 80 80 Proxy 8080 8081 Still most container deployments use this method!
7. Project Calico is sponsored by @projectcalico World is moving to “IP per container” Container Network Interface (CNI) Container Network Model (libnetwork, 0.19) net-modules (0.26) (future: CNI?)
8. Project Calico is sponsored by @projectcalico We’ve solved “IP per VM” before… VM 1 VM 2 VM 3 Virtual Switch
9. Project Calico is sponsored by @projectcalico We’ve solved “IP per VM” before… VM 1 VM 2 VM 3 Virtual Switch VM 1 VM 2 VM 3 Virtual Switch
10. Project Calico is sponsored by @projectcalico Consequences for containers (gotcha #2): Scale Hundreds of servers, low churn Millions of containers, high churn
11. Project Calico is sponsored by @projectcalico pHost 1 Virtual Switch / encapsulation vNIC pNIC vNIC VM1 Consequences for containers (gotcha #3): Layering Packets are double encap’d! Container A Container B Container C Virtual Switch / encapsulation veth0 veth1 veth2 pHost 2 Virtual Switch / encapsulation VM2 Container D Container E Container F Virtual Switch / encapsulation pNIC vNIC vNIC veth0 veth1 veth2 Physical Switch
12. Project Calico is sponsored by @projectcalico Consequences for containers (gotcha #4): walled gardens Legacy App pHost 1 Virtual Switch / encapsulation vNIC pNIC vNIC VM1 Container A Container B Container C Virtual Switch / encapsulation veth0 veth1 veth2 Physical Switch
13. Project Calico is sponsored by @projectcalico “Any intelligent fool can make things bigger, more complex… It takes a touch of genius – and a lot of courage – to move in the opposite direction.”
14. Project Calico is sponsored by @projectcalico A Saner Approach: just route IP from the container pHost 1 Virtual underlay vNIC pNIC vNIC VM1 Container A Container B Container C Linux kernel routing (no encapsulation) veth0 veth1 veth2 pHost 2 Virtual Underlay VM2 Container D Container E Container F Linux kernel routing (no encapsulation) pNIC vNIC vNIC veth0 veth1 veth2 Physical Underlay
15. Project Calico is sponsored by @projectcalico Variant: 1 vm per host, no virtual underlay, straight-up IP pHost 1 pNIC vNIC VM1 Container A Container B Container C Linux kernel routing (no encapsulation) veth0 veth1 veth2 pHost 2 VM2 Container D Container E Container F Linux kernel routing (no encapsulation) pNIC vNIC veth0 veth1 veth2 Physical Underlay
16. Project Calico is sponsored by @projectcalico Results: bare metal performance from virtual networks 0 1 2 3 4 5 6 7 8 9 10 Bare metal Calico OVS+VXLAN Throughput Gbps 0 20 40 60 80 100 120 Bare metal Calico OVS+VXLAN CPU % per Gbps Source: https://www.projectcalico.org/calico-dataplane-performance/
17. Project Calico is sponsored by @projectcalico § Some container frameworks still assume port mapping § E.g. Marathon load balancer service (but being fixed…) § Some PaaS’s not yet supporting IP per container § But several moving to build on Kubernetes, and will likely pick it up Gotcha #5: IP per container not yet universally supported
18. Project Calico is sponsored by @projectcalico § You can easily get your configuration wrong and get sub- optimal performance, e.g. § select wrong Flannel back-end for your fabric § turn off AWS src-dest IP checks § get MTU size wrong for the underlay… Gotcha #6: running on public cloud
19. Project Calico is sponsored by @projectcalico Consequences of MTU size… 0 50 100 150 200 250 300 t2.micro m4.xlarge qperf bandwidth Bare Metal Calico
20. Project Calico is sponsored by @projectcalico Consequences of MTU size… 0 50 100 150 200 250 300 t2.micro m4.xlarge qperf bandwidth Bare Metal Calico (MTU=1440) Calico (MTU=8980)
21. Project Calico is sponsored by @projectcalico § Suppose we assign a /24 per Kubernetes node (=> 254 pods) § Run 10 VMs per server, each with a Kubernetes node § 40 servers per rack § 20 racks per data center § 4 data centers § => now need a /15 for the rack, a /10 space for the data center, and the entire 10/8 rfc1918 range to cover 4 data centers. § … and hope your business doesn’t expand to need a 5th data center! Gotcha #7: IP addresses aren’t infinite
22. Project Calico is sponsored by @projectcalico § Kubernetes § CNI fairly stable § Fine-grained policy being added – will move from alpha (annotation— based) to first-class citizen API § Mesos – multiple ways to network your container § Net-modules – but only supports Mesos containerizer § Docker networking – but then not fully integrated e.g. into MesosDNS § CNI – possible future, but not here today § Roll-your-own orchestrator-network co-ordination – the approach some of our users have taken § Docker § Swarm / Docker Datacenter still early; libnetwork evolution? policy? Gotcha #8: orchestration platforms support still evolving
23. Project Calico is sponsored by @projectcalico § Docker libnetwork provides limited functionality / visibility to plug-ins § E.g. network name you specify as a user is NOT passed to the underlying SDN § Consequences: § Diagnostics hard to correlate § Hard to enable ”side loaded” commands referring to networks created on Docker command line (e.g. Calico advanced policy) § Hard to network between Docker virtual network domain and non- containerized workloads Gotcha #9: Docker libnetwork is “special”
24. Project Calico is sponsored by @projectcalico § “Can you write a function that tells me when all nodes have caught up to the global state?” § Sure… Gotcha #10: at cloud scale, nothing ever converges function is_converged() return false
25. Project Calico is sponsored by @projectcalico