No virtualization, no containers, raw metal • Thousands of IP addresses/subnets (anycast) • Multiple applications ◦ HTTP (HTTP, TLS 1.3, HTTP2, QUIC) ◦ DNS (Auth, Resolver) ◦ Other
is hard • More concurrency primitives (cmpxchg16b?) • "Traffic policing in eBPF: applying token bucket algorithm" ◦ http://vger.kernel.org/lpc-bpf.html#session-9
TPROXY https://blog.cloudflare.com/how-we-built-spectrum/ • TPROXY UDP Socket dispatch - AnyIP Single IP Many subnets Single port bind() SO_BINDTOPREFIX Many ports TPROXY TPROXY
QoS: saturation north (public) anycast eyeball requests 1500 inbound: attack outbound: traffic spike south (public) origin origin pulls 1500 - east-west - L4LB inbound requests 1544 - east-west - cache cache traffic jumbo hot assets • MTU is hard • LRO can get disabled on large MSS • tc qdisc work better on physical device