Some numbers • +12M users. • 40 billion pageviews a month. • 40k req/s in core site at peak time (1.8 gbps). • 10k req/s in image routing layer (2gbps). • +500 frontend servers
Past • Linux boxes running LVS and ldirectord. • DSR strategy for load balancing. • Frontends used to have a external public IP. • Double investment in networking gear and its redundancy. • SSL balanced across all the frontends.
Present • New hardware. 4+1 LB instead of 10 LB (5+5) • New load balancing strategy using HAProxy layer 7 capabilities. • SSL terminated in the load balancers.
Networking • 4 links for internal and 4 for external • Connected to different stack member units • 4gbps theorical capacity limit per node. load balancer member unit 0 member unit 1 member unit 0 member unit 1
Networking • We tune IRQ SMP affinity for sharding IRQs across multiple cores that share the same L2 cache [1] • We do ECMP (Equal Cost Multi Path) [2] in our edge routers for sharding traffic across the load balancers. lb lb lb lb router ip route 95.131.168.x/32 x.x.x.2 ip route 95.131.168.x/32 x.x.x.1 ip route 95.131.168.x/32 x.x.x.3 ip route 95.131.168.x/32 x.x.x.4
HAProxy: Why? • Layer7 load balancing: Content inspection, persistence, slow start, throttling, anti-DoS features, supervision, content switching, keep-alive, etc. • Very robust and reliable. • Designed to be a load balancer. • Offers high control over HTTP delivery and status: response codes, connections per frontend, queued request, etc.
HAProxy: Concepts • Frontend: Section where we listen() for incoming connections. • Backend: Pool of servers. We define algorithm, configure healthy checks, etc. • Listen section: frontend+backend. Useful for TCP. • Connection != request: One connection can hold multiple requests (keep-alive). Only the first one is analyzed, logged and processed.
HAProxy: Health checks • Standard health check # Backend section backend www_farm mode http balance roundrobin option httpchk GET /server_health # Servers server fe01 x.x.x.1:80 check inter 2s downinter 5s rise 2 fall 3 weight 100 server fe02 x.x.x.2:80 check inter 2s downinter 5s rise 2 fall 3 weight 100
HAProxy: Cookie persistence • Map requests between cookie value and backend server. You can issue these cookies from the code and play with them. • Ideal for deploying code by stages, or caching locally user data. • If the server becomes unreachable the traffic will be directed to other server within the same pool.
HAProxy: URL persistence • Specially interesting for balancing HTTP caching servers (i.e.Varnish). Without this feature the cache pool will be inefficient. • The URLs are hashed and assigned to a server in the pool (using a modulo operation). A server will serve always the same object regardless of the load balancer that attends the request. • Adding/removing/losing servers to the pool is not harmful thanks to consistent hashing.
HAProxy: URL persistence Our images URLs always look like: http://img3.tuenti.net/HyUdrohQQAFnCyjMJ2ekAA We can choose the first block from the URI and use it for persistence decisions. # balance roundrobin balance uri depth 1 hash-type consistent
HAProxy: URL persistence Our images URLs always look like: http://img3.tuenti.net/MdlIdrAOilul8ldcRwD7AdzwAeAdB4AMtgAy We can choose the first block from the URI and use it for persistence decisions. # balance roundrobin balance uri depth 1 hash-type consistent
HAProxy: Content switching and ACLs • Same frontend, different backend. • Take decisions about which backend will attend the connection based on: • Layer 7 information (HTTP headers, methods, URI, version, status) • Layer4 information (source IP, destination IP, port) • Internal HAProxy information (amount of backend connections, active servers in the backend, etc) • Too much options for showing all on this presentation. [1]
HAProxy: Content switching and ACLs # Another example using internal HAProxy information frontend http bind x.x.x.x:80 mode http option forwardfor except 127.0.0.1/8 header X-Forwarded-For # Insert 250ms delay if the session rate is over 35k req/s acl too_fast fe_sess_rate ge 35000 tcp-request inspect-delay 250ms tcp-request content accept if ! too_fast tcp-request content accept if WAIT_END
HAProxy: Content blocking # Another example using internal HAProxy information frontend http bind x.x.x.x:80 mode http option forwardfor except 127.0.0.1/8 header X-Forwarded-For # Block requests with negative Content-Length value acl invalid-cl hdr_val(content-length) le 0 block if invalid-cl
HAProxy: Monitoring •Traffic through different frontend interfaces. Easy to aggregate incoming/outgoing traffic. • Amount of different HTTP response codes • /proc/net/sockstat
Client-side load balancing • When user logs into the site the browser loads a javascript API. Browser talks to it. • Browser communicates with the API and this one uses EasyXDM. • Using application logic we control user request to a defined farm. • A/B testing based in any criteria. • Where are from? • How old are you?
SSL • TCP load balancing is not useful for us. • We deployed stunnel and it worked fine for a while. • Then we started to suffer contention when accepting new connections. • We are currently using stud [2] for terminating SSL in our load balancers.
SSL: Legal issues • You can’t use this strategy of SSL termination in your PCI compliant platform. • We transport client IP information into X-Forwarded-For headers in order to log users IPs because law enforcements. • We terminate SSL in the load balancer because balancing TCP (SSL) you can’t inform the backend about the client IP.
stud: The Scalable TLS Unwrapping Daemon • Supports both SSL and TLS using OpenSSL. • Uses a process-per-core model. • Asynchronous I/O using libev. • Very little overhead per connection. • Designed for long-living connections. • Supports PROXY protocol. • Recently they added inter-process communication [5] .
PROXY protocol • Created by HAProxy [5] author for safely transport connection information across multiple layers of NAT or TCP proxies. • Native support in stud. Patches available for stunnel4. • We use it for stud informing to HAProxy about the real IP of the client, converting this information to X-Forwarded-For header that we can read and store in our application.
loglass2019
onlinesalon
hhiroshell
s2terminal
minorun365
knsg16
okamototakuyasr2
hiashisan
niftycorp
coconala_engineer
sat
lwug
bkeepers
bryan
tanoku
maggiecrowley
cherdarchuk
frogandcode
rasmusluckow
danielanewman
revolveconf
jeffersonlam
thoeni
afnizarnur