Load balancing at tuenti

Slides related to madrid-devops monthly meetings (November 2011)

1. Load balancing @Tuenti Ricardo Bartolomé, Senior Systems Engineer
2. Some numbers • +12M users. • 40 billion pageviews a month. • 40k req/s in core site at peak time (1.8 gbps). • 10k req/s in image routing layer (2gbps). • +500 frontend servers
3. Past • Linux boxes running LVS and ldirectord. • DSR strategy for load balancing. • Frontends used to have a external public IP. • Double investment in networking gear and its redundancy. • SSL balanced across all the frontends.
4. The (old) big picture client HTTP request HTTP response LVS f01 f02 fN External network Internal network External API
5. Present • New hardware. 4+1 LB instead of 10 LB (5+5) • New load balancing strategy using HAProxy layer 7 capabilities. • SSL terminated in the load balancers.
6. The big picture client HTTP request HAProxy f01 f02 fN External network Internal network HTTP response HTTP response HTTP proxy External API
7. Hardware • Intel Xeon X5677 (4 core, 8 threads @ 3.47GHz) • 8 gigabit network interfaces (Broadcon NextExtreme 5702 w/ multiqueue support) • 16 GB of memory
8. Networking • 4 links for internal and 4 for external • Connected to different stack member units • 4gbps theorical capacity limit per node. load balancer member unit 0 member unit 1 member unit 0 member unit 1
9. Networking • We tune IRQ SMP affinity for sharding IRQs across multiple cores that share the same L2 cache [1] • We do ECMP (Equal Cost Multi Path) [2] in our edge routers for sharding traffic across the load balancers. lb lb lb lb router ip route 95.131.168.x/32 x.x.x.2 ip route 95.131.168.x/32 x.x.x.1 ip route 95.131.168.x/32 x.x.x.3 ip route 95.131.168.x/32 x.x.x.4
10. HAProxy: Why? • Layer7 load balancing: Content inspection, persistence, slow start, throttling, anti-DoS features, supervision, content switching, keep-alive, etc. • Very robust and reliable. • Designed to be a load balancer. • Offers high control over HTTP delivery and status: response codes, connections per frontend, queued request, etc.
11. HAProxy: Concepts • Frontend: Section where we listen() for incoming connections. • Backend: Pool of servers. We define algorithm, configure healthy checks, etc. • Listen section: frontend+backend. Useful for TCP. • Connection != request: One connection can hold multiple requests (keep-alive). Only the first one is analyzed, logged and processed.
12. HAProxy: Health checks • Standard health check # Backend section backend www_farm mode http balance roundrobin option httpchk GET /server_health # Servers server fe01 x.x.x.1:80 check inter 2s downinter 5s rise 2 fall 3 weight 100 server fe02 x.x.x.2:80 check inter 2s downinter 5s rise 2 fall 3 weight 100
13. HAProxy: Health checks • Observe mode # Backend section backend www_farm mode http balance roundrobin option httpchk GET /server_health observe layer7 # Servers server fe01 x.x.x.1:80 check inter 2s downinter 5s rise 2 fall 3 weight 100 server fe02 x.x.x.2:80 check inter 2s downinter 5s rise 2 fall 3 weight 100
14. HAProxy: Persistence • Cookie • URI & URI parameter • Source IP • Header (i.e. Host header) • RDP cookie (Anyone using MS Terminal Server?)
15. HAProxy: Cookie persistence • Map requests between cookie value and backend server. You can issue these cookies from the code and play with them. • Ideal for deploying code by stages, or caching locally user data. • If the server becomes unreachable the traffic will be directed to other server within the same pool.
16. HAProxy: Cookie persistence backend www mode http balance roundrobin option redispatch cookie mycookie insert maxidle 120 maxlife 900 indirect preserve domain .tuenti.com server fe01 1.1.1.1:80 weight 100 cookie 1111 server fe02 1.1.1.2:80 weight 100 cookie 1112 server fe03 1.1.1.3:80 weight 100 cookie 1113
17. HAProxy: URL persistence • Specially interesting for balancing HTTP caching servers (i.e.Varnish). Without this feature the cache pool will be inefficient. • The URLs are hashed and assigned to a server in the pool (using a modulo operation). A server will serve always the same object regardless of the load balancer that attends the request. • Adding/removing/losing servers to the pool is not harmful thanks to consistent hashing.
18. HAProxy: URL persistence A B C D E F 1 2 3 4 5 6 7 8 9 map-based hashing
19. HAProxy: URL persistence A B C D E F 1 2 3 4 5 6 7 8 9 map-based hashing
20. HAProxy: URL persistence A B C D E F 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 10 10 map-based hashing
21. HAProxy: URL persistence A B C D E F 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 10 10 High miss rate. #FAIL map-based hashing
22. HAProxy: URL persistence A B C D E F 1 2 3 4 5 6 7 8 9 consistent hashing
23. HAProxy: URL persistence A B C D E F 1 2 3 4 5 6 7 8 9 consistent hashing
24. HAProxy: URL persistence A B C D E F 1 2 3 4 5 6 7 8 9 consistent hashing
25. HAProxy: URL persistence A B C D E F 1 2 3 4 5 6 7 8 9 consistent hashing 1/6 misses = ~17% miss
26. HAProxy: URL persistence Our images URLs always look like: http://img3.tuenti.net/HyUdrohQQAFnCyjMJ2ekAA We can choose the first block from the URI and use it for persistence decisions. # balance roundrobin balance uri depth 1 hash-type consistent
27. HAProxy: URL persistence Our images URLs always look like: http://img3.tuenti.net/MdlIdrAOilul8ldcRwD7AdzwAeAdB4AMtgAy We can choose the first block from the URI and use it for persistence decisions. # balance roundrobin balance uri depth 1 hash-type consistent
28. HAProxy: Content switching and ACLs • Same frontend, different backend. • Take decisions about which backend will attend the connection based on: • Layer 7 information (HTTP headers, methods, URI, version, status) • Layer4 information (source IP, destination IP, port) • Internal HAProxy information (amount of backend connections, active servers in the backend, etc) • Too much options for showing all on this presentation. [1]
29. HAProxy: Content switching and ACLs # Frontend section frontend http bind x.x.x.x:80 mode http option forwardfor except 127.0.0.1/8 header X-Forwarded-For # Farm content switching acl acl-api-uri path /api acl acl-mobile-site hdr(host) -i m.tuenti.com acl acl-cdn-service hdr(host) -i cdn.tuenti.net use_backend mobile_farm if acl-mobile-site use_backend api_farm if acl-api-uri use_backend cdn_farm if acl-cdn-service default_backend www_farm
30. HAProxy: Content switching and ACLs # Backend section backend www_farm mode http balance roundrobin # Servers server fe01 x.x.x.1:80 weight 100 server fe02 x.x.x.2:80 weight 100 backend mobile_farm mode http balance roundrobin # Servers server mfe01 x.x.x.1:80 weight 100
31. HAProxy: Content switching and ACLs # Another example using internal HAProxy information frontend http bind x.x.x.x:80 mode http option forwardfor except 127.0.0.1/8 header X-Forwarded-For # Insert 250ms delay if the session rate is over 35k req/s acl too_fast fe_sess_rate ge 35000 tcp-request inspect-delay 250ms tcp-request content accept if ! too_fast tcp-request content accept if WAIT_END
32. HAProxy: Content blocking # Another example using internal HAProxy information frontend http bind x.x.x.x:80 mode http option forwardfor except 127.0.0.1/8 header X-Forwarded-For # Block requests with negative Content-Length value acl invalid-cl hdr_val(content-length) le 0 block if invalid-cl
33. HAProxy: Slow start # Backend section backend www_farm mode http balance roundrobin option httpchk GET /server_health # Servers server fe01 x.x.x.1:80 check inter 2s downinter 5s slowstart 60s rise 2 fall 3 weight 100 server fe02 x.x.x.2:80 check inter 2s downinter 5s slowstart 60s rise 2 fall 3 weight 100
34. HAProxy: Graceful shutdown # Backend section backend www_farm mode http balance roundrobin option httpchk GET /server_health http-check disable-on-404 # Servers server fe01 x.x.x.1:80 check inter 2s downinter 5s slowstart 60s rise 2 fall 3 weight 100 server fe02 x.x.x.2:80 check inter 2s downinter 5s slowstart 60s rise 2 fall 3 weight 100
35. HAProxy: Monitoring •Traffic through different frontend interfaces. Easy to aggregate incoming/outgoing traffic. • Amount of different HTTP response codes • /proc/net/sockstat
36. HAProxy: Monitoring frontend stats1 mode http bind-process 1 bind :8081 default_backend haproxy-stats1 backend haproxy-stats1 bind-process 1 mode http stats enable stats refresh 60s stats uri / stats auth mgmt:password
37. Client-side load balancing • When user logs into the site the browser loads a javascript API. Browser talks to it. • Browser communicates with the API and this one uses EasyXDM. • Using application logic we control user request to a defined farm. • A/B testing based in any criteria. • Where are from? • How old are you?
38. Client-side load balancing ‘frontend_farm_map‘ => array( 1 => 'www1', // x% (Alava) 2 => 'www4', // y% (Albacete) 3 => 'www4', // z% (Alicante) … ) ‘users_using_staging => array( ‘level’ => ‘limited’, ‘percent’ => 10, )
39. SSL • TCP load balancing is not useful for us. • We deployed stunnel and it worked fine for a while. • Then we started to suffer contention when accepting new connections. • We are currently using stud [2] for terminating SSL in our load balancers.
40. SSL: Legal issues • You can’t use this strategy of SSL termination in your PCI compliant platform. • We transport client IP information into X-Forwarded-For headers in order to log users IPs because law enforcements. • We terminate SSL in the load balancer because balancing TCP (SSL) you can’t inform the backend about the client IP.
41. stud: The Scalable TLS Unwrapping Daemon • Supports both SSL and TLS using OpenSSL. • Uses a process-per-core model. • Asynchronous I/O using libev. • Very little overhead per connection. • Designed for long-living connections. • Supports PROXY protocol. • Recently they added inter-process communication [5] .
42. PROXY protocol • Created by HAProxy [5] author for safely transport connection information across multiple layers of NAT or TCP proxies. • Native support in stud. Patches available for stunnel4. • We use it for stud informing to HAProxy about the real IP of the client, converting this information to X-Forwarded-For header that we can read and store in our application.
43. PROXY protocol # stud --ssl -c OPENSSL_CIPHERS -b 127.0.0.1 8888 -f x.x.x.x 443 -n 2 -u stud --write-proxy certificate.pem frontend http-localhost-proxy-443 bind 127.0.0.1:8888 accept-proxy mode http reqadd X-Protocol:\ SSL reqadd X-Port:\ 443 default_backend www_farm
44. REST API • Not official feature (yet) [6] • You can easily communicate to the server via HTTP. • Awesome for orchestrating your web tier.
45. Questions?
46. Related links • [1] http://software.intel.com/en-us/articles/improved-linux-smp-scaling- user-directed-processor-affinity/ • [2] http://en.wikipedia.org/wiki/Equal-cost_multi-path_routing • [3] stud repo: https://github.com/bumptech/stud • [4] Scaling SSL: http://blog.exceliance.fr/2011/11/07/scaling-out-ssl/ • [5] PROXY protocol: http://haproxy.1wt.eu/download/1.5/doc/proxy- protocol.txt • [6] REST API patch: https://github.com/jbuchbinder/haproxy-forked • HAProxy configuration doc: http://haproxy.1wt.eu/download/1.5/doc/configuration.txt