Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Load balancing at tuenti

Load balancing at tuenti

Slides related to madrid-devops monthly meetings (November 2011)

22b40be6f4b531a87932575526364001?s=128

Ricardo Bartolomé

November 18, 2011
Tweet

Transcript

  1. Load balancing @Tuenti Ricardo Bartolomé, Senior Systems Engineer

  2. Some numbers • +12M users. • 40 billion pageviews a

    month. • 40k req/s in core site at peak time (1.8 gbps). • 10k req/s in image routing layer (2gbps). • +500 frontend servers
  3. Past • Linux boxes running LVS and ldirectord. • DSR

    strategy for load balancing. • Frontends used to have a external public IP. • Double investment in networking gear and its redundancy. • SSL balanced across all the frontends.
  4. The (old) big picture client HTTP request HTTP response LVS

    f01 f02 fN External network Internal network External API
  5. Present • New hardware. 4+1 LB instead of 10 LB

    (5+5) • New load balancing strategy using HAProxy layer 7 capabilities. • SSL terminated in the load balancers.
  6. The big picture client HTTP request HAProxy f01 f02 fN

    External network Internal network HTTP response HTTP response HTTP proxy External API
  7. Hardware • Intel Xeon X5677 (4 core, 8 threads @

    3.47GHz) • 8 gigabit network interfaces (Broadcon NextExtreme 5702 w/ multiqueue support) • 16 GB of memory
  8. Networking • 4 links for internal and 4 for external

    • Connected to different stack member units • 4gbps theorical capacity limit per node. load balancer member unit 0 member unit 1 member unit 0 member unit 1
  9. Networking • We tune IRQ SMP affinity for sharding IRQs

    across multiple cores that share the same L2 cache [1] • We do ECMP (Equal Cost Multi Path) [2] in our edge routers for sharding traffic across the load balancers. lb lb lb lb router ip route 95.131.168.x/32 x.x.x.2 ip route 95.131.168.x/32 x.x.x.1 ip route 95.131.168.x/32 x.x.x.3 ip route 95.131.168.x/32 x.x.x.4
  10. HAProxy: Why? • Layer7 load balancing: Content inspection, persistence, slow

    start, throttling, anti-DoS features, supervision, content switching, keep-alive, etc. • Very robust and reliable. • Designed to be a load balancer. • Offers high control over HTTP delivery and status: response codes, connections per frontend, queued request, etc.
  11. HAProxy: Concepts • Frontend: Section where we listen() for incoming

    connections. • Backend: Pool of servers. We define algorithm, configure healthy checks, etc. • Listen section: frontend+backend. Useful for TCP. • Connection != request: One connection can hold multiple requests (keep-alive). Only the first one is analyzed, logged and processed.
  12. HAProxy: Health checks • Standard health check # Backend section

    backend www_farm mode http balance roundrobin option httpchk GET /server_health # Servers server fe01 x.x.x.1:80 check inter 2s downinter 5s rise 2 fall 3 weight 100 server fe02 x.x.x.2:80 check inter 2s downinter 5s rise 2 fall 3 weight 100
  13. HAProxy: Health checks • Observe mode # Backend section backend

    www_farm mode http balance roundrobin option httpchk GET /server_health observe layer7 # Servers server fe01 x.x.x.1:80 check inter 2s downinter 5s rise 2 fall 3 weight 100 server fe02 x.x.x.2:80 check inter 2s downinter 5s rise 2 fall 3 weight 100
  14. HAProxy: Persistence • Cookie • URI & URI parameter •

    Source IP • Header (i.e. Host header) • RDP cookie (Anyone using MS Terminal Server?)
  15. HAProxy: Cookie persistence • Map requests between cookie value and

    backend server. You can issue these cookies from the code and play with them. • Ideal for deploying code by stages, or caching locally user data. • If the server becomes unreachable the traffic will be directed to other server within the same pool.
  16. HAProxy: Cookie persistence backend www mode http balance roundrobin option

    redispatch cookie mycookie insert maxidle 120 maxlife 900 indirect preserve domain .tuenti.com server fe01 1.1.1.1:80 weight 100 cookie 1111 server fe02 1.1.1.2:80 weight 100 cookie 1112 server fe03 1.1.1.3:80 weight 100 cookie 1113
  17. HAProxy: URL persistence • Specially interesting for balancing HTTP caching

    servers (i.e.Varnish). Without this feature the cache pool will be inefficient. • The URLs are hashed and assigned to a server in the pool (using a modulo operation). A server will serve always the same object regardless of the load balancer that attends the request. • Adding/removing/losing servers to the pool is not harmful thanks to consistent hashing.
  18. HAProxy: URL persistence A B C D E F 1

    2 3 4 5 6 7 8 9 map-based hashing
  19. HAProxy: URL persistence A B C D E F 1

    2 3 4 5 6 7 8 9 map-based hashing
  20. HAProxy: URL persistence A B C D E F 1

    2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 10 10 map-based hashing
  21. HAProxy: URL persistence A B C D E F 1

    2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 10 10 High miss rate. #FAIL map-based hashing
  22. HAProxy: URL persistence A B C D E F 1

    2 3 4 5 6 7 8 9 consistent hashing
  23. HAProxy: URL persistence A B C D E F 1

    2 3 4 5 6 7 8 9 consistent hashing
  24. HAProxy: URL persistence A B C D E F 1

    2 3 4 5 6 7 8 9 consistent hashing
  25. HAProxy: URL persistence A B C D E F 1

    2 3 4 5 6 7 8 9 consistent hashing 1/6 misses = ~17% miss
  26. HAProxy: URL persistence Our images URLs always look like: http://img3.tuenti.net/HyUdrohQQAFnCyjMJ2ekAA

    We can choose the first block from the URI and use it for persistence decisions. # balance roundrobin balance uri depth 1 hash-type consistent
  27. HAProxy: URL persistence Our images URLs always look like: http://img3.tuenti.net/MdlIdrAOilul8ldcRwD7AdzwAeAdB4AMtgAy

    We can choose the first block from the URI and use it for persistence decisions. # balance roundrobin balance uri depth 1 hash-type consistent
  28. HAProxy: Content switching and ACLs • Same frontend, different backend.

    • Take decisions about which backend will attend the connection based on: • Layer 7 information (HTTP headers, methods, URI, version, status) • Layer4 information (source IP, destination IP, port) • Internal HAProxy information (amount of backend connections, active servers in the backend, etc) • Too much options for showing all on this presentation. [1]
  29. HAProxy: Content switching and ACLs # Frontend section frontend http

    bind x.x.x.x:80 mode http option forwardfor except 127.0.0.1/8 header X-Forwarded-For # Farm content switching acl acl-api-uri path /api acl acl-mobile-site hdr(host) -i m.tuenti.com acl acl-cdn-service hdr(host) -i cdn.tuenti.net use_backend mobile_farm if acl-mobile-site use_backend api_farm if acl-api-uri use_backend cdn_farm if acl-cdn-service default_backend www_farm
  30. HAProxy: Content switching and ACLs # Backend section backend www_farm

    mode http balance roundrobin # Servers server fe01 x.x.x.1:80 weight 100 server fe02 x.x.x.2:80 weight 100 backend mobile_farm mode http balance roundrobin # Servers server mfe01 x.x.x.1:80 weight 100
  31. HAProxy: Content switching and ACLs # Another example using internal

    HAProxy information frontend http bind x.x.x.x:80 mode http option forwardfor except 127.0.0.1/8 header X-Forwarded-For # Insert 250ms delay if the session rate is over 35k req/s acl too_fast fe_sess_rate ge 35000 tcp-request inspect-delay 250ms tcp-request content accept if ! too_fast tcp-request content accept if WAIT_END
  32. HAProxy: Content blocking # Another example using internal HAProxy information

    frontend http bind x.x.x.x:80 mode http option forwardfor except 127.0.0.1/8 header X-Forwarded-For # Block requests with negative Content-Length value acl invalid-cl hdr_val(content-length) le 0 block if invalid-cl
  33. HAProxy: Slow start # Backend section backend www_farm mode http

    balance roundrobin option httpchk GET /server_health # Servers server fe01 x.x.x.1:80 check inter 2s downinter 5s slowstart 60s rise 2 fall 3 weight 100 server fe02 x.x.x.2:80 check inter 2s downinter 5s slowstart 60s rise 2 fall 3 weight 100
  34. HAProxy: Graceful shutdown # Backend section backend www_farm mode http

    balance roundrobin option httpchk GET /server_health http-check disable-on-404 # Servers server fe01 x.x.x.1:80 check inter 2s downinter 5s slowstart 60s rise 2 fall 3 weight 100 server fe02 x.x.x.2:80 check inter 2s downinter 5s slowstart 60s rise 2 fall 3 weight 100
  35. HAProxy: Monitoring •Traffic through different frontend interfaces. Easy to aggregate

    incoming/outgoing traffic. • Amount of different HTTP response codes • /proc/net/sockstat
  36. HAProxy: Monitoring frontend stats1 mode http bind-process 1 bind :8081

    default_backend haproxy-stats1 backend haproxy-stats1 bind-process 1 mode http stats enable stats refresh 60s stats uri / stats auth mgmt:password
  37. Client-side load balancing • When user logs into the site

    the browser loads a javascript API. Browser talks to it. • Browser communicates with the API and this one uses EasyXDM. • Using application logic we control user request to a defined farm. • A/B testing based in any criteria. • Where are from? • How old are you?
  38. Client-side load balancing ‘frontend_farm_map‘ => array( 1 => 'www1', //

    x% (Alava) 2 => 'www4', // y% (Albacete) 3 => 'www4', // z% (Alicante) … ) ‘users_using_staging => array( ‘level’ => ‘limited’, ‘percent’ => 10, )
  39. SSL • TCP load balancing is not useful for us.

    • We deployed stunnel and it worked fine for a while. • Then we started to suffer contention when accepting new connections. • We are currently using stud [2] for terminating SSL in our load balancers.
  40. SSL: Legal issues • You can’t use this strategy of

    SSL termination in your PCI compliant platform. • We transport client IP information into X-Forwarded-For headers in order to log users IPs because law enforcements. • We terminate SSL in the load balancer because balancing TCP (SSL) you can’t inform the backend about the client IP.
  41. stud: The Scalable TLS Unwrapping Daemon • Supports both SSL

    and TLS using OpenSSL. • Uses a process-per-core model. • Asynchronous I/O using libev. • Very little overhead per connection. • Designed for long-living connections. • Supports PROXY protocol. • Recently they added inter-process communication [5] .
  42. PROXY protocol • Created by HAProxy [5] author for safely

    transport connection information across multiple layers of NAT or TCP proxies. • Native support in stud. Patches available for stunnel4. • We use it for stud informing to HAProxy about the real IP of the client, converting this information to X-Forwarded-For header that we can read and store in our application.
  43. PROXY protocol # stud --ssl -c OPENSSL_CIPHERS -b 127.0.0.1 8888

    -f x.x.x.x 443 -n 2 -u stud --write-proxy certificate.pem frontend http-localhost-proxy-443 bind 127.0.0.1:8888 accept-proxy mode http reqadd X-Protocol:\ SSL reqadd X-Port:\ 443 default_backend www_farm
  44. REST API • Not official feature (yet) [6] • You

    can easily communicate to the server via HTTP. • Awesome for orchestrating your web tier.
  45. Questions?

  46. Related links • [1] http://software.intel.com/en-us/articles/improved-linux-smp-scaling- user-directed-processor-affinity/ • [2] http://en.wikipedia.org/wiki/Equal-cost_multi-path_routing •

    [3] stud repo: https://github.com/bumptech/stud • [4] Scaling SSL: http://blog.exceliance.fr/2011/11/07/scaling-out-ssl/ • [5] PROXY protocol: http://haproxy.1wt.eu/download/1.5/doc/proxy- protocol.txt • [6] REST API patch: https://github.com/jbuchbinder/haproxy-forked • HAProxy configuration doc: http://haproxy.1wt.eu/download/1.5/doc/configuration.txt