Herald Haproxy agent for load feedback

Herald (https://github.com/helpshift/herald) is a haproxy agent we wrote to implement load feedback. This talk will explain the Haproxy load balancing problem we faced, why load feedback was required and how Herald solved the problem.

This should be interesting to folks familiar with load balancers, especially Haproxy.

Transcript

1. Herald Haproxy agent for load feedback Raghu Udiyar Production Engineering Manager @ Helpshift https://github.com/helpshift/herald
2. Outline • Problem with Haproxy load balancing • What is load feedback • How Herald implements load feedback • Herald features overview
3. Haproxy Swiss Army Knife Load Balancer
4. Haproxy distributes incoming requests equally to all servers Clients Server1 Server2 Server3 Haproxy (round-robin) Haproxy Load Balancer 30 rps 10 rps 10 rps 10 rps
5. Normal haproxy round-robin load balancing
6. Load imBalance Uneven response time causes imbalance in load distribution
7. Uneven distribution when some requests take longer to serve Clients Server1 (•̀o•́)ง Server2 (•‿•) Server3 ( •_•) Haproxy (round-robin) Load imbalance 30 rps 20 rps 6 rps 4 rps
8. Load imbalance due to uneven response time
9. Impact • Performance degradation ◦ Server unable to perform with increasing loads ◦ Response time degrades further • Server underutilization
10. How do we solve this problem ?
11. Haproxy Weights Haproxy can assign weights to each backend But these are largely static when used by themselves
12. Haproxy has static “Weights” associated with each server Clients Server1 Server2 Server3 Haproxy (round-robin) Haproxy Weights 30 rps Weight 100% Weight 20% Weight 50%
13. Enter Load Feedback • The backend server can send feedback to the load balancer • I’m at peak load send me less traffic • I’m free now, please send more!
14. Servers can send feedback by adjusting the “Weights” Clients Server1 (•̀o•́)ง Server2 (•‿•) Server3 ( •_•) Haproxy Load Feedback 30 rps 20 rps 6 rps I can do 60%! Change my load to 0%!! I can do 40% more! 4 rps
15. Feedback with Weights We can do this by dynamically adjusting the aforementioned Haproxy server weights
16. The weights re-adjust the incoming traffic load over time Clients Server1 (•̀ᴗ•́)و ̑̑ Server2 (•‿•) Server3 ツ Haproxy Load Feedback 30 rps 12 rps 10 rps Weight 0% Weight 40% 8 rps Weight 60%
17. How do we communicate to Haproxy to change weights?
18. Haproxy Agent-check The haproxy agent-check feature allows Haproxy to receive commands from a backend server
19. Haproxy connects on port 5555 every 2s for commands frontend myservice bind *:8080 option tcp-smart-accept default_backend mybackend backend mybackend balance leastconn option httpchk /health-check/ default-server weight 100 agent-port 5555 server be01 10.0.1.132:8080 check agent-check server be02 10.0.1.133:8080 check agent-check server be03 10.0.1.148:8080 check agent-check
20. Agent-check commands The server on port 5555 must reply to the TCP connection with a string that can be : • 75% • MAINT • DOWN • UP, READY
21. Herald Herald implements the agent-check protocol and responds to Haproxy on port 5555
22. Haproxy connects to 5555 for agent-check actions Clients Server1 Herald Server2 Herald Server3 Herald Haproxy agent-check Haproxy Agent Check 30 rps Tcp connect 5555 MAINT UP 75%
23. Load Feedback Haproxy + Weights + agent-check + Herald Herald sits alongside your application to send the load feedback to Haproxy based on application load metrics
24. Herald Responsibilities • Respond to Haproxy agent-check requests, and • Query application load, and calculate the haproxy agent response
25. Herald communicates with the App and Haproxy Haproxy MyApp (•‿•) Herald (⌐▪_▪) Get Current Load Agent Check on Port 5555 App requests unaffected Response : 75%
26. Application Load Metrics • The App needs to implement a interface to query current load (rps, response time, connections, etc) • E.g. ◦ HTTP : GET http://localhost/my-load ◦ FILE: /tmp/myapp_state.json
27. Example Herald Configuration name: myapp bind: 0.0.0.0 port: 5555 plugins: - default_response: noop herald_plugin_name: herald_http interval: 30 is_json: true name: myapp staleness_interval: 120 staleness_response: noop stop_timeout: 60 thresholds: - min_threshold_response: 1 pct: 9000 thresholds_metric: r['requests-per-second'] url: http://localhost:9004/health-check/
28. Herald communicates with the App and Haproxy Haproxy MyApp (•‿•) Herald (⌐▪_▪) Get Current Load Agent Check on Port 5555 App requests unaffected Response : 75%
29. Graph showing Herald Weight changes over time
30. Herald Production Readiness • Uses Gevent - single threaded, minimal resources • Async polling for metrics, with response cache ◦ Smoothens responses to Haproxy • Stale response detection and fallback if app metrics fail for any reason
31. Check every 30s, and respond with noop if stale after 120s name: myapp bind: 0.0.0.0 port: 5555 plugins: - default_response: noop herald_plugin_name: herald_http interval: 30 is_json: true name: myapp staleness_interval: 120 staleness_response: noop stop_timeout: 60 thresholds: - min_threshold_response: 1 pct: 9000 thresholds_metric: r['requests-per-second'] url: http://localhost:9004/health-check/
32. Herald Metrics Parsing Rules Simple DSL for specifying thresholds and pattern matching rules
33. Json parsing and automatic % calculation with pct keyword name: myapp bind: 0.0.0.0 port: 5555 plugins: - default_response: noop herald_plugin_name: herald_http interval: 30 is_json: true name: myapp staleness_interval: 120 staleness_response: noop stop_timeout: 60 thresholds: - min_threshold_response: 1 pct: 9000 thresholds_metric: r['requests-per-second'] url: http://localhost:9004/health-check/
34. Match regex and mark server as up or down name: myapp bind: 0.0.0.0 port: 5555 plugins: - default_response: noop herald_plugin_name: herald_http interval: 30 is_json: false name: myapp staleness_interval: 120 staleness_response: noop stop_timeout: 60 patterns: - down: '.*unhealthy.*' - up: '.*healthy.*' url: http://localhost:9004/health-check/
35. Herald Metric Plugins • Herald supports pluggable metric plugins to query application load metric • Built in plugins are : ◦ herald_http ◦ herald_file
36. Easy interface for plugins, e.g. : Herald file plugin
37. Plugin Ideas • Change weight based on system loadavg, Cpu, memory, etc • Query External Metrics Systems, Graphite, Prometheus, etc
38. Herald Load Feedback • Being used in Helpshift production since last 2 years • Consistent response times for Users • Optimal utilization of resources • Cost Savings
39. Thank You https://github.com/helpshift/herald