Who Needs Clouds?: HA in Your Datacenter

Tony Pitluga and Paul Hinze at Surge Conference 2012

Building highly available services can be a challenge. At Braintree, we value simplicity, open source and cheap commodity hardware. While many companies attempt to achieve high-availability through expensive, proprietary components like SANs and hardware load balancers, these sorts of devices are fraught with their own peril. We'll describe how we built a robust infrastructure using off-the-shelf hardware, Linux-based tools, and a few custom applications to tie everything together. By open-sourcing our work and telling our story, we'll hope to share with the community what we believe is a clean, elegant strategy for structuring highly available services of all shapes and sizes.

Transcript

2. Tony Pitluga Paul Hinze @pitluga @phinze
4. Our world
5. Lots of people trust us
6. > 1 million API calls per day
7. > 1 million API calls per day $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $$ $ $ $ $ $ important
8. Downtime really sucks $ $ $ $ $ $ $ $ $ $
9. we have interesting jobs in other words
10. What we value
11. simple easy Open closed logic not not not magic
12. What we choose
13. not not not linux-ha
14. Today
15. our beloved Load balancers
16. scalability availability +
17. linux-ha
18. Thou Shalt Do One Thing Well
19. Don’t Panic!
20. lots of simple pieces
21. Whole > Sum ( parts )
22. 204.109.13.100 app01 204.109.13.7 load01 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} POST /transactions 10.0.0.1 201 Created app02 app{N} balancing act 1 2 3 4
23. 204.109.13.100 app01 204.109.13.7 load01 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} POST /transactions 10.0.0.1 201 Created app02 app{N} (1) ip virtual server
24. 204.109.13.100 app01 204.109.13.7 load01 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} POST /transactions 10.0.0.1 201 Created app02 app{N} (2) litmus_paper litmus_paper 100
25. 204.109.13.100 app01 204.109.13.7 load01 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} POST /transactions 10.0.0.1 201 Created app02 app{N} (3) big_brother litmus_paper 201 Created big_brother 100
26. 204.109.13.100 app01 204.109.13.7 load01 POST /transactions 10.0.0.1 201 Created app02 app{N} (4) pacemaker litmus_paper 100 load02 load03
27. here goes!
28. load01 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} IP Virtual server (ipvs)
29. on your mark iptables --table mangle \ --append PREROUTING \ --destination 205.109.13.7/32 \ --interface eth70 \ --protocol tcp \ --match tcp --dport 443 \ --jump MARK --set-mark 0x07
30. iptables --table mangle \ --append PREROUTING \ --destination 205.109.13.7/32 \ --interface eth70 \ --protocol tcp \ --match tcp --dport 443 \ --jump MARK --set-mark 0x07 on your mark
31. get set ipvsadm --add-service --fwmark-service 7 --scheduler wlc
32. get set ipvsadm --add-service --fwmark-service 7 --scheduler wlc
33. and they’re off! ipvsadm --add-server --fwmark-service 7 --real-server 10.0.0.1 --ipip --weight 100
34. and they’re off! ipvsadm --add-server --fwmark-service 7 --real-server 10.0.0.1 --ipip --weight 100
35. and they’re off! ipvsadm --add-server --fwmark-service 7 --real-server 10.0.0.2 --ipip --weight 100
36. 204.109.13.100 app01 204.109.13.7 load01 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} POST /transactions 10.0.0.1 201 Created app02 app{N} now you’re balancing with FOSS!
37. ok, so what?
38. weight weight don’t tell me ipvsadm --add-server --fwmark-service 7 --real-server 10.0.0.1 --ipip --weight 100
39. weight weight don’t tell me ipvsadm --add-service --fwmark-service 7 --scheduler wlc
40. The Power is yours!
41. weight == “Health” of node
42. how do we get health?
43. litmus_paper braintree/litmus_paper
44. app01 Determines health of single node litmus_paper
45. health == Dependencies + Metrics
46. a litmus_paper check service "webapp" do |s| s.depends Dependency::HTTP, "http://localhost:3000/heartbeat" s.measure_health Metric::CPULoad, :weight => 50 s.measure_health Metric::AvailableMemory, :weight => 50 end
47. a litmus_paper check service "webapp" do |s| s.depends Dependency::HTTP, "http://localhost:3000/heartbeat" s.measure_health Metric::CPULoad, :weight => 50 s.measure_health Metric::AvailableMemory, :weight => 50 end services i *require* to be useful
48. a litmus_paper check service "webapp" do |s| s.depends Dependency::HTTP, "http://localhost:3000/heartbeat" s.measure_health Metric::CPULoad, :weight => 50 s.measure_health Metric::AvailableMemory, :weight => 50 end how healthy am i generally?
49. easy to extend require 'postfix_litmus' service "mailman" do |s| s.depends PostfixLitmus::Dependency, "localhost:543" s.measure_health PostfixLitmus::MailQSize, :max_size => 1000, :weight => 100 end
50. easy httpeasy $ curl localhost Litmus Paper 0.6.0 Services monitored: * webapp (87.5)
51. easy httpeasy $ curl localhost/webapp/status Health: 87.5 Dependency::HTTP(http://localhost:3000/heartbeat): OK Metric::CPULoad(50): 49.5 Metric::AvailableMemory(50): 38
52. Maintenance w/ litmusctl $ litmusctl force down webapp Reason? apt upgrades $ curl localhost/webapp/status Health: 0 apt upgrades
53. 204.109.13.100 app01 204.109.13.7 load01 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} POST /transactions 10.0.0.1 201 Created app02 app{N} litmus keeping tabs litmus_paper litmus_paper litmus_paper 100 92 55
54. litmus will answer but who will ask?
55. big_brother braintree/big_brother
56. load01 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} monitors litmus updates ipvs big_brother
57. it’s simple, really while true health = `curl node/app/status \ | grep Health \ | awk ‘{print $2}’` `ipvsadm --edit-server --weight #{health}` sleep 2 end
58. config, brother webapp: checkInterval: 2000 scheduler: wlc fwmark: 7 nodes: - address: 10.0.0.1 port: 80 path: /webapp/status - address: 10.0.0.2 port: 80 path: /webapp/status
59. 204.109.13.100 app01 204.109.13.7 load01 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} POST /transactions 10.0.0.1 201 Created app02 app{N} big_brother is watching litmus_paper litmus_paper litmus_paper 100 92 55 big_brother 92
60. big_brother what can Do FOR YOU?
61. Yale Collection of Western Americana, Beinecke Rare Book and Manuscript Library
62. 15 load01 balancing by health app01 app02 10 app03 25 30% 20% 50%
63. 15 load01 new kid on the block app01 app02 10 app03 25 30% 20% 50% app04 100 yeah!
64. 15 load01 accosted like a boy band app01 app02 10 app03 25 10% 6% 16% app04 100 66%
65. load01 accosted like a boy band app04 66% 5 ouch! 15 app01 app02 10 app03 25 10% 6% 16%
66. config, brother webapp: checkInterval: 2000 scheduler: wlc fwmark: 7 ramp_up_time: 120 nodes: - address: 10.0.0.1 port: 80 path: /webapp/status - address: 10.0.0.2 port: 80 path: /webapp/status try this
67. 15 load01 Humble kid on the block app01 app02 10 app03 25 30% 20% 50% app04 5 easy now
68. 15 load01 ease off the pressure 10 25 27% 18% 45% app04 5 9% app01 app02 app03
69. 15 load01 ease off the pressure 10 25 25% 16% 41% app04 5 16% app01 app02 app03
70. 15 load01 even out 10 25 app04 5 app01 app02 app03 23% 24% 27% 26%
71. 78 load01 we all get healthy together 82 95 app04 93 app01 app02 app03 23% 24% 27% 26%
72. 204.109.13.100 app01 204.109.13.7 load01 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} POST /transactions 10.0.0.1 201 Created app02 app{N} big_brother is watching litmus_paper litmus_paper litmus_paper 100 92 55 big_brother 92 201 Created
73. who load balances the load balancers ?
77. load01 load02 load03 manages resources across a cluster
78. resource == open cluster framework (ocf)
79. ocf scripts /usr/lib/ocf/resource.d/braintree/ocf_big_brother <monitor|start|stop>
80. ocf monitor /usr/lib/ocf/resource.d/braintree/ocf_big_brother <monitor|start|stop> $ curl localhost:9292/cluster/app Running: true
81. ocf monitor /usr/lib/ocf/resource.d/braintree/ocf_big_brother <monitor|start|stop> $ curl -XPUT localhost:9292/cluster/app OK
82. ocf monitor /usr/lib/ocf/resource.d/braintree/ocf_big_brother <monitor|start|stop> $ curl -XDELETE localhost:9292/cluster/app OK
83. resource == virtual IP(s) + IPVS cluster
84. IPaddr2
85. XmHELL <group description="Braintree Webapp" id="webapp_group"> <primitive class="ocf" id="ipvs_webapp" provider="braintree" type="ocf_big_brother"> <instance_attributes id="ipvs_webapp_ia"> <nvpair id="ipvs_monitor_webapp_cluster" name="cluster" value="webapp"/> </instance_attributes> <operations> <op id="ipvs_webapp_op_start" interval="0" name="start" timeout="20s"/> <op id="ipvs_webapp_op_stop" interval="0" name="stop" timeout="40s"/> <op id="ipvs_webapp_op_monitor" interval="61s" name="monitor" timeout="30s"/> </operations> </primitive> <primitive class="ocf" id="ip_204.109.13.7" provider="heartbeat" type="IPaddr2"> <instance_attributes id="ip_204.109.13.7_ia"> <nvpair id="ip_204.109.13.7_ip" name="ip" value="204.109.13.7"/> <nvpair id="ip_204.109.13.7_nm" name="cidr_netmask" value="32"/> <nvpair id="ip_204.109.13.7_nic" name="nic" value="eth70"/> </instance_attributes> <operations> <op id="ip_204.109.13.7_op_start" interval="0" name="start" timeout="30s"/> <op id="ip_204.109.13.7_op_stop" interval="0" name="stop" timeout="90s"/> <op id="ip_204.109.13.7_op_monitor" interval="61s" name="monitor" timeout="30s"/> </operations> </primitive> </group>
86. groups flock together <group description="Braintree Webapp" id="webapp_group"> <primitive class="ocf" id="ipvs_webapp" provider="braintree" type="ocf_big_brother"> <instance_attributes id="ipvs_webapp_ia"> <nvpair id="ipvs_monitor_webapp_cluster" name="cluster" value="webapp"/> </instance_attributes> <operations> <op id="ipvs_webapp_op_start" interval="0" name="start" timeout="20s"/> <op id="ipvs_webapp_op_stop" interval="0" name="stop" timeout="40s"/> <op id="ipvs_webapp_op_monitor" interval="61s" name="monitor" timeout="30s"/> </operations> </primitive> <primitive class="ocf" id="ip_204.109.13.7" provider="heartbeat" type="IPaddr2"> <instance_attributes id="ip_204.109.13.7_ia"> <nvpair id="ip_204.109.13.7_ip" name="ip" value="204.109.13.7"/> <nvpair id="ip_204.109.13.7_nm" name="cidr_netmask" value="32"/> <nvpair id="ip_204.109.13.7_nic" name="nic" value="eth70"/> </instance_attributes> <operations> <op id="ip_204.109.13.7_op_start" interval="0" name="start" timeout="30s"/> <op id="ip_204.109.13.7_op_stop" interval="0" name="stop" timeout="90s"/> <op id="ip_204.109.13.7_op_monitor" interval="61s" name="monitor" timeout="30s"/> </operations> </primitive> </group>
87. the ipvs resource <group description="Braintree Webapp" id="webapp_group"> <primitive class="ocf" id="ipvs_webapp" provider="braintree" type="ocf_big_brother"> <instance_attributes id="ipvs_webapp_ia"> <nvpair id="ipvs_monitor_webapp_cluster" name="cluster" value="webapp"/> </instance_attributes> <operations> <op id="ipvs_webapp_op_start" interval="0" name="start" timeout="20s"/> <op id="ipvs_webapp_op_stop" interval="0" name="stop" timeout="40s"/> <op id="ipvs_webapp_op_monitor" interval="61s" name="monitor" timeout="30s"/> </operations> </primitive> <primitive class="ocf" id="ip_204.109.13.7" provider="heartbeat" type="IPaddr2"> <instance_attributes id="ip_204.109.13.7_ia"> <nvpair id="ip_204.109.13.7_ip" name="ip" value="204.109.13.7"/> <nvpair id="ip_204.109.13.7_nm" name="cidr_netmask" value="32"/> <nvpair id="ip_204.109.13.7_nic" name="nic" value="eth70"/> </instance_attributes> <operations> <op id="ip_204.109.13.7_op_start" interval="0" name="start" timeout="30s"/> <op id="ip_204.109.13.7_op_stop" interval="0" name="stop" timeout="90s"/> <op id="ip_204.109.13.7_op_monitor" interval="61s" name="monitor" timeout="30s"/> </operations> </primitive> </group>
88. the ipvs important bits <group description="Braintree Webapp" id="webapp_group"> <primitive class="ocf" id="ipvs_webapp" provider="braintree" type="ocf_big_brother"> <instance_attributes id="ipvs_webapp_ia"> <nvpair id="ipvs_monitor_webapp_cluster" name="cluster" value="webapp"/> </instance_attributes> <operations> <op id="ipvs_webapp_op_start" interval="0" name="start" timeout="20s"/> <op id="ipvs_webapp_op_stop" interval="0" name="stop" timeout="40s"/> <op id="ipvs_webapp_op_monitor" interval="61s" name="monitor" timeout="30s"/> </operations> </primitive> <primitive class="ocf" id="ip_204.109.13.7" provider="heartbeat" type="IPaddr2"> <instance_attributes id="ip_204.109.13.7_ia"> <nvpair id="ip_204.109.13.7_ip" name="ip" value="204.109.13.7"/> <nvpair id="ip_204.109.13.7_nm" name="cidr_netmask" value="32"/> <nvpair id="ip_204.109.13.7_nic" name="nic" value="eth70"/> </instance_attributes> <operations> <op id="ip_204.109.13.7_op_start" interval="0" name="start" timeout="30s"/> <op id="ip_204.109.13.7_op_stop" interval="0" name="stop" timeout="90s"/> <op id="ip_204.109.13.7_op_monitor" interval="61s" name="monitor" timeout="30s"/> </operations> </primitive> </group>
89. the ipaddr resource <group description="Braintree Webapp" id="webapp_group"> <primitive class="ocf" id="ipvs_webapp" provider="braintree" type="ocf_big_brother"> <instance_attributes id="ipvs_webapp_ia"> <nvpair id="ipvs_monitor_webapp_cluster" name="cluster" value="webapp"/> </instance_attributes> <operations> <op id="ipvs_webapp_op_start" interval="0" name="start" timeout="20s"/> <op id="ipvs_webapp_op_stop" interval="0" name="stop" timeout="40s"/> <op id="ipvs_webapp_op_monitor" interval="61s" name="monitor" timeout="30s"/> </operations> </primitive> <primitive class="ocf" id="ip_204.109.13.7" provider="heartbeat" type="IPaddr2"> <instance_attributes id="ip_204.109.13.7_ia"> <nvpair id="ip_204.109.13.7_ip" name="ip" value="204.109.13.7"/> <nvpair id="ip_204.109.13.7_nm" name="cidr_netmask" value="32"/> <nvpair id="ip_204.109.13.7_nic" name="nic" value="eth70"/> </instance_attributes> <operations> <op id="ip_204.109.13.7_op_start" interval="0" name="start" timeout="30s"/> <op id="ip_204.109.13.7_op_stop" interval="0" name="stop" timeout="90s"/> <op id="ip_204.109.13.7_op_monitor" interval="61s" name="monitor" timeout="30s"/> </operations> </primitive> </group>
90. the ipaddr important bits <group description="Braintree Webapp" id="webapp_group"> <primitive class="ocf" id="ipvs_webapp" provider="braintree" type="ocf_big_brother"> <instance_attributes id="ipvs_webapp_ia"> <nvpair id="ipvs_monitor_webapp_cluster" name="cluster" value="webapp"/> </instance_attributes> <operations> <op id="ipvs_webapp_op_start" interval="0" name="start" timeout="20s"/> <op id="ipvs_webapp_op_stop" interval="0" name="stop" timeout="40s"/> <op id="ipvs_webapp_op_monitor" interval="61s" name="monitor" timeout="30s"/> </operations> </primitive> <primitive class="ocf" id="ip_204.109.13.7" provider="heartbeat" type="IPaddr2"> <instance_attributes id="ip_204.109.13.7_ia"> <nvpair id="ip_204.109.13.7_ip" name="ip" value="204.109.13.7"/> <nvpair id="ip_204.109.13.7_nm" name="cidr_netmask" value="32"/> <nvpair id="ip_204.109.13.7_nic" name="nic" value="eth70"/> </instance_attributes> <operations> <op id="ip_204.109.13.7_op_start" interval="0" name="start" timeout="30s"/> <op id="ip_204.109.13.7_op_stop" interval="0" name="stop" timeout="90s"/> <op id="ip_204.109.13.7_op_monitor" interval="61s" name="monitor" timeout="30s"/> </operations> </primitive> </group>
91. XmHELL <group description="Braintree Webapp" id="webapp_group"> <primitive class="ocf" id="ipvs_webapp" provider="braintree" type="ocf_big_brother"> <instance_attributes id="ipvs_webapp_ia"> <nvpair id="ipvs_monitor_webapp_cluster" name="cluster" value="webapp"/> </instance_attributes> <operations> <op id="ipvs_webapp_op_start" interval="0" name="start" timeout="20s"/> <op id="ipvs_webapp_op_stop" interval="0" name="stop" timeout="40s"/> <op id="ipvs_webapp_op_monitor" interval="61s" name="monitor" timeout="30s"/> </operations> </primitive> <primitive class="ocf" id="ip_204.109.13.7" provider="heartbeat" type="IPaddr2"> <instance_attributes id="ip_204.109.13.7_ia"> <nvpair id="ip_204.109.13.7_ip" name="ip" value="204.109.13.7"/> <nvpair id="ip_204.109.13.7_nm" name="cidr_netmask" value="32"/> <nvpair id="ip_204.109.13.7_nic" name="nic" value="eth70"/> </instance_attributes> <operations> <op id="ip_204.109.13.7_op_start" interval="0" name="start" timeout="30s"/> <op id="ip_204.109.13.7_op_stop" interval="0" name="stop" timeout="90s"/> <op id="ip_204.109.13.7_op_monitor" interval="61s" name="monitor" timeout="30s"/> </operations> </primitive> </group>
92. puppetize! node "load1" { pacemaker::ipvs_resource { "webapp": description => "Braintree Webapp", supervisor_id => "webapp", ip_resources => [ { 'address' => '204.109.13.7', 'nic' => 'eth10' } ] } }
93. 201 Created 204.109.13.7 load02 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} POST /transactions 201 Created big_brother 92 load01 load03 204.109.13.100 app01 10.0.0.1 app02 app{N} litmus_paper 100 litmus_paper litmus_paper 92 55
94. 204.109.13.100 app01 204.109.13.7 load02 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} POST /transactions 10.0.0.1 201 Created app02 app{N} litmus_paper litmus_paper litmus_paper 100 92 55 big_brother 92 load01 load03 201 Created
95. lots of simple pieces
96. today what we value what we choose what we build
97. we have interesting jobs ™