Who Needs Clouds?: HA in Your Datacenter

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Tony Pitluga Paul Hinze @pitluga @phinze

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Our world

Slide 5

Slide 5 text

Lots of people trust us

Slide 6

Slide 6 text

> 1 million API calls per day

Slide 7

Slide 7 text

> 1 million API calls per day $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $$ $ $ $ $ $ important

Slide 8

Slide 8 text

Downtime really sucks $ $ $ $ $ $ $ $ $ $

Slide 9

Slide 9 text

we have interesting jobs in other words

Slide 10

Slide 10 text

What we value

Slide 11

Slide 11 text

simple easy Open closed logic not not not magic

Slide 12

Slide 12 text

What we choose

Slide 13

Slide 13 text

not not not linux-ha

Slide 14

Slide 14 text

Today

Slide 15

Slide 15 text

our beloved Load balancers

Slide 16

Slide 16 text

scalability availability +

Slide 17

Slide 17 text

linux-ha

Slide 18

Slide 18 text

Thou Shalt Do One Thing Well

Slide 19

Slide 19 text

Don’t Panic!

Slide 20

Slide 20 text

lots of simple pieces

Slide 21

Slide 21 text

Whole > Sum ( parts )

Slide 22

Slide 22 text

204.109.13.100 app01 204.109.13.7 load01 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} POST /transactions 10.0.0.1 201 Created app02 app{N} balancing act 1 2 3 4

Slide 23

Slide 23 text

204.109.13.100 app01 204.109.13.7 load01 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} POST /transactions 10.0.0.1 201 Created app02 app{N} (1) ip virtual server

Slide 24

Slide 24 text

204.109.13.100 app01 204.109.13.7 load01 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} POST /transactions 10.0.0.1 201 Created app02 app{N} (2) litmus_paper litmus_paper 100

Slide 25

Slide 25 text

204.109.13.100 app01 204.109.13.7 load01 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} POST /transactions 10.0.0.1 201 Created app02 app{N} (3) big_brother litmus_paper 201 Created big_brother 100

Slide 26

Slide 26 text

204.109.13.100 app01 204.109.13.7 load01 POST /transactions 10.0.0.1 201 Created app02 app{N} (4) pacemaker litmus_paper 100 load02 load03

Slide 27

Slide 27 text

here goes!

Slide 28

Slide 28 text

load01 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} IP Virtual server (ipvs)

Slide 29

Slide 29 text

on your mark iptables --table mangle \ --append PREROUTING \ --destination 205.109.13.7/32 \ --interface eth70 \ --protocol tcp \ --match tcp --dport 443 \ --jump MARK --set-mark 0x07

Slide 30

Slide 30 text

iptables --table mangle \ --append PREROUTING \ --destination 205.109.13.7/32 \ --interface eth70 \ --protocol tcp \ --match tcp --dport 443 \ --jump MARK --set-mark 0x07 on your mark

Slide 31

Slide 31 text

get set ipvsadm --add-service --fwmark-service 7 --scheduler wlc

Slide 32

Slide 32 text

get set ipvsadm --add-service --fwmark-service 7 --scheduler wlc

Slide 33

Slide 33 text

and they’re off! ipvsadm --add-server --fwmark-service 7 --real-server 10.0.0.1 --ipip --weight 100

Slide 34

Slide 34 text

and they’re off! ipvsadm --add-server --fwmark-service 7 --real-server 10.0.0.1 --ipip --weight 100

Slide 35

Slide 35 text

and they’re off! ipvsadm --add-server --fwmark-service 7 --real-server 10.0.0.2 --ipip --weight 100

Slide 36

Slide 36 text

204.109.13.100 app01 204.109.13.7 load01 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} POST /transactions 10.0.0.1 201 Created app02 app{N} now you’re balancing with FOSS!

Slide 37

Slide 37 text

ok, so what?

Slide 38

Slide 38 text

weight weight don’t tell me ipvsadm --add-server --fwmark-service 7 --real-server 10.0.0.1 --ipip --weight 100

Slide 39

Slide 39 text

weight weight don’t tell me ipvsadm --add-service --fwmark-service 7 --scheduler wlc

Slide 40

Slide 40 text

The Power is yours!

Slide 41

Slide 41 text

weight == “Health” of node

Slide 42

Slide 42 text

how do we get health?

Slide 43

Slide 43 text

litmus_paper braintree/litmus_paper

Slide 44

Slide 44 text

app01 Determines health of single node litmus_paper

Slide 45

Slide 45 text

health == Dependencies + Metrics

Slide 46

Slide 46 text

a litmus_paper check service "webapp" do |s| s.depends Dependency::HTTP, "http://localhost:3000/heartbeat" s.measure_health Metric::CPULoad, :weight => 50 s.measure_health Metric::AvailableMemory, :weight => 50 end

Slide 47

Slide 47 text

Slide 48

Slide 48 text

Slide 49

Slide 49 text

easy to extend require 'postfix_litmus' service "mailman" do |s| s.depends PostfixLitmus::Dependency, "localhost:543" s.measure_health PostfixLitmus::MailQSize, :max_size => 1000, :weight => 100 end

Slide 50

Slide 50 text

easy httpeasy $ curl localhost Litmus Paper 0.6.0 Services monitored: * webapp (87.5)

Slide 51

Slide 51 text

easy httpeasy $ curl localhost/webapp/status Health: 87.5 Dependency::HTTP(http://localhost:3000/heartbeat): OK Metric::CPULoad(50): 49.5 Metric::AvailableMemory(50): 38

Slide 52

Slide 52 text

Maintenance w/ litmusctl $ litmusctl force down webapp Reason? apt upgrades $ curl localhost/webapp/status Health: 0 apt upgrades

Slide 53

Slide 53 text

204.109.13.100 app01 204.109.13.7 load01 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} POST /transactions 10.0.0.1 201 Created app02 app{N} litmus keeping tabs litmus_paper litmus_paper litmus_paper 100 92 55

Slide 54

Slide 54 text

litmus will answer but who will ask?

Slide 55

Slide 55 text

big_brother braintree/big_brother

Slide 56

Slide 56 text

load01 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} monitors litmus updates ipvs big_brother

Slide 57

Slide 57 text

it’s simple, really while true health = `curl node/app/status \ | grep Health \ | awk ‘{print $2}’` `ipvsadm --edit-server --weight #{health}` sleep 2 end

Slide 58

Slide 58 text

config, brother webapp: checkInterval: 2000 scheduler: wlc fwmark: 7 nodes: - address: 10.0.0.1 port: 80 path: /webapp/status - address: 10.0.0.2 port: 80 path: /webapp/status

Slide 59

Slide 59 text

204.109.13.100 app01 204.109.13.7 load01 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} POST /transactions 10.0.0.1 201 Created app02 app{N} big_brother is watching litmus_paper litmus_paper litmus_paper 100 92 55 big_brother 92

Slide 60

Slide 60 text

big_brother what can Do FOR YOU?

Slide 61

Slide 61 text

Yale Collection of Western Americana, Beinecke Rare Book and Manuscript Library

Slide 62

Slide 62 text

15 load01 balancing by health app01 app02 10 app03 25 30% 20% 50%

Slide 63

Slide 63 text

15 load01 new kid on the block app01 app02 10 app03 25 30% 20% 50% app04 100 yeah!

Slide 64

Slide 64 text

15 load01 accosted like a boy band app01 app02 10 app03 25 10% 6% 16% app04 100 66%

Slide 65

Slide 65 text

load01 accosted like a boy band app04 66% 5 ouch! 15 app01 app02 10 app03 25 10% 6% 16%

Slide 66

Slide 66 text

config, brother webapp: checkInterval: 2000 scheduler: wlc fwmark: 7 ramp_up_time: 120 nodes: - address: 10.0.0.1 port: 80 path: /webapp/status - address: 10.0.0.2 port: 80 path: /webapp/status try this

Slide 67

Slide 67 text

15 load01 Humble kid on the block app01 app02 10 app03 25 30% 20% 50% app04 5 easy now

Slide 68

Slide 68 text

15 load01 ease off the pressure 10 25 27% 18% 45% app04 5 9% app01 app02 app03

Slide 69

Slide 69 text

15 load01 ease off the pressure 10 25 25% 16% 41% app04 5 16% app01 app02 app03

Slide 70

Slide 70 text

15 load01 even out 10 25 app04 5 app01 app02 app03 23% 24% 27% 26%

Slide 71

Slide 71 text

78 load01 we all get healthy together 82 95 app04 93 app01 app02 app03 23% 24% 27% 26%

Slide 72

Slide 72 text

Slide 73

Slide 73 text

who load balances the load balancers ?

Slide 74

Slide 74 text

No content

Slide 75

Slide 75 text

No content

Slide 76

Slide 76 text

No content

Slide 77

Slide 77 text

load01 load02 load03 manages resources across a cluster

Slide 78

Slide 78 text

resource == open cluster framework (ocf)

Slide 79

Slide 79 text

ocf scripts /usr/lib/ocf/resource.d/braintree/ocf_big_brother

Slide 80

Slide 80 text

ocf monitor /usr/lib/ocf/resource.d/braintree/ocf_big_brother $ curl localhost:9292/cluster/app Running: true

Slide 81

Slide 81 text

ocf monitor /usr/lib/ocf/resource.d/braintree/ocf_big_brother $ curl -XPUT localhost:9292/cluster/app OK

Slide 82

Slide 82 text

ocf monitor /usr/lib/ocf/resource.d/braintree/ocf_big_brother $ curl -XDELETE localhost:9292/cluster/app OK

Slide 83

Slide 83 text

resource == virtual IP(s) + IPVS cluster

Slide 84

Slide 84 text

IPaddr2

Slide 85

Slide 85 text

XmHELL

Slide 86

Slide 86 text

groups flock together

Slide 87

Slide 87 text

the ipvs resource

Slide 88

Slide 88 text

the ipvs important bits

Slide 89

Slide 89 text

the ipaddr resource

Slide 90

Slide 90 text

the ipaddr important bits

Slide 91

Slide 91 text

XmHELL

Slide 92

Slide 92 text

puppetize! node "load1" { pacemaker::ipvs_resource { "webapp": description => "Braintree Webapp", supervisor_id => "webapp", ip_resources => [ { 'address' => '204.109.13.7', 'nic' => 'eth10' } ] } }

Slide 93

Slide 93 text

201 Created 204.109.13.7 load02 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} POST /transactions 201 Created big_brother 92 load01 load03 204.109.13.100 app01 10.0.0.1 app02 app{N} litmus_paper 100 litmus_paper litmus_paper 92 55

Slide 94

Slide 94 text

204.109.13.100 app01 204.109.13.7 load02 10.0.0.1 100 10.0.0.2 100 ... 10.0.0.{N} {W} POST /transactions 10.0.0.1 201 Created app02 app{N} litmus_paper litmus_paper litmus_paper 100 92 55 big_brother 92 load01 load03 201 Created