Tweet
Tweet

Slide 1

Slide 1 text

Smoke and Mirrors The Primitives of High Availability

Slide 2

Slide 2 text

paul hinze @phinze

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

abstraction “ ”

Slide 5

Slide 5 text

abstrahere (v.) to draw away

Slide 6

Slide 6 text

The internet Abstraction in Action

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

1 2 3 4 5 6 7 8 9 GitHub · Build software better, together. 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51

Slide 10

Slide 10 text

278
  • Blog
    • 279
  • About
    • 280 281 282 283 284 285 286
      287
    • © 2015 GitHub, Inc.
      • 288
    • Terms
      • 289
    • Privacy
      • 290
    • Security
      • 291
    • Contact
      • 292
    293 294 295 296 297
    298
    299
    300 301
    302
    303
    304
    305
    306
    307 308 309 310 312 313 314
    315
    316 317 318 319 320 321
    322 323 324 Something went wrong with that request. Please try again. 325
    326 327 328 329 330 331 332 333 334 335

    Slide 11

    Slide 11 text

    No content

    Slide 12

    Slide 12 text

    github.com … ? 192.30.252.130 DNS Domain Name System

    Slide 13

    Slide 13 text

    GET / HTTP/1.1 HTTP/1.1 200 OK ( … ) HTTP Hypertext Transport Protocol 192.30.252.130

    Slide 14

    Slide 14 text

    GET / HTTP/1.1 HTTP/1.1 200 OK TCP Transmission Control Protocol

    Slide 15

    Slide 15 text

    TCP Transmission Control Protocol

    Slide 16

    Slide 16 text

    IP Internet Protocol

    Slide 17

    Slide 17 text

    mtr github.com

    Slide 18

    Slide 18 text

    192.168.101.1 73.8.160.1 69.139.232.241 69.139.185.81 68.86.197.125 68.86.187.149 68.87.232.89 68.87.210.73 IP Internet Protocol 192.30.252.207 192.30.252.130 68.86.197.113 68.86.92.33 68.86.84.210 68.86.86.225 68.86.85.25 68.86.85.1 68.86.82.98 50.242.151.74 ?

    Slide 19

    Slide 19 text

    192.30.252.130 1.1.0.0/16 C 1.2.0.0/16 B 1.3.0.0/16 A 1.4.0.0/16 C 1.5.0.0/16 D 1.6.0.0/16 A 1.7.0.0/16 D 1.8.0.0/16 B 1.9.0.0/16 C 1.10.0.0/16 C 1.11.0.0/16 B 1.12.0.0/16 B 1.13.0.0/16 D IP Internet Protocol A B C D PLEASE SEND TO:

    Slide 20

    Slide 20 text

    BGP Border Gateway Protocol A B C D ANNOUNCE WITHDRAW UPDATE

    Slide 21

    Slide 21 text

    BGP Border Gateway Protocol

    Slide 22

    Slide 22 text

    IP Internet Protocol

    Slide 23

    Slide 23 text

    IP Internet Protocol

    Slide 24

    Slide 24 text

    TCP Transmission Control Protocol

    Slide 25

    Slide 25 text

    GET / HTTP/1.1 HTTP/1.1 200 OK HTTP Hypertext Transport Protocol

    Slide 26

    Slide 26 text

    No content

    Slide 27

    Slide 27 text

    No content

    Slide 28

    Slide 28 text

    The internet Abstraction in Action

    Slide 29

    Slide 29 text

    high AVAILABILITY

    Slide 30

    Slide 30 text

    Not Highly Available

    Slide 31

    Slide 31 text

    High Availability ≈ Fault Tolerance

    Slide 32

    Slide 32 text

    No content

    Slide 33

    Slide 33 text

    Failure Happens

    Slide 34

    Slide 34 text

    Anticipate Prepare React

    Slide 35

    Slide 35 text

    What could fail? What should we do when it fails? What did we learn?

    Slide 36

    Slide 36 text

    Primitives

    Slide 37

    Slide 37 text

    Redundancy

    Slide 38

    Slide 38 text

    Treat many as one

    Slide 39

    Slide 39 text

    Hardware Components Fail

    Slide 40

    Slide 40 text

    LACP Link Aggregation Control Protocol RAID Redundant Array of Independent Disks

    Slide 41

    Slide 41 text

    Servers Fail

    Slide 42

    Slide 42 text

    Server Redundancy

    Slide 43

    Slide 43 text

    Transparent Proxy 192.30.252.128 10.0.0.11 10.0.0.12 10.0.0.13

    Slide 44

    Slide 44 text

    Load Balancing 100% 33% 33% 33%

    Slide 45

    Slide 45 text

    Load Balancing 100% 50% 50% 0%

    Slide 46

    Slide 46 text

    Heartbeat You OK? I’m OK! You OK? I’m OK! You OK? … timeout!

    Slide 47

    Slide 47 text

    Single Point of Failure SPOF

    Slide 48

    Slide 48 text

    github.com … ? 192.30.252.130 DNS Domain Name System 192.30.252.129 192.30.252.128

    Slide 49

    Slide 49 text

    github.com … ? 192.30.252.130 DNS Domain Name System 192.30.252.1 192.30.252.128

    Slide 50

    Slide 50 text

    192.30.252.130 DNS Domain Name System 192.30.252.129 192.30.252.1 192.30.252.128 15m 192.30.252.128 5m

    Slide 51

    Slide 51 text

    192.30.252.130 DNS Domain Name System 192.30.252.129 192.30.252.1 192.30.252.128 15m 192.30.252.128 5m

    Slide 52

    Slide 52 text

    Clustering

    Slide 53

    Slide 53 text

    Clustering 192.30.252.130

    Slide 54

    Slide 54 text

    Clustering 192.30.252.130 B A C

    Slide 55

    Slide 55 text

    Failover 192.30.252.130 B A C

    Slide 56

    Slide 56 text

    Timeouts and Retries

    Slide 57

    Slide 57 text

    No content

    Slide 58

    Slide 58 text

    Managing State

    Slide 59

    Slide 59 text

    CAP Theorem

    Slide 60

    Slide 60 text

    Replication Synchronous Asynchronous

    Slide 61

    Slide 61 text

    Replication Synchronous Asynchronous +C -P +P -C

    Slide 62

    Slide 62 text

    Replication + Clustering + Automatic Failover + Load Balancing

    Slide 63

    Slide 63 text

    Monitoring

    Slide 64

    Slide 64 text

    Know Your Limits

    Slide 65

    Slide 65 text

    Disk Space Memory CPU I/O Network (Entropy)

    Slide 66

    Slide 66 text

    Anticipate Prepare React

    Slide 67

    Slide 67 text

    What happens when it fails?

    Slide 68

    Slide 68 text

    What’s the HA story for $TECH?

    Slide 69

    Slide 69 text

    SomeDay

    Slide 70

    Slide 70 text

    Abstraction Redundancy Load Balancing Heartbeats Clustering Automatic Failover Timeouts and Retries Replication Monitoring

    Slide 71

    Slide 71 text

    Perseverance Abstraction Redundancy Load Balancing Heartbeats Clustering Automatic Failover Timeouts and Retries Replication Monitoring

    Slide 72

    Slide 72 text

    thank you Maps from Free Vector Maps Other Graphics from The Noun Project by hunotika by Joe Harrison by Ben Rizzo by Yazmin Alanis by Ham Stanford by gira Park by NAMIRUS by Jamie Carrion by MikaDo Nguyen BGP Data from the BGP Instability Report