Pro Yearly is on sale from $80 to $50! »

Berlin 2013 - Session - Ryan Smith

0580d500edfdb2e5e80e4732ac8df1ea?s=47 Monitorama
September 19, 2013
210

Berlin 2013 - Session - Ryan Smith

0580d500edfdb2e5e80e4732ac8df1ea?s=128

Monitorama

September 19, 2013
Tweet

Transcript

  1. Predictable Failure Building systems that fail in predictable ways.

  2. Failure is the inability to handle failure.

  3. Southern Airways Flight 242

  4. None
  5. What went wrong • Radar misguided pilots into the storm

    • Pilots applied thrust to a stalling engine • Close landing field was not suggested
  6. How will the system fail?

  7. How will Redis fail?

  8. None
  9. British Airways Flight 009

  10. What went wrong • All engines failed • St. Elmos

    fire but nothing on the radar • Radio to tower was not 100% • 1st officers oxygen mask broke
  11. What happens when the system fails

  12. No Redundancy - Simple func Query(c Conn, query string) Result

    { return c.DoQuery(query) }
  13. Redundancy - Complex func Query(conns []Conn, query string) Result {

    ch := make(chan Result, len(conns)) for _, conn := range conns { go func(c Conn) { ch <- c.DoQuery(query): }(conn) } return <-ch }
  14. Independence

  15. Same function Different implementation

  16. How many redundancies?

  17. A component that just works

  18. Dormant Failures

  19. Don't wait until disaster strikes to find out that Your

    secondary RDMS has a full disk
  20. Propagation

  21. Danke! @ryandotsmith