Planning to Fail (PHPNE)

Slightly longer version of my talk at PHP North East, on avoiding failure by failing all the time.

Transcript

1. Planning
2. dave!
3. the taxi app !
4. Planning
5. Planning
6. Planning
7. Why? h"p://en.wikipedia.org/wiki/High_availability    
8. 99.9%      (three  nines)     Down?me:   43.8  minutes  per  month   8.76  hours  per  year  
9. 99.99%    (four  nines)     Down?me:   4.32  minutes  per  month   52.56  minutes  per  year  
10. 99.999%      (five  nines)     Down?me:   25.9  seconds  per  month   5.26  minutes  per  year  
11. www.whoownsmyavailability.com     ?  
12. www.whoownsmyavailability.com     YOU  
13. The beginning
14. <?php
15. My  website:  single  VPS  running  PHP  +  MySQL  
16. No  growth,  low  volume,  simple  func?onality,  one  engineer  (me!)  
17. Large  growth,  high  volume,  complex  func?onality,  lots  of  engineers  
18. • Launched  in  London   November  2011   • Now  in  5  ci?es  in  3  countries   (30%+  growth  every  month)   • A  Hailo  hail  is  accepted  around   the  world  every  5  seconds  
19. “..  Brooks  [1]  reveals  that  the  complexity   of  a  so^ware  project  grows  as  the  square   of  the  number  of  engineers  and  Leveson   [17]  cites  evidence  that  most  failures  in   complex  systems  result  from  unexpected   inter-­‐component  interac?on  rather  than   intra-­‐component  bugs,  we  conclude  that   less  machinery  is  (quadra?cally)  be"er.”     h"p://lab.mscs.mu.edu/Dist2012/lectures/HarvestYield.pdf  
20. • SOA  (10+  services)   • AWS  (3  regions,  9  AZs,  lots  of   instances)   • 10+  engineers  building  services   and you?! (hailo is hiring)!
21. Our  overall   reliability  is  in   danger  
22. Embracing failure   (a  coping  strategy)
24. VPC   (running  PHP+MySQL)   reliable?!
25. Reliable   !==   Resilient  
26. Choosing a stack
27. “Hailo”   (running  PHP+MySQL)   reliable?!
28. Service   each service does one job well! Service   Service   Service   Service  Oriented  Architecture  
29. • Fewer  lines  of  code   • Fewer  responsibili?es   • Changes  less  frequently   • Can  swap  en?re  implementa?on   if  needed  
30. Service   (running  PHP+MySQL)   reliable?!
31. Service   MySQL   MySQL  running  on  different  box  
32. Service   MySQL   MySQL   MySQL  running  in  Mul?-­‐Master  mode  
33. Going  global  
34. MySQL   Separa?ng  concerns   CRUD   Locking   Search   Analy?cs   ID  genera?on   also queuing…!
35. At  Hailo  we  look  for  technologies  that  are:   • Distributed   run  on  more  than  one  machine   • Homogenous   all  nodes  look  the  same   • Resilient   can  cope  with  the  loss  of  node(s)  with  no   loss  of  data  
36. “There  is  no  such  thing  as  standby   infrastructure:  there  is  stuff  you   always  use  and  stuff  that  won’t   work  when  you  need  it.”       h"p://blog.b3k.us/2012/01/24/some-­‐rules.html  
37. • Highly  performant,  scalable  and   resilient  data  store   • Underpins  much  of  what  we  do   at  Hailo   • Makes  mul?-­‐DC  easy!  
38. • Highly  reliable  distributed   coordina?on   • We  implement  locking  and   leadership  elec?on  on  top  of  ZK   and  use  sparingly   ZooKeeper
39. • Distributed,  RESTful,  Search   Engine  built  on  top  of  Apache   Lucene   • Replaced  basic  foo  LIKE  ‘%bar%’   queries  (so  much  be"er)  
40. • Real?me  message  processing   system  designed  to  handle   billions  of  messages  per  day   • Fault  tolerant,  highly  available   with  reliable  message  delivery   guarantee   NSQ
41. • Real  ?me  incremental  analy?cs   plasorm,  backed  by  Apache   Cassandra   • Powerful  SQL-­‐like  interface   • Scalable  and  highly  available  
42. • Distributed  ID  genera?on  with   no  coordina?on  required   • Rock  solid   Cruftflake
43. • All  these  technologies  have   similar  proper?es  of  distribu?on   and  resilience     • They  are  designed  to  cope  with   failure   • They  are  not  broken  by  design  
44. Lessons learned
45. Minimise  the   cri?cal  path  
46. What  is  the  minimum  viable  service?  
47. class HailoMemcacheService { private $mc = null; public function __call() { $mc = $this->getInstance(); // do stuff } private function getInstance() { if ($this->instance === null) { $this->mc = new \Memcached; $this->mc->addServers($s); } return $this->mc; } } Lazy-­‐init  instances;  connect  on  use  
48. Configure  clients   carefully  
49. $this->mc = new \Memcached; $this->mc->addServers($s); $this->mc->setOption( \Memcached::OPT_CONNECT_TIMEOUT, $connectTimeout); $this->mc->setOption( \Memcached::OPT_SEND_TIMEOUT, $sendRecvTimeout); $this->mc->setOption( \Memcached::OPT_RECV_TIMEOUT, $sendRecvTimeout); $this->mc->setOption( \Memcached::OPT_POLL_TIMEOUT, $connectionPollTimeout); Make  sure  ?meouts  are  configured  
50. Choose  ?meouts  based  on  data   here?!
51. “Fail  Fast:  Set  aggressive  ?meouts   such  that  failing  components   don’t  make  the  en?re  system   crawl  to  a  halt.”       h"p://techblog.neslix.com/2011/04/lessons-­‐ neslix-­‐learned-­‐from-­‐aws-­‐outage.html  
52. 95th  percen?le   here?!
53. Test  
54. • Kill  memcache  on  box  A,   measure  impact  on  applica?on   • Kill  memcache  on  box  B,   measure  impact  on  applica?on     All  fine..  we’ve  got  this  covered!  
55. FAIL  
56. • Box  A,  running  in  AWS,  locks  up   • Any  parts  of  applica?on  that   touch  Memcache  stop  working  
57. Things  fail  in   exo?c  ways  
58. $ iptables -A INPUT -i eth0 \ -p tcp --dport 11211 -j REJECT $ php test-memcache.php Working OK! Packets  rejected  and  source  no?fied  by  ICMP.  Expect  fast  fails.  
59. $ iptables -A INPUT -i eth0 \ -p tcp --dport 11211 -j DROP $ php test-memcache.php Working OK! Packets  silently  dropped.  Expect  long  ?me  outs.  
60. $ iptables -A INPUT -i eth0 \ -p tcp --dport 11211 \ -m state --state ESTABLISHED \ -j DROP $ php test-memcache.php Hangs!  Uh  oh.  
61. • When  AWS  instances  hang  they   appear  to  accept  connec?ons   but  drop  packets   • Bug!     h"ps://bugs.launchpad.net/libmemcached/ +bug/583031  
62. Fix,  rinse,  repeat  
63. RabbitMQ   RabbitMQ   RabbitMQ   Service   AMQP  (port  5672)   HA  cluster  
64. $ iptables -A INPUT -i eth0 \ -p tcp --dport 5672 \ -m state --state ESTABLISHED \ -j DROP $ php test-rabbitmq.php Fantas?c!  Block  AMQP  port,  client  ?mes  out  
65. FAIL  
66. “RabbitMQ  clusters  do  not   tolerate  network  par??ons   well.”         h"p://www.rabbitmq.com/par??ons.html    
67. $ epmd –names epmd: up and running on port 4369 with data: name rabbit at port 60278 Each  node  listens  on  a  port  assigned  by  EPMD  
69. $ iptables -A INPUT -i eth0 \ -p tcp --dport 60278 \ -m state --state ESTABLISHED \ -j DROP $ php test-rabbitmq.php Hangs!  Uh  oh.  
70. Mnesia('rabbit@dmzutilities03-global01- test'): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, 'rabbit@dmzutilities01-global01-test'} application: rabbitmq_management exited: shutdown type: temporary RabbitMQ  logs  show  par??oned  network  error;  nodes  shutdown  
72. while ($read < $n && !feof($this->sock->real_sock()) && (false !== ($buf = fread( $this->sock->real_sock(), $n - $read)))) { $read += strlen($buf); $res .= $buf; } PHP  library  didn’t  have  any  ?me  limit  on  reading  a  frame  
73. Fix,  rinse,  repeat  
74. It  would  be     nice  if  we  could   automate  this  
75. Automate!  
76. • Hailo  run  a  dedicated  automated   test  environment   • Powered  by  bash,  JMeter  and   Graphite   • Con?nuous  automated  tes?ng   with  failure  simula?ons  
77. Fix  a"empt  1:  bad  ?meouts  configured  
78. Fix  a"empt  2:  be"er  ?meouts  
79. Simulate  in   system  tests  
80. Simulate  failure   Assert  monitoring  endpoint   picks  this  up   Assert  features  sBll  work  
81. In conclusion
82. “the  best  way  to  avoid   failure  is  to  fail  constantly.”         h"p://www.codinghorror.com/blog/2011/04/ working-­‐with-­‐the-­‐chaos-­‐monkey.html  
83. You  should  test  for   failure       How  does  the  so^ware  react?   How  does  the  PHP  client  react?  
84. AutomaBon  makes   conBnuous  failure   tesBng  feasible  
85. Systems  that  cope  well   with  failure  are  easier   to  operate  
86. TIMED  BLOCK  ALL     THE  THINGS  
87. Thanks     So^ware  used  at  Hailo   h"p://cassandra.apache.org/   h"p://zookeeper.apache.org/   h"p://www.elas?csearch.org/   h"p://www.acunu.com/acunu-­‐analy?cs.html   h"ps://github.com/bitly/nsq   h"ps://github.com/davegardnerisme/cru^flake   h"ps://github.com/davegardnerisme/nsqphp   Plus  a  load  of  other  things  I’ve  not  men?oned.  
88. Further  reading   Hystrix:  Latency  and  Fault  Tolerance  for  Distributed  Systems   h"ps://github.com/Neslix/Hystrix   Timelike:  a  network  simulator   h"p://aphyr.com/posts/277-­‐?melike-­‐a-­‐network-­‐simulator   Notes  on  distributed  systems  for  young  bloods   h"p://www.somethingsimilar.com/2013/01/14/notes-­‐on-­‐distributed-­‐ systems-­‐for-­‐young-­‐bloods/   Stream  de-­‐duplica?on  (relevant  to  NSQ)   h"p://www.davegardner.me.uk/blog/2012/11/06/stream-­‐de-­‐ duplica?on/    ID  genera?on  in  distributed  systems   h"p://www.slideshare.net/davegardnerisme/unique-­‐id-­‐genera?on-­‐in-­‐ distributed-­‐systems