How we made an API service that never went down...

Transcript

1. How  we  made  an  API  service  that   never  went  down  and  why  that   wasn’t  the  victory  I’d  intended.   Ken  Wronkiewicz  (@wirehead)   Manager,  SoBware  Development   Rackspace  
3. “So,  do  you  have   Auto  Scale?”   “No”   “Yes!”  
6. Ticker   Plant   Ticker   Plant   NYSE  /  NASDAQ   Central   Distributor   Central   Distributor   Regional   Distributor   Customer   Distributor   Customer   Distributor   Stock   Trader   Stock   Trader   Stock   Trader   Control   Plane   Control   Plane   DB   DB   DB   DB  
7. Ticker   Plant   Ticker   Plant   NYSE  /  NASDAQ   Central   Distributor   Central   Distributor   Rack-­‐level   Distributor   Rack-­‐level   Distributor   Rack-­‐level   Distributor  
11. -­‐ Test  coverage:   -­‐ Unit  tests   -­‐ IntegraXon  tests   -­‐ CI/CD  pipelines   -­‐ Code  review  pracXce   -­‐ OperaXonal  pracXce   -­‐ AutomaXon  instead  of  manual  steps.  
12. Cassandra   Cassandra   Cassandra   Cassandra   API  /   Worker   API  /   Worker   API  /   Worker   API  /   Worker   Cloud  Load   Balancer  
13. Some  lies  we  tell  ourselves  about   producXon  operaXons:   •  Servers  won’t  go  down   •  You’ll  never  loose  a  colo   •  You’ll  never  see  a  net  split  inside  of  a  colo   •  Users  will  never  noXce  a  brief  blip  
15. Availability   ParXXon-­‐ Tolerance   Consistency   Cassandra   Zookeeper  
17. Get     Lock   Release   Lock   Request   Response   Get     Lock   Release   Lock   Get     Lock   Release   Lock   Request  
18. 2   4   1   10  buckets   [1-­‐3]   [4-­‐7]   [8-­‐10]   Front  end  nodes   Stale   Events   WaiXng   Events  
21. Disaster  day  drills   •  Start  in  staging,  but  do  it  in  producXon   •  Pick  a  Dungeon  Master   •  If  your  monitoring/alerXng/paging  pipeline  is   working,  the  second  the  DM  breaks  something,   you  should  noXce.   •  You  make  sure  that  people  are  comfortable  with   each  and  every  runbook.  
22. 90%  =  More  than  a  month  downXme  per  year   99%  =  Less  than  4  days  per  year   99.9%  =  Less  than  9  hours  per  year   99.99%  =  Less  than  an  hour  per  year   99.999%  =  ~5  minutes  per  year   99.9999%  =  ~31  seconds  per  year   HTTP  500  Nein!  
24. A  victory,  yes..   But  not  the   one  I’d   intended..  
25. Thanks!   Ken  Wronkiewicz  (@wirehead)