Building Large Scale Services - LISA 2013

Transcript

1. Building  Large  Scale  Services   PRESENTED  BY  Jennifer  Davis⎪  November  8,  2013  
2. Twitter: @sigje Email: sigje@yahoo.com
3. SysAdmin Controls all the things 11/11/13   3  
4. Shared Dependencies 11/11/13   4  
5. The Reality…   11/11/13   5  
6. The Dream… 11/11/13   6  
7. How?
8. Define Core Principles 11/11/13   8   § Common     ›  CollaboraGon  across  teams,  companies,  industry,   define  standards   ›  Incident,  Problem,  Change,  Config,  Release   management   § DisGnct   ›  Specifics  to  an  applicaGon  or  service   ›  Availability,  Service,  Business  ConGnuity,  Capacity    
9. Kill the Myths 11/11/13   9   § Stupid  User    
10. Kill the Myths   11/11/13   10   § Stupid  User   § System  Admin  ==  Operator    
11. 11/11/13   11   Failing Gracefully puppet ruby SKILLS perl nosql operability security mysql unix TCP/ IP bash CHEF
12. 11/11/13   12  
13. Kill the Myths   11/11/13   13   § Stupid  User   § System  Admin  ==  Operator   § Words  have  a  common  universal  implicit  meaning      
14. 11/11/13   14  
15. Learn to Modulate your Message   11/11/13   15      
16. 11/11/13   16   Team Manager Customer
17. Team 11/11/13   17   § People  working  towards  common  goal.   § Different  roles.     § Different  views.   § Same  objecGves.  
18. 11/11/13   18    Image  Credit:  Kyle  LaGno  
19. Team 11/11/13   19   Sugges/on:  Don’t  talk  about  the  “devs”  request,   talk  about  Elaine’s  request.    
20. Team 11/11/13   20   Sugges/on:  Don’t  talk  about  the  “devs”  request,   talk  about  Elaine’s  request.     Sugges/on:  Verify  that  your  team  has  the  same   vision.  
21. Understand the vision. 11/11/13   21   §  Are  there  other  opGons,  open  source  or  not  within  the  company?   §  Are  there  other  opGons  outside  the  company?   §  Is  EVERYONE  on  the  same  page  about  what  the  service  is?  
22. Vision Statement 11/11/13   22   §  Clear  statement  about  the  problem  that  the  service  is  solving.   ›  DirecGon   ›  IdenGty  management   ›  Team  cohesion   New  product?  Be  part  of  creaGng  that  vision!  
23. Sherpa’s Vision 11/11/13   23   ..  Distributed  replicated  eventually  consistent  key  value  store  that  had  a  focus   on  scalability  ..    
24. My Job 11/11/13   24   §  Examine  soaware   §  Define  risk   §  Communicate  cost  of  risks     §  MiGgate  risks   §  IdenGfy  events   §  Manage  events  
25. Fragile Platforms are Bad. 11/11/13   25  
26. Change is inevitable 11/11/13   26   §  Products  pivot  based  on  needs.   §  Requirements  change  and  evolve.   §  Know  core  issues.  
27. Know Core Issues 11/11/13   27   §  Limit  the  scope  of  focus.    
28. Know Core Issues 11/11/13   28   §  Limit  the  scope  of  focus.   §  Focus  on  the  biggest  prioriGes.    
29. Know Core Issues 11/11/13   29   §  Limit  the  scope  of  focus.   §  Focus  on  the  biggest  prioriGes.   ›  Understand  Development  Methodology:  Waterfall,  Scrum,  ?    
30. Know Core Issues 11/11/13   30   §  Limit  the  scope  of  focus.   §  Focus  on  the  biggest  prioriGes.   ›  Understand  Development  Methodology:  Waterfall,  Scrum,  ?   ›  IdenGfy  the  key  “Gme”  elements.    
31. Know Core Issues 11/11/13   31   §  Limit  the  scope  of  focus.   §  Focus  on  the  biggest  prioriGes.   ›  Understand  Development  Methodology:  Waterfall,  Scrum,  ?   ›  IdenGfy  the  key  “Gme”  elements.   ›  Talk  to  them.  IdenGfy  their  key  terms.  “Enhancements”,  “Defects”    
32. Know Core Issues 11/11/13   32   §  Limit  the  scope  of  focus.   §  Focus  on  the  biggest  prioriGes.   ›  Understand  Development  Methodology:  Waterfall,  Scrum,  ?   ›  IdenGfy  the  key  “Gme”  elements.   ›  Talk  to  them.  IdenGfy  their  key  terms.  “Enhancements”,  “Defects”   ›  Establish  the  “Top”  list.      
33. Create checklists 11/11/13   33   §  Not  because  people  are  dumb.   §  Not  only  because  of  automaGon.   §  When  things  break,  knowing  what  needs  focus.   §  During  normal  maintenance,  can  idenGfy  “not  OK”.   ›  Audit  checklists  for  deployment  through  staging  environment.  
34. Know Outputs 11/11/13   34   §  IdenGfy  components.   §  Well  defined  protocols  between  components.   §  Expected  Inputs.   §  Expected  Outputs.  
35. 11/11/13   35  
36. 11/11/13   36  
37. 11/11/13   37  
38. 11/11/13   38  
39. 11/11/13   39  
40. Know State Transitions Explicitly. 11/11/13   40   §  When  component  is  installed  but  not  ready  
41. Know State Transitions Explicitly. 11/11/13   41   §  When  component  is  installed  but  not  ready   §  When  the  colo  is  going  away   §  Go  through  What  If  Scenarios.   ›  Document  them.  
42. Know choke points explicitly. 11/11/13   42   §  Memory   §  Disk   §  Bandwidth   Now  and  in  6  months.   JIT?  
43. Failure will happen. 11/11/13   43   §  There  are  no  0  failure  systems.   §   “Give  me  the  brain”  documentaGon  so  that  anyone  can  be  the  brain.   §  Repeatable/Reliable  failure  handling.   §  Run  fire  drills.  Really.    
44. 11/11/13   44  
45. System Administration is Gardening. 11/11/13   45   §  No  guarantee  of  resources.   §  Only  guarantee  is  change.  
46. System Administration is Gardening. 11/11/13   46   §  Nurture  relaGonships.   ›  Be  authenGc.   ›  Be  trusGng  and  trustworthy.   ›  Have  integrity.  
47. Success At Scale is Collaboration & Cooperation across Teams.
48. Decreasing Value 11/11/13   48  
49. 11/11/13   49   0 2 4 6 8 Jan Apr Jul Oct # of Support Engineers # of Support Engineers
50. 11/11/13   50   0 1 2 3 4 5 6 Jan Apr Jul Oct # of Support Engineers # of Support Engineers
51. 11/11/13   51  
52. Documentation is not the cure. 11/11/13   52   § DocumentaGon  doesn’t  guarantee   understanding.   ›  OperaGons  Sandbox  Environment   § Don’t  spend  Gme  at  the  end  documenGng.  
53. 53   11/11/13  
54. Summary
55. Be Expendable. Feed your brain. 11/11/13   55  
56. Acknowledgements 11/11/13   56   •  hkp://www.flickr.com/photos/levork     •  hkp://www.flickr.com/photos/puggles   •  hkp://www.flickr.com/photos/byteorder   •  hkp://www.flickr.com/photos/egoant   •  hkp://www.flickr.com/photos/happymonkey   •  Kyle  LaGno     •  Greg  Connor    
57. Thanks! 11/11/13   57   sigje@yahoo.com http://www.slideshare.net/sigje/ presentation-lisa