Berlin 2013 - Session - Jeff Weinstein

Transcript

1. How  monitoring  can  improve   the  rest  of  the  company

         Monitorama  EU  2013   @jeff_weinstein  

2. I real-time and batch data analytics

3. Monitoring  can  wildly  improve     the  whole  company  by

     sharing  data     and  sharing  techniques.  

4. Monitoring  Folks   Developers   Business     Analysts  

   ExecuIves   &  Product   Data     ScienIsts   Data  

5. Apps  &   Services  &   Systems   Users  

   Data   Code  &   Config   Monitoring  

6. Some  problems…  

7. Data  Processing   Apps   Systems   Logs  /  

   Events   Metrics   Graphs   &  Alerts   Apps   3rd  Party   Reports  &   Queries   ETL   AnalyIc   Systems   Monitoring:  Streaming   BI:  Batch  

8. Data  Needs   Logs   Metrics   Logs   Metrics

     Streaming   Batch   Data   Monitoring   BI  

9. Data  Tools  Stack   Monitoring   •  Ad  hoc  

   –  sed,  grep,  awk   –  ES,  LogStash,  Splunk,  …   •  Storage   –  Hosts,  Ganglia,  OTSDB   –  Central  syslog  server   •  VisualizaIon/ReporIng   –  Graphite,  RRDTool,  3rd  party   –  Homegrown   •  AlerIng/EscalaIon     –  Nagios,  Sensu,  PagerDuty,  …   Rest  of  company   •  Ad  hoc   –  Excel,  SQL,  Hive   –  MapReduce,  …   •  Storage   –  Lots  o’  databases,  Excel   –  Hadoop,  RDBMS…   •  VisualizaIon/ReporIng   –  Excel,  R,  Tableau  ...   –  Dinosaur  apps,  …   •  AlerIng/EscalaIon     –  nada  

10. Metrics  

11. Views   Unintelligible  generated  views   Too  granular  for  long

     term  trends   Lack  of  historical   Intolerant  to  anomalies  

12. Team  and  incenIves   •  What  team?   •  Change

     vs.  reliability   •  Planning   •  Budget   •  Churn  

13. Good  or  bad?   •  Specific  Tools   •  Decentralized

      •  Focus   •  Ownership   •  Lost  context   •  Siloed  work   •  Data  dark   •  Misunderstanding  

14. Some  fixes  

15. End  to  End  Data  Pipeline   ü Structured  logs   ü (Config)

      ü Measure  once   ü AutomaIc  metrics   ü API   ü Graph  tools   ü Glossary   ü AnnotaIons  and  tags   ü Pipeline  

16. Structured  events   •  JSON  (or  whatever)   •  (opIonal)

     config   •  Tags  per  key   – Type   – Tag:  latency,  funnel,…   – DescripIon   – Storage  

17. Auto:  Graphs,  Glossary,  &  Storage   •  Graphs  and  dashboards

      •  *  templates   •  Views  and  stats   •  Glossary   •  Batch  analyIcs   •  Long  term  storage  

18. build   learn   communicate   inspire  

19. Developers   •  Logging  toolkit   •  Data  pipeline  

    •  Pain  points   •  Outage  causes   •  Deployment  pracIces   •  EscalaIon  playbook   •  Measurement  as  TDD   •  Monitor  staging  env  

20. Business  Analysts   •  Structured  logs     •  Config

     for  ETL   •  Metrics  definiIons     •  Slices  and  visualizaIons   •  Data  size  and  cardinality   •  Outages  and  delays   •  Flexibility   •  VisualizaIon  and  tools  

21. Data  ScienIsts   •  Access  to  (meta)data   •  Query

     monitoring   •  StaIsIcs  and  models   •  New  data  streams   •  Context  of  data  issues   •  What’s  in  the  logs   •  Validate  algorithms   •  Teach  stats  and  models!  

22. Product  &  ExecuIves   •  Curated  dashboards   •  Graph/alert

     tools   •  Learn  the  business   •  PrioriIze  alerts  by  $   •  Incident  post  mortems     •  Metrics  granularity   •  Data  driven  decisions   •  Recognize  and  celebrate  

23. Monitoring  can  become  the  data   plahorm  and  improve  all

     teams   with  its  techniques.  

24. Icons  from  The  Noun  Project:  Dmitry  Baranovskiy,  Benjamin  Orlovski,  Luis

     Prado,  MikaDo  Nguyen,  Yarden  Gilboa,  Javier  Cabezas,  Icons  Pusher,  Jeremy  Bristol,  Blake  Thomas,  RiIka  Khasgiwale,   Mayene  de  Leon,  Yorlmar  Campos,  Sergey  Shmid   @jeff_weinstein   Thanks!  hiring  ;)