Splunk as our Platform

Tim and Jim Splunk Live from 2011(?)

Transcript

1. Splunk>    as  our  pla-orm  
2. University Network Services Group Tim Hartmann Unix SA Jim Donn NMS Engineer
3. A  li1le  about  us   James  Donn   Sr.  Network  and  Systems   Management  Engineer   •  Design  and  implement   enterprise  level  monitoring   systems  and  tools   •  Deliver  solu?ons  for  log   management  and  audi?ng   •  Develop  custom  monitoring   solu?ons   •  Part  ?me  Sys  Admin   Tim  Hartmann   Sr.  Network  and  Services  Engineer   •  Unix  installa?on,  configura?on,   opera?ons,  and  maintenance  of   soGware  and  networked  servers   •  Systems  diagnos?cs  and   troubleshoo?ng     •  Specifica?on,  design  and  delivery   of  system  projects   •  Manage  security  issues  including   authen?ca?on,  authoriza?on,   and  secure  access   •  Likes  Care  Bears  
4. What  are  we  talking  about?   •  Meta-­‐Splunk  ==  Meta-­‐talk   –  Meta?  Say  what  now?   •  How  Splunk  evolved  over  the  past  4  years   –  Ini?al  deployment  as  opera?onal  tool   –  What  are  we  are  doing  with  Splunk   –  How  to  provide  solu?ons  for  diverse  customer  groups   –  How  to  maintain  a  successful  an  manageable   deployment   –  How  to  provide  Splunk  as  a  service  
5. Ini?al  Deployment   •  Single  server  deployment   •  Mirrored  for  redundancy   •  Syslog  based   •  Very  few  agents   •  Was  the  appropriate   architecture  for  20  GB   •  Splunk  is  rooted  and   takes  off   Phase 1 Applications which receive Syslogs Network Devices: Routers Switches Firewalls Primary syslog destinations Linux, Unix, Solaris Servers Network Appliances: TACACS VPN SourceFire StealthWatch Syslog-ng-1 Syslog-ng-2 Splunk-2 Splunk-1 Syslog-ng Various destiniations Pages Emails
6. Current  Deployment   •  Separate  search  heads   •  Indexing  pool   •  “Splunk  Collectors”   •  Agents  on  all  servers*   •  Reduce  UDP  traffic   •  Horizontal  scaling     •  Site  redundancy   •  Separate  license  pools   User Searches Network Devices: Routers Switches Firewalls Network Appliances: TACACS VPN SourceFire StealthWatch Splunk Forwarders: Linux Solaris Windows Servers Phase 4, version 2 splunkcollector1 splunkcollector1 Various destinations: • Qradar • LCE • Nessus Pages Emails Filtered Selective unfiltered Unfiltered Opsware TCP UDP splunkindex1 splunksearch1 splunksearch2 splunksearch1 splunksearch2 splunkindex2 splunkindex3 splunkindex4 splunkindex1 splunkindex2 splunkindex3 splunkindex4
7. From  a  customer  perspec?ve   •  Simple  setup   –  Point  into  a  cloud   –  Search  from  your  own   search  head   •  “Set  it  and  forget  it”   •  Start  looking  at  data   right  away   •  Manage  your  own   searches,  dashboards,   reports,  and  apps   User Searches Network Devices: Routers Switches Firewalls Network Appliances: TACACS VPN SourceFire StealthWatch Splunk Forwarders: Linux Solaris Windows Servers Phase 4, version 2 Pages Emails Splunk- search-1 Splunk- search-2 primary splunk indexers and syslog secondary splunk indexers and syslog
8. What  are  we  looking  at?   •  2,300+  Unix,  Windows,  and  other  servers   –  Infrastructure  hosts   –  Research  Compu?ng  Cluster   •  1,500  Cisco  routers  and  switches   •  2,600  Wireless  Access  Points   •  300  Firewall  contexts  and  ASAs   •  TACACS  /  RADIUS  authen?ca?on  logs   •  Command  script  outputs   •  Applica?on  logs  
9. “We  have  chosen  Splunk  as  our   pla-orm”   •  Where  did  it  come  from?   –  Splunk  has  been  running  for  3  years   –  Has  been  widely  accepted  by  several  groups   –  Recent  merger   –  Tools  discussions   •  What  does  it  mean?   –  We  are  keeping  Splunk   –  More  groups  are  interested  in  using  our  infrastructure   •  Did  it  s?ck?  
10. Why  is  it  hard  to  say  what  we  are   doing  with  Splunk?   •  Systems  monitoring   –  Unix  App   –  Windows  App   –  SNMP  Trap  processor   –  File  monitoring   –  Alert  views     •  Engineering  tool   –  Triage     –  Troubleshoo?ng     –  Forensics  tool   •  Custom  reports   –  DNS  /  DHCP  stats   –  Wireless  client  u?liza?on   –  Server  access  tracking   –  DCMA   –  VPN  Usage  /  Loca?on  data   –  Email  sta?s?cs     •  Replace  Oracle  DBs   –  New  event  data  store   •  Results  at  the  speed  of  life   •  Portal  to  all  logs   –  Applica?on  troubleshoo?ng   –  Applica?on  trending   –  Security     –  API  for  custom  web  interface   •  No?fica?on  delivery   –  PDFs   –  Text  /  Tables   –  SMSs   •  Custom  Apps   –  Config   –  Collector   –  Manager  of  Managers   •  SplunkBase  Apps   –  Deployment  Monitor   –  Unix   –  Windows   –  Google  Maps   –  Web  Page  monitor   –  VMWare  
11. Why  are  we  doing  so  many  things  with   Splunk?   •  Splunk  answers  who,  what,  where,  and  when.      …and  WHY?       •  It’s  a  one  stop  shop     •  We  resolve  issues  much  faster   •  Provides  endless  opportuni?es  to  build   applica?ons  and  reports   •  Splunk  allows  us  to  offer  mul?ple  APIs  to  data   •  It  keeps  us  really  busy!  
12. How  do  we  provide  Splunk  solu?ons   for  everyone?   •  Mul?ple  authen?ca?on  mechanisms   •  Roles  for  various  groups   •  Management  of  Splunk  agents  for  servers  that   we  do  not  administer   •  Splunk  Applica?on  development   •  All  of  the  above  allow  us  to  offer  Splunk  as  a   service  
13. Splunk  as  a  service   The  work:   •  Install  hardware   •  Maintain  hardware   •  Install  OS   •  Maintain  OS   •  Install  Splunk   •  Configure  Splunk   •  Install  agents   •  Configure  agents   The  fun  stuff:   •  Search  your  logs   •  Make  pre1y  graphs   •  Build  reports   •  Create  alerts   •  Develop  apps  
14. Who  uses  our  service?   •  Current  customers:   –  Networking   –  Systems   –  Research  Compu?ng   –  Security   –  Web  /  Applica?on  developers   –  Library   –  VPN  admins  and  customers   –  DBAs   •  New  customer  surges  keep  us  extremely  busy.   •  A  queue  is  forming  
15. How  to  maintain  a  successful  and   manageable  deployment   •  Architecture   –  Horizontal  scaling   •  DevOps     –  Version  control   –  Configura?on  Management   –  Deployment   •  Licensing   –  Is  oGen  difficult  to  determine   •  Compliance  and  Privacy  concerns   –  Data  and  Role  separa?on  based  on  indexes   •  Log  rota?on  policies   –  90  Days  max  for  non-­‐summary  indexes   –  Summary  indexes  have  size  restraints  
16. Real  life  examples   Opera8onal   •  Intern  discovers  clear  text   passwords   •  DNS  query  logs   •  Stolen  laptops   •  Provide  data  to  the  Office  of   General  Counsel   •  NTP  issues   Pla;orm   •  Poten?al  DMCA  viola?ons   •  Registra?on  portal   •  Within  our  Apps   –  Track  A  Mac   –  WiSM  IP  satura?on  (Expect   scripts)   –  VPN  User  Map   –  Unix  app  (new  skin)   –  MTRG  replacement   –  Event  tables  
17. NTP  -­‐  This  happened  Tuesday…  
18. NTP  -­‐  This  happened  last  Thursday!  
19. NTP  -­‐  This  happened  last  Thursday!  
20. MoM  –  Fault  Management  
21. MOM  –  Forms  for  logs  
22. MoM  –  Fault  Management  
23. Track  A  Mac  
24. WiSM  IP  satura?on  
25. WiSM  IP  satura?on  
26. MTRG  replacement   MRTG  –  legacy  non-­‐RRD   Splunk  
27. Unix  app  (new  skin)  
28. Unix  app  (new  skin)  
29. Ac?ve  VPN  User  Map  
30. Ac?ve  VPN  User  Map  
31. “Splunk  as  our  pla-orm”   •  We  like  that  Splunk’s  func?ons  cannot  be   summed  up  in  a  single  phrase   – It  does  almost  everything   •  “We  can  Splunk  that”   – Developers  and  Engineers  feedback     •  The  value  of  Splunk  is  recognized  at  all  levels   – It  means  something  on  all  tech  ?ers  and  to  the   highest  levels  of  Management  
32. Current  projects   •  Splunk  as  a  Service   •  Manager  of  Managers   – Big  project,  s?ll  working  on  it   – As  new  problems  are  discovered,  we  quickly  add   more  views   •  Deployment  Server   •  Execu?ve  Dashboards     •  Encrypted  agent  traffic