Share
Embed
Transcript
32 slides
-
A li1le about us James Donn Sr. Network
and Systems Management Engineer • Design and implement enterprise level monitoring systems and tools • Deliver solu?ons for log management and audi?ng • Develop custom monitoring solu?ons • Part ?me Sys Admin Tim Hartmann Sr. Network and Services Engineer • Unix installa?on, configura?on, opera?ons, and maintenance of soGware and networked servers • Systems diagnos?cs and troubleshoo?ng • Specifica?on, design and delivery of system projects • Manage security issues including authen?ca?on, authoriza?on, and secure access • Likes Care Bears -
What are we talking about? • Meta-‐Splunk == Meta-‐talk
– Meta? Say what now? • How Splunk evolved over the past 4 years – Ini?al deployment as opera?onal tool – What are we are doing with Splunk – How to provide solu?ons for diverse customer groups – How to maintain a successful an manageable deployment – How to provide Splunk as a service -
Ini?al Deployment • Single server deployment • Mirrored
for redundancy • Syslog based • Very few agents • Was the appropriate architecture for 20 GB • Splunk is rooted and takes off Phase 1 Applications which receive Syslogs Network Devices: Routers Switches Firewalls Primary syslog destinations Linux, Unix, Solaris Servers Network Appliances: TACACS VPN SourceFire StealthWatch Syslog-ng-1 Syslog-ng-2 Splunk-2 Splunk-1 Syslog-ng Various destiniations Pages Emails -
Current Deployment • Separate search heads • Indexing
pool • “Splunk Collectors” • Agents on all servers* • Reduce UDP traffic • Horizontal scaling • Site redundancy • Separate license pools User Searches Network Devices: Routers Switches Firewalls Network Appliances: TACACS VPN SourceFire StealthWatch Splunk Forwarders: Linux Solaris Windows Servers Phase 4, version 2 splunkcollector1 splunkcollector1 Various destinations: • Qradar • LCE • Nessus Pages Emails Filtered Selective unfiltered Unfiltered Opsware TCP UDP splunkindex1 splunksearch1 splunksearch2 splunksearch1 splunksearch2 splunkindex2 splunkindex3 splunkindex4 splunkindex1 splunkindex2 splunkindex3 splunkindex4 -
From a customer perspec?ve • Simple setup –
Point into a cloud – Search from your own search head • “Set it and forget it” • Start looking at data right away • Manage your own searches, dashboards, reports, and apps User Searches Network Devices: Routers Switches Firewalls Network Appliances: TACACS VPN SourceFire StealthWatch Splunk Forwarders: Linux Solaris Windows Servers Phase 4, version 2 Pages Emails Splunk- search-1 Splunk- search-2 primary splunk indexers and syslog secondary splunk indexers and syslog -
What are we looking at? • 2,300+ Unix, Windows,
and other servers – Infrastructure hosts – Research Compu?ng Cluster • 1,500 Cisco routers and switches • 2,600 Wireless Access Points • 300 Firewall contexts and ASAs • TACACS / RADIUS authen?ca?on logs • Command script outputs • Applica?on logs -
“We have chosen Splunk as our pla-orm” •
Where did it come from? – Splunk has been running for 3 years – Has been widely accepted by several groups – Recent merger – Tools discussions • What does it mean? – We are keeping Splunk – More groups are interested in using our infrastructure • Did it s?ck? -
Why is it hard to say what we are
doing with Splunk? • Systems monitoring – Unix App – Windows App – SNMP Trap processor – File monitoring – Alert views • Engineering tool – Triage – Troubleshoo?ng – Forensics tool • Custom reports – DNS / DHCP stats – Wireless client u?liza?on – Server access tracking – DCMA – VPN Usage / Loca?on data – Email sta?s?cs • Replace Oracle DBs – New event data store • Results at the speed of life • Portal to all logs – Applica?on troubleshoo?ng – Applica?on trending – Security – API for custom web interface • No?fica?on delivery – PDFs – Text / Tables – SMSs • Custom Apps – Config – Collector – Manager of Managers • SplunkBase Apps – Deployment Monitor – Unix – Windows – Google Maps – Web Page monitor – VMWare -
Why are we doing so many things with Splunk?
• Splunk answers who, what, where, and when. …and WHY? • It’s a one stop shop • We resolve issues much faster • Provides endless opportuni?es to build applica?ons and reports • Splunk allows us to offer mul?ple APIs to data • It keeps us really busy! -
How do we provide Splunk solu?ons for everyone?
• Mul?ple authen?ca?on mechanisms • Roles for various groups • Management of Splunk agents for servers that we do not administer • Splunk Applica?on development • All of the above allow us to offer Splunk as a service -
Splunk as a service The work: • Install
hardware • Maintain hardware • Install OS • Maintain OS • Install Splunk • Configure Splunk • Install agents • Configure agents The fun stuff: • Search your logs • Make pre1y graphs • Build reports • Create alerts • Develop apps -
Who uses our service? • Current customers: –
Networking – Systems – Research Compu?ng – Security – Web / Applica?on developers – Library – VPN admins and customers – DBAs • New customer surges keep us extremely busy. • A queue is forming -
How to maintain a successful and manageable deployment
• Architecture – Horizontal scaling • DevOps – Version control – Configura?on Management – Deployment • Licensing – Is oGen difficult to determine • Compliance and Privacy concerns – Data and Role separa?on based on indexes • Log rota?on policies – 90 Days max for non-‐summary indexes – Summary indexes have size restraints -
Real life examples Opera8onal • Intern discovers clear
text passwords • DNS query logs • Stolen laptops • Provide data to the Office of General Counsel • NTP issues Pla;orm • Poten?al DMCA viola?ons • Registra?on portal • Within our Apps – Track A Mac – WiSM IP satura?on (Expect scripts) – VPN User Map – Unix app (new skin) – MTRG replacement – Event tables -
“Splunk as our pla-orm” • We like that Splunk’s
func?ons cannot be summed up in a single phrase – It does almost everything • “We can Splunk that” – Developers and Engineers feedback • The value of Splunk is recognized at all levels – It means something on all tech ?ers and to the highest levels of Management -
Current projects • Splunk as a Service •
Manager of Managers – Big project, s?ll working on it – As new problems are discovered, we quickly add more views • Deployment Server • Execu?ve Dashboards • Encrypted agent traffic