Framework and Product Comparison for Big Data Log Analytics and ITOA

IT systems and applications generate more and more machine data due to millions of mobile devices, Internet of Things, social network users, and other new emerging technologies. However, organizations experience challenges when monitoring and managing their IT systems and technology infrastructure. They struggle with network and server monitoring/troubleshooting, security analysis, custom application monitoring and debugging, compliance standards, and others.
This session discusses how to solve the challenges of analyzing Terabytes and more of different log data to leverage the “digital business” – a term defined by Gartner and others to explain that IT is not just a tool to enable a business, but IT is the business.
The main part of the session compares different solutions for operational intelligence and log analytics to create “digital business”, such as Splunk, TIBCO LogLogic and the open source “ELK stack” (ElasticSearch, Logstash, Kibana).
A common use case will be demonstrated in a live demo: Monitoring, analyzing and correlating a complex E-Commerce transaction running through different custom applications such as a Java EE web application, an integration middleware and analytics processes.
The end of the session explains the distinction of the discussed solutions to Apache Hadoop, and how they can complement each other in a big data architecture.

Keywords:
Big Data, Log Management, Log Analytics, IT Operations Analytics, ITOA, Gartner, Forrester, SIEM, Security, Open Source, Cloud, SaaS, Middleware, Hardware Appliance, SOA, Microservices, Hadoop, Data Warehouse, Stream Processing, Event Processing, TIBCO, LogLogic, TIBCO Unity, Splunk, IBM, QRadar, Papertrail, Loggly, fluentd, sumologic, ELK Stack, Logstash, Elasticsearch, Kibana, Graylog, AppDynamics, New Relic

Transcript

1. Big Data Log Analytics and IT Operations Analytics (ITOA) with Splunk, TIBCO LogLogic and the Open Source “ELK Stack” Kai Wähner kwaehner@tibco.com @KaiWaehner www.kai-waehner.de LinkedIn / Xing  Please connect!
2. 3 Rapid Growth in Machine Big Data Challenges IT © Copyright 2000-2015 TIBCO Software Inc. ?
3. 4 When a Threat or Opportunity is Discovered in Your Logs… © Copyright 2000-2015 TIBCO Software Inc. • Can you issue a single search across all your machine data- regardless of source or type? • Can you set an alert that would trigger from any source in your enterprise? • What about „predictive monitoring“? • Are you storing all of your logs for enough time to answer the question “What happened?” a week from now? How about a year from now?
4. Key Messages – Log Analytics enables IT Operations Analytics for Machine Data – Correlation of Events is the Key for Added Business Value – Log Management is complementary to other Big Data Components
5. Agenda – Real World Use Cases – Introduction to Log Analytics – Market Overview – Live Demo – Relation to other Big Data Components
6. Agenda – Real World Use Cases – Introduction to Log Analytics – Market Overview – Live Demo – Relation to other Big Data Components
7. 8 © Copyright 2000-2015 TIBCO Software Inc. Real World Use Cases Infrastructure • Log Management – Applications – SOA – Microservices – SaaS • Transaction Tracing • Root Cause Analysis • Visual Analytics on Machine Data Competitive Undermining • Filtering / Cost Avoidance Solution IT Operations • Troubleshooting Connectivity • Outage Troubleshooting • Application Monitoring / Tracking • Service Level Confirmation for IT Outsourcing Security • Centralized Log/Event Management Platform • Security • Fraud Detection Compliance • PCI Compliance • Retention Compliance • Service Level Confirmation for IT Outsourcing
8. Agenda – Real World Use Cases – Introduction to Log Analytics – Market Overview – Live Demo – Relation to other Big Data Components
9. Service Level Assurance Compliance Security Business Activity IT Operations Problem: Point to Point Architecture Cloud Same information being stored multiples times = more HW, more cost Redundant solutions create network burden by collecting same data multiple times © Copyright 2000-2015 TIBCO Software Inc.
10. Solution: Operation Intelligence Platform Cloud Cloud © Copyright 2000-2015 TIBCO Software Inc. Service Level Assurance Compliance Security Business Activity IT Operations Log Management
11. Key Benefits of the Operational Intelligence Platform © Copyright 2000-2015 TIBCO Software Inc. SLA Compliance Security Identity IT Ops LogLogic Cloud Cloud
12. How an Operation Intelligence Platform Works © Copyright 2000-2015 TIBCO Software Inc. Collect Data from Any Source Device Logs Web Logs Application & DB Logs Configuration Files OS Metrics Sensor Data INGEST
13. How an Operation Intelligence Platform Works © Copyright 2000-2015 TIBCO Software Inc. Collect Data from Any Source Device Logs Web Logs Application & DB Logs Configuration Files OS Metrics Sensor Data Make Unstructured Data Usable Normalize Enrich Transform Index Aggregate INGEST OPERATIONALIZE
14. How an Operation Intelligence Platform Works © Copyright 2000-2015 TIBCO Software Inc. INGEST OPERATIONALIZE ANALYZE Collect Data from Any Source Device Logs Web Logs Application & DB Logs Configuration Files OS Metrics Sensor Data Make Unstructured Data Usable Normalize Enrich Transform Index Aggregate Gain Actionable Insight Search Report Alert Correlate Visualize
15. 34 Characteristics of Log Management Solutions © Copyright 2000-2015 TIBCO Software Inc. Data Sources – Log information (standard protocols like TCP, UDP, File, Syslog) – All events (logs, messaging, streams, ...) – Extendable plugins (connectors, SDK, API) Features – Collect, parse, correlate, search, report, forward, etc. – Store and index – Query Lanaguage (SQL, Custom)  sliding windows, correlations, etc. – Retention – Compliance Templates Frequency – Historical data – Near Real Time Processing (seconds or minutes) Deployment Options – On-premise vs. Cloud (SaaS) – Open Source vs. Commercial – Software vs. Hardware Appliance Pricing – Free (open source) vs. CPU-based vs. Volume-based  Be careful here: IoT... Data grows exponentialy
16. Agenda – Real World Use Cases – Introduction to Log Analytics – Market Overview – Live Demo – Relation to other Big Data Components
17. 36 Security information and event management (SIEM) © Copyright 2000-2015 TIBCO Software Inc. SIEM is a specific part of Log Analytics focusing on Security: • Threat management: Early detection of targeted attacks and data breaches • Compliance: Collect, store, analyze and report on log data for incident response, forensics and regulatory compliance • Aggregates event data produced by security devices, network infrastructures, systems and applications Log Analytics handles all kinds of use cases, not focusing on security. http://www.gartner.com/document/3097022 https://www-01.ibm.com/marketing/iwm/dre/signup?source=swg-WW_Security_Organic&S_PKG=ov37658&cm_mmc=Blog_SI-_-Sec_Int-_-Organic-_-IBM-is-a-leader-again-in-2015-gartner-magic-quadrant-for-SIEM SIEM is out-of-scope for this presentation!
18. 37 Market Analysis * Market size data from various sources (sources in notes) Rapidly Emerging and Evolving, Encompasses Many Segments Traditional: Log Management, IT Operations Monitoring (ITOM), Security (SIEM) Current: IT Operations Analytics (ITOA), Application Performance Management (APM) Future: DevOps & Continuous Improvement Segment CAGR Incumbents Challengers Log Management 15% Splunk, TIBCO LogLogic, etc. Open Source (Graylog, “ELK Stack”) SIEM RSA, ArcSight, LogRhythm Splunk, MSSPs (Managed Security Service Provider) ITOA (1.6B) 100% TIBCO Unity, Splunk, SumoLogic, AppDynamics, NewRelic APM (2.9B) 10% AppDynamics, NewRelic ITOM (19B) 4% IBM, CA, BMC, MS, HP AppDynamics, NewRelic, Chef, Puppet, Docker, CloudFoundry (2.9B)
19. 38 Alternatives for Log Analytics Time to Market Log Analytics Product Middleware Suite (includes Log Analytics Product) Slow Fast Log Analytics Framework Includes Includes © Copyright 2000-2015 TIBCO Software Inc.
20. 39 Alternatives for Log Management © Copyright 2000-2015 TIBCO Software Inc. Open Source Closed Source SaaS On Premise (no complete list)
21. 40 Alternatives for Log Management © Copyright 2000-2015 TIBCO Software Inc. Open Source Closed Source SaaS On Premise (no complete list) Open Source Framework
22. 41 Alternatives for Log Analytics Time to Market Log Analytics Product Middleware Suite (includes Log Analytics Product) Slow Fast Log Analytics Framework © Copyright 2000-2015 TIBCO Software Inc. Library (Java, .NET, Python) Operators (Collect, Filter, Sort, Aggregate, Alert) Scalability (Horizontal and Vertical, Fail Over) Connectivity (Standards, Technologies, Products) User Interface (Basic Monitoring and Reporting)
23. 42 ELK Stack (Logstash, Elasticsearch, Kibana) © Copyright 2000-2015 TIBCO Software Inc. Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing Facts • Combination of Open Source Frameworks – Complex setup and usage (coding and configuration) • Targeted for developers – Mainly focused on helping developers detect and fix errors in their apps – Entirely open source, i.e. free to use – Commerical support available – Combination of different mature frameworks • Less enterprise-focused – Very basic user interface – Based on ElasticSearch, Logstash and Kibana – Plenty of connectors + easy to extend (with coding) – Missing extensive reporting and analytics
24. 43 graylog © Copyright 2000-2015 TIBCO Software Inc. Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing Facts • Combination of Open Source Frameworks – Complex setup and usage (coding and configuration) • Targeted for developers – Mainly focused on helping developers detect and fix errors in their apps – Entirely open source, i.e. free to use – Commerical support available – Young solution (1.0 GA in 2015) – not as mature as others yet • Less enterprise-focused – Very basic user interface – Based on MongoDB, ElasticSearch and Apache Kafka – Marketplace for connectors + easy to extend (with coding) – Missing extensive reporting and analytics
25. 44 Alternatives for Log Management © Copyright 2000-2015 TIBCO Software Inc. Open Source Closed Source SaaS On Premise (no complete list) SaaS Cloud Service
26. 45 Alternatives for Log Analytics Time to Market Log Analytics Product Middleware Suite (includes Log Analytics Product) Slow Fast Log Analytics Framework © Copyright 2000-2015 TIBCO Software Inc. Library Operators Scalability Connectivity User Interface Visual Configuration (Analysis, Correlation, Alerting) Simulation (Feed Testing, Test Generation) User Interface (Advanced Monitoring, Reporting, Analytics) Maturity (product, 24h support, consulting)
27. 46 papertrail © Copyright 2000-2015 TIBCO Software Inc. Facts • Easy setup and very simple to use • Targeted for developers – „Very small“ free version available (100MB/month) – Cheap pricing, e.g. 1GB/month: 5 USD; 1000GB/month: 875 USD • Less enterprise-focused – Stripped down and basic log analyzer – Mostly text-based – User interface is very similar to looking at a log on your machine – No advanced integrations, predictive or reporting capabilities • SaaS – Upload (masses of) data to the cloud – Worse latency than on-premise solutions – Efforts to anonymize sensitive data Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing
28. 47 loggly © Copyright 2000-2015 TIBCO Software Inc. Facts • Easy setup and very simple to use – Custom performance and DevOps dashboards • Targeted for developers and DevOps – Pricing from 50 USD to some thousand USD – Feature-limited free version available (200MB/day) • Less enterprise-focused – Focus especially on logs from application servers – Anything beyond that has to be built – Find and fix operational problems – Primary use cases are for troubleshooting / customer support scenarios • SaaS – Upload (masses of) data to the cloud – Worse latency than on-premise solutions – Efforts to anonymize sensitive data Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing
29. 48 sumologic © Copyright 2000-2015 TIBCO Software Inc. Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing Facts • Easy setup and simple to use • Targeted for developer, security teams, business – Pricing from 90 USD to some thousand USD – Feature-limited free version available (500MB/day) • Most enterprise-focused SaaS product – Founded as „Splunk for the Cloud“ – Most feature-rich SaaS solution – Many features of „enterprise grade solutions“ • SaaS – Upload (masses of) data to the cloud – Worse latency than on-premise solutions – Efforts to anonymize sensitive data
30. 49 Alternatives for Log Management © Copyright 2000-2015 TIBCO Software Inc. Open Source Closed Source SaaS On Premise (no complete list) Enterprise Product
31. 50 Splunk © Copyright 2000-2015 TIBCO Software Inc. Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing Facts • Complex setup (especially for larger scale) • Simple to use for the end user • Targeted for all use cases (including SIEM) – Not just for log files, but also other events / messaging – „Enterprise Pricing“ - Very High pricing (for medium and high volume) – No access to your data if limit is reached! (contrary to other vendors) • Enterprise Class – Market leader – Most feature-rich solution – Available as SaaS offering – Moving into ITOA market – No hardware appliance (just via partner „SBOX“) – Just log analytics, no complete middleware suite
32. 51 Alternatives for Log Analytics Time to Market Log Analytics Product Middleware Suite (includes Log Analytics Product) Slow Fast Log Analytics Framework © Copyright 2000-2015 TIBCO Software Inc. Library Operators Scalability Connectivity User Interface Visual Configuration Simulation Advanced User Interface Maturity Out-of-the-Box Integration and Support (Messaging, ESB, MDM, etc.)
33. 52 IBM QRadar © Copyright 2000-2015 TIBCO Software Inc. Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing Facts • Complex setup • Simple to use for the end user • Targeted for all use cases (including SIEM) – Not just for log files, but also other events / messaging – „Enterprise Pricing“ - High pricing (for medium and high volume) • Enterprise Class – Part of a complete middlware suite – Very feature-rich solution – Available as SaaS offering – Available as hardware appliance – Moving into ITOA market
34. 53 TIBCO LogLogic © Copyright 2000-2015 TIBCO Software Inc. Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing Facts • Easy setup (small and large scale) • Simple to use for the end user – Powerful user interface – Not as powerful as Splunk or IBM QRadar • Targeted for all use cases – Not just for log files, but also other events / messaging – „Enterprise Pricing“ - Low costs compared to competitors – „Always on“ – even after limit is reached • Enterprise Class – Part of a complete middlware suite – Most advanced analytics (via TIBCO Spotfire add-on) – Available as hardware appliance – Ready for ITOA (via TIBCO LogLogic Unity)
35. 54 Spoilt for Choice? © Copyright 2000-2015 TIBCO Software Inc. Does it make sense to combine different Log Analytics solutions?
36. 55 Example: TIBCO LogLogic - A Splunk Management Solution © Copyright 2000-2015 TIBCO Software Inc. http://www.tibco.de/assets/blt0da0bc2ea7d5b9b7/solution-brief-tibco-loglogic-splunk-management-solution.pdf
37. 56 Conclusion - Market Analysis © Copyright 2000-2015 TIBCO Software Inc. Log Management • SaaS  Easy to setup and use, but cloud cons (not flexible, public cloud) • Open Source  Free and extendable, but coding / config instead of tooling • Enterprise  Most feature-rich and powerful tooling, but more expensive IT Operations Analytics (ITOA) • Enterprise vendors entering this market these days – Extending existing solutions • Focus on complex correlations, real time processing, predictive monitoring
38. 57 Market Analysis * Market size data from various sources (sources in notes) Rapidly Emerging and Evolving, Encompasses Many Segments Traditional: Log Management, IT Operations Monitoring (ITOM), Security (SIEM) Current: IT Operations Analytics (ITOA), Application Performance Management (APM) Future: DevOps & Continuous Improvement Segment CAGR Incumbents Challengers Log Management 15% Splunk, TIBCO LogLogic, etc. Open Source (Graylog, “ELK Stack”) SIEM RSA, ArcSight, LogRhythm Splunk, MSSPs (Managed Security Service Provider) ITOA (1.6B) 100% TIBCO Unity, Splunk, SumoLogic, AppDynamics, NewRelic APM (2.9B) 10% AppDynamics, NewRelic ITOM (19B) 4% IBM, CA, BMC, MS, HP AppDynamics, NewRelic, Chef, Puppet, Docker, CloudFoundry (2.9B)
39. 58 IT Operations Analytics (ITOA) © Copyright 2000-2015 TIBCO Software Inc. http://www.evolven.com/blog/gartner-analysts-have-high-expectations-for-it-operations-analytics.html
40. Agenda – Real World Use Cases – Introduction to Log Analytics – Market Overview – Live Demo – Relation to other Big Data Components
41. Papertrail (SaaS), ELK Stack (Open Source) and TIBCO LogLogic / Unity (Enterprise) in Action… Live Demo
42. Agenda – Real World Use Cases – Introduction to Log Analytics – Market Overview – Live Demo – Relation to other Big Data Components
43. 64 When to use Log Analytics Time of Action Historical Data Near Real Time Real Time Predictive IT Operations Analytics (ITOA) Log Management Data Warehouse Streaming Analytics Data Discovery Hadoop (Variety of different Frameworks) Log Analytics
44. 65 Relation to other Big Data Components © Copyright 2000-2015 TIBCO Software Inc. • Data Warehouse – Historical data – Only structured data – Reporting • Apache Hadoop – Historical and near real time data – All data – Storage and Analytics (e.g. MapReduce, Spark) • NoSQL – Specific Storage (graph, document, key/value, ...) – Search (e.g. ElasticSearch) • Stream Processing – Especially real time data • Predictive Analytics – R, Machine Learning, SAS, etc. – Combined with the others! Log Analytics Forward Forward Parse, Filter, Structure, Forward Parse, Filter, Structure, Forward Parse, Filter, Structure, Forward
45. 66 Log Management / ITOA vs. Hadoop and Log Collectors © Copyright 2000-2015 TIBCO Software Inc. Why not use just Hadoop? You can also store and analyze all data on its cluster! Why not just use Log Collectors and send data directly without Log Analytics “in the middle”? • In general: Fluentd, Logstash, • Hadoop specific: Apache Flume or Apache Kafka DIFFERENTIATORS OF LOG MANAGEMENT / IT OPERATIONS ANALYTICS • Integrated solution for data analysis (tooling, consulting, support) • Built exactly for these use cases (Log Management, ITOA) • Involves data indexing, data processing (querying) and data visualization by means of dashboards and other tools • Tooling for Easy-of-Use and Time-to-Market • Graphical user interface for operational intelligence
46. Did you get the Key Message?
47. – Log Analytics enables IT Operations Analytics for Machine Data – Correlation of Events is the Key for Added Business Value – Log Management is complementary to other Big Data Components Key Messages
48. Questions? Kai Wähner kwaehner@tibco.com @KaiWaehner www.kai-waehner.de LinkedIn / Xing  Please connect!