Big Data Log Analytics and IT Operations Analytics (ITOA) with Splunk, TIBCO LogLogic and the Open Source “ELK Stack” Kai Wähner [email protected] @KaiWaehner www.kai-waehner.de LinkedIn / Xing  Please connect!

3 Rapid Growth in Machine Big Data Challenges IT © Copyright 2000-2015 TIBCO Software Inc. ?

4 When a Threat or Opportunity is Discovered in Your Logs… © Copyright 2000-2015 TIBCO Software Inc. • Can you issue a single search across all your machine data- regardless of source or type? • Can you set an alert that would trigger from any source in your enterprise? • What about „predictive monitoring“? • Are you storing all of your logs for enough time to answer the question “What happened?” a week from now? How about a year from now?

Key Messages – Log Analytics enables IT Operations Analytics for Machine Data – Correlation of Events is the Key for Added Business Value – Log Management is complementary to other Big Data Components

Agenda – Real World Use Cases – Introduction to Log Analytics – Market Overview – Live Demo – Relation to other Big Data Components

Agenda – Real World Use Cases – Introduction to Log Analytics – Market Overview – Live Demo – Relation to other Big Data Components

8 © Copyright 2000-2015 TIBCO Software Inc. Real World Use Cases Infrastructure • Log Management – Applications – SOA – Microservices – SaaS • Transaction Tracing • Root Cause Analysis • Visual Analytics on Machine Data Competitive Undermining • Filtering / Cost Avoidance Solution IT Operations • Troubleshooting Connectivity • Outage Troubleshooting • Application Monitoring / Tracking • Service Level Confirmation for IT Outsourcing Security • Centralized Log/Event Management Platform • Security • Fraud Detection Compliance • PCI Compliance • Retention Compliance • Service Level Confirmation for IT Outsourcing

Agenda – Real World Use Cases – Introduction to Log Analytics – Market Overview – Live Demo – Relation to other Big Data Components

Service Level Assurance Compliance Security Business Activity IT Operations Problem: Point to Point Architecture Cloud Same information being stored multiples times = more HW, more cost Redundant solutions create network burden by collecting same data multiple times © Copyright 2000-2015 TIBCO Software Inc.

Solution: Operation Intelligence Platform Cloud Cloud © Copyright 2000-2015 TIBCO Software Inc. Service Level Assurance Compliance Security Business Activity IT Operations Log Management

Key Benefits of the Operational Intelligence Platform © Copyright 2000-2015 TIBCO Software Inc. SLA Compliance Security Identity IT Ops LogLogic Cloud Cloud

How an Operation Intelligence Platform Works © Copyright 2000-2015 TIBCO Software Inc. Collect Data from Any Source Device Logs Web Logs Application & DB Logs Configuration Files OS Metrics Sensor Data INGEST

How an Operation Intelligence Platform Works © Copyright 2000-2015 TIBCO Software Inc. Collect Data from Any Source Device Logs Web Logs Application & DB Logs Configuration Files OS Metrics Sensor Data Make Unstructured Data Usable Normalize Enrich Transform Index Aggregate INGEST OPERATIONALIZE

How an Operation Intelligence Platform Works © Copyright 2000-2015 TIBCO Software Inc. INGEST OPERATIONALIZE ANALYZE Collect Data from Any Source Device Logs Web Logs Application & DB Logs Configuration Files OS Metrics Sensor Data Make Unstructured Data Usable Normalize Enrich Transform Index Aggregate Gain Actionable Insight Search Report Alert Correlate Visualize

34 Characteristics of Log Management Solutions © Copyright 2000-2015 TIBCO Software Inc. Data Sources – Log information (standard protocols like TCP, UDP, File, Syslog) – All events (logs, messaging, streams, ...) – Extendable plugins (connectors, SDK, API) Features – Collect, parse, correlate, search, report, forward, etc. – Store and index – Query Lanaguage (SQL, Custom)  sliding windows, correlations, etc. – Retention – Compliance Templates Frequency – Historical data – Near Real Time Processing (seconds or minutes) Deployment Options – On-premise vs. Cloud (SaaS) – Open Source vs. Commercial – Software vs. Hardware Appliance Pricing – Free (open source) vs. CPU-based vs. Volume-based  Be careful here: IoT... Data grows exponentialy

Agenda – Real World Use Cases – Introduction to Log Analytics – Market Overview – Live Demo – Relation to other Big Data Components

36 Security information and event management (SIEM) © Copyright 2000-2015 TIBCO Software Inc. SIEM is a specific part of Log Analytics focusing on Security: • Threat management: Early detection of targeted attacks and data breaches • Compliance: Collect, store, analyze and report on log data for incident response, forensics and regulatory compliance • Aggregates event data produced by security devices, network infrastructures, systems and applications Log Analytics handles all kinds of use cases, not focusing on security. http://www.gartner.com/document/3097022 https://www-01.ibm.com/marketing/iwm/dre/signup?source=swg-WW_Security_Organic&S_PKG=ov37658&cm_mmc=Blog_SI-_-Sec_Int-_-Organic-_-IBM-is-a-leader-again-in-2015-gartner-magic-quadrant-for-SIEM SIEM is out-of-scope for this presentation!

37 Market Analysis * Market size data from various sources (sources in notes) Rapidly Emerging and Evolving, Encompasses Many Segments Traditional: Log Management, IT Operations Monitoring (ITOM), Security (SIEM) Current: IT Operations Analytics (ITOA), Application Performance Management (APM) Future: DevOps & Continuous Improvement Segment CAGR Incumbents Challengers Log Management 15% Splunk, TIBCO LogLogic, etc. Open Source (Graylog, “ELK Stack”) SIEM RSA, ArcSight, LogRhythm Splunk, MSSPs (Managed Security Service Provider) ITOA (1.6B) 100% TIBCO Unity, Splunk, SumoLogic, AppDynamics, NewRelic APM (2.9B) 10% AppDynamics, NewRelic ITOM (19B) 4% IBM, CA, BMC, MS, HP AppDynamics, NewRelic, Chef, Puppet, Docker, CloudFoundry (2.9B)

38 Alternatives for Log Analytics Time to Market Log Analytics Product Middleware Suite (includes Log Analytics Product) Slow Fast Log Analytics Framework Includes Includes © Copyright 2000-2015 TIBCO Software Inc.

39 Alternatives for Log Management © Copyright 2000-2015 TIBCO Software Inc. Open Source Closed Source SaaS On Premise (no complete list)

40 Alternatives for Log Management © Copyright 2000-2015 TIBCO Software Inc. Open Source Closed Source SaaS On Premise (no complete list) Open Source Framework

41 Alternatives for Log Analytics Time to Market Log Analytics Product Middleware Suite (includes Log Analytics Product) Slow Fast Log Analytics Framework © Copyright 2000-2015 TIBCO Software Inc. Library (Java, .NET, Python) Operators (Collect, Filter, Sort, Aggregate, Alert) Scalability (Horizontal and Vertical, Fail Over) Connectivity (Standards, Technologies, Products) User Interface (Basic Monitoring and Reporting)

42 ELK Stack (Logstash, Elasticsearch, Kibana) © Copyright 2000-2015 TIBCO Software Inc. Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing Facts • Combination of Open Source Frameworks – Complex setup and usage (coding and configuration) • Targeted for developers – Mainly focused on helping developers detect and fix errors in their apps – Entirely open source, i.e. free to use – Commerical support available – Combination of different mature frameworks • Less enterprise-focused – Very basic user interface – Based on ElasticSearch, Logstash and Kibana – Plenty of connectors + easy to extend (with coding) – Missing extensive reporting and analytics

43 graylog © Copyright 2000-2015 TIBCO Software Inc. Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing Facts • Combination of Open Source Frameworks – Complex setup and usage (coding and configuration) • Targeted for developers – Mainly focused on helping developers detect and fix errors in their apps – Entirely open source, i.e. free to use – Commerical support available – Young solution (1.0 GA in 2015) – not as mature as others yet • Less enterprise-focused – Very basic user interface – Based on MongoDB, ElasticSearch and Apache Kafka – Marketplace for connectors + easy to extend (with coding) – Missing extensive reporting and analytics

44 Alternatives for Log Management © Copyright 2000-2015 TIBCO Software Inc. Open Source Closed Source SaaS On Premise (no complete list) SaaS Cloud Service

45 Alternatives for Log Analytics Time to Market Log Analytics Product Middleware Suite (includes Log Analytics Product) Slow Fast Log Analytics Framework © Copyright 2000-2015 TIBCO Software Inc. Library Operators Scalability Connectivity User Interface Visual Configuration (Analysis, Correlation, Alerting) Simulation (Feed Testing, Test Generation) User Interface (Advanced Monitoring, Reporting, Analytics) Maturity (product, 24h support, consulting)

46 papertrail © Copyright 2000-2015 TIBCO Software Inc. Facts • Easy setup and very simple to use • Targeted for developers – „Very small“ free version available (100MB/month) – Cheap pricing, e.g. 1GB/month: 5 USD; 1000GB/month: 875 USD • Less enterprise-focused – Stripped down and basic log analyzer – Mostly text-based – User interface is very similar to looking at a log on your machine – No advanced integrations, predictive or reporting capabilities • SaaS – Upload (masses of) data to the cloud – Worse latency than on-premise solutions – Efforts to anonymize sensitive data Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing

47 loggly © Copyright 2000-2015 TIBCO Software Inc. Facts • Easy setup and very simple to use – Custom performance and DevOps dashboards • Targeted for developers and DevOps – Pricing from 50 USD to some thousand USD – Feature-limited free version available (200MB/day) • Less enterprise-focused – Focus especially on logs from application servers – Anything beyond that has to be built – Find and fix operational problems – Primary use cases are for troubleshooting / customer support scenarios • SaaS – Upload (masses of) data to the cloud – Worse latency than on-premise solutions – Efforts to anonymize sensitive data Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing

48 sumologic © Copyright 2000-2015 TIBCO Software Inc. Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing Facts • Easy setup and simple to use • Targeted for developer, security teams, business – Pricing from 90 USD to some thousand USD – Feature-limited free version available (500MB/day) • Most enterprise-focused SaaS product – Founded as „Splunk for the Cloud“ – Most feature-rich SaaS solution – Many features of „enterprise grade solutions“ • SaaS – Upload (masses of) data to the cloud – Worse latency than on-premise solutions – Efforts to anonymize sensitive data

49 Alternatives for Log Management © Copyright 2000-2015 TIBCO Software Inc. Open Source Closed Source SaaS On Premise (no complete list) Enterprise Product

50 Splunk © Copyright 2000-2015 TIBCO Software Inc. Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing Facts • Complex setup (especially for larger scale) • Simple to use for the end user • Targeted for all use cases (including SIEM) – Not just for log files, but also other events / messaging – „Enterprise Pricing“ - Very High pricing (for medium and high volume) – No access to your data if limit is reached! (contrary to other vendors) • Enterprise Class – Market leader – Most feature-rich solution – Available as SaaS offering – Moving into ITOA market – No hardware appliance (just via partner „SBOX“) – Just log analytics, no complete middleware suite

51 Alternatives for Log Analytics Time to Market Log Analytics Product Middleware Suite (includes Log Analytics Product) Slow Fast Log Analytics Framework © Copyright 2000-2015 TIBCO Software Inc. Library Operators Scalability Connectivity User Interface Visual Configuration Simulation Advanced User Interface Maturity Out-of-the-Box Integration and Support (Messaging, ESB, MDM, etc.)

52 IBM QRadar © Copyright 2000-2015 TIBCO Software Inc. Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing Facts • Complex setup • Simple to use for the end user • Targeted for all use cases (including SIEM) – Not just for log files, but also other events / messaging – „Enterprise Pricing“ - High pricing (for medium and high volume) • Enterprise Class – Part of a complete middlware suite – Very feature-rich solution – Available as SaaS offering – Available as hardware appliance – Moving into ITOA market

53 TIBCO LogLogic © Copyright 2000-2015 TIBCO Software Inc. Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing Facts • Easy setup (small and large scale) • Simple to use for the end user – Powerful user interface – Not as powerful as Splunk or IBM QRadar • Targeted for all use cases – Not just for log files, but also other events / messaging – „Enterprise Pricing“ - Low costs compared to competitors – „Always on“ – even after limit is reached • Enterprise Class – Part of a complete middlware suite – Most advanced analytics (via TIBCO Spotfire add-on) – Available as hardware appliance – Ready for ITOA (via TIBCO LogLogic Unity)

54 Spoilt for Choice? © Copyright 2000-2015 TIBCO Software Inc. Does it make sense to combine different Log Analytics solutions?

55 Example: TIBCO LogLogic - A Splunk Management Solution © Copyright 2000-2015 TIBCO Software Inc. http://www.tibco.de/assets/blt0da0bc2ea7d5b9b7/solution-brief-tibco-loglogic-splunk-management-solution.pdf

56 Conclusion - Market Analysis © Copyright 2000-2015 TIBCO Software Inc. Log Management • SaaS  Easy to setup and use, but cloud cons (not flexible, public cloud) • Open Source  Free and extendable, but coding / config instead of tooling • Enterprise  Most feature-rich and powerful tooling, but more expensive IT Operations Analytics (ITOA) • Enterprise vendors entering this market these days – Extending existing solutions • Focus on complex correlations, real time processing, predictive monitoring

57 Market Analysis * Market size data from various sources (sources in notes) Rapidly Emerging and Evolving, Encompasses Many Segments Traditional: Log Management, IT Operations Monitoring (ITOM), Security (SIEM) Current: IT Operations Analytics (ITOA), Application Performance Management (APM) Future: DevOps & Continuous Improvement Segment CAGR Incumbents Challengers Log Management 15% Splunk, TIBCO LogLogic, etc. Open Source (Graylog, “ELK Stack”) SIEM RSA, ArcSight, LogRhythm Splunk, MSSPs (Managed Security Service Provider) ITOA (1.6B) 100% TIBCO Unity, Splunk, SumoLogic, AppDynamics, NewRelic APM (2.9B) 10% AppDynamics, NewRelic ITOM (19B) 4% IBM, CA, BMC, MS, HP AppDynamics, NewRelic, Chef, Puppet, Docker, CloudFoundry (2.9B)

58 IT Operations Analytics (ITOA) © Copyright 2000-2015 TIBCO Software Inc. http://www.evolven.com/blog/gartner-analysts-have-high-expectations-for-it-operations-analytics.html

Agenda – Real World Use Cases – Introduction to Log Analytics – Market Overview – Live Demo – Relation to other Big Data Components

Papertrail (SaaS), ELK Stack (Open Source) and TIBCO LogLogic / Unity (Enterprise) in Action… Live Demo

Agenda – Real World Use Cases – Introduction to Log Analytics – Market Overview – Live Demo – Relation to other Big Data Components

64 When to use Log Analytics Time of Action Historical Data Near Real Time Real Time Predictive IT Operations Analytics (ITOA) Log Management Data Warehouse Streaming Analytics Data Discovery Hadoop (Variety of different Frameworks) Log Analytics

65 Relation to other Big Data Components © Copyright 2000-2015 TIBCO Software Inc. • Data Warehouse – Historical data – Only structured data – Reporting • Apache Hadoop – Historical and near real time data – All data – Storage and Analytics (e.g. MapReduce, Spark) • NoSQL – Specific Storage (graph, document, key/value, ...) – Search (e.g. ElasticSearch) • Stream Processing – Especially real time data • Predictive Analytics – R, Machine Learning, SAS, etc. – Combined with the others! Log Analytics Forward Forward Parse, Filter, Structure, Forward Parse, Filter, Structure, Forward Parse, Filter, Structure, Forward

66 Log Management / ITOA vs. Hadoop and Log Collectors © Copyright 2000-2015 TIBCO Software Inc. Why not use just Hadoop? You can also store and analyze all data on its cluster! Why not just use Log Collectors and send data directly without Log Analytics “in the middle”? • In general: Fluentd, Logstash, • Hadoop specific: Apache Flume or Apache Kafka DIFFERENTIATORS OF LOG MANAGEMENT / IT OPERATIONS ANALYTICS • Integrated solution for data analysis (tooling, consulting, support) • Built exactly for these use cases (Log Management, ITOA) • Involves data indexing, data processing (querying) and data visualization by means of dashboards and other tools • Tooling for Easy-of-Use and Time-to-Market • Graphical user interface for operational intelligence

Did you get the Key Message?

– Log Analytics enables IT Operations Analytics for Machine Data – Correlation of Events is the Key for Added Business Value – Log Management is complementary to other Big Data Components Key Messages

Questions? Kai Wähner [email protected] @KaiWaehner www.kai-waehner.de LinkedIn / Xing  Please connect!