Monitor Microservices with Log Analytics - Comparison of Open Source Frameworks, SaaS and Enterprise Products

Log Analytics and Operational Intelligence for Distributed Microservices.

IT systems and applications generate more and more distributed machine data due to millions of mobile devices, Internet of Things, social network users, and other new emerging technologies. However, organizations experience challenges when monitoring and managing their IT systems and technology infrastructure. They struggle with distributed Microservices and Cloud architectures, custom application monitoring and debugging, network and server monitoring / troubleshooting, security analysis, compliance standards, and others.

This session discusses how to solve the challenges of monitoring and analyzing Terabytes and more of different distributed machine data to leverage the “digital business”. The main part of the session compares different open source frameworks and SaaS cloud solutions for Log Management and operational intelligence, such as Graylog , the “ELK stack”, Papertrail, Splunk or TIBCO LogLogic Unity). A live demo will demonstrate how to monitor and analyze distributed Microservices and sensor data from the “Internet of Things”.

The session also explains the distinction of the discussed solutions to other big data components such as Apache Hadoop, Data Warehouse or Machine Learning, and how they can complement each other in a big data architecture.

The session concludes with an outlook to the new, advanced concept of IT Operations Analytics (ITOA).

Presented at O'Reilly Software Architecture Conference in London UK 2016. #OreillySACon.

Keywords:
Log analytics, , Operational Intelligence, big data, machine data, IoT, Internet of Things, microservices, log management, SIEM, SOA, ITOA, analytics, Hadoop, spark, apache, open source ,TIBCO, LogLogic, Splunk, IBM, QRadar, Greylog, ELK Stack, ELK, Elasticsearch, Logstash, Kibana, Loggly, Papertrail, sumologic, event processing, streaming analytics, data discovery, visual analytics, data warehouse, live datamart, StreamBase, Apama, Infosphere, Oracle, Microsoft Azure, Amazon AWS

Transcript

1. Kai Wähner Technology Evangelist kontakt@kai-waehner.de LinkedIn @KaiWaehner www.kai-waehner.de O’Reilly Software Architecture Conference 2016 (London, UK) Log Analytics for Distributed Microservices
2. © Copyright 2000-2016 TIBCO Software Inc. Can you answer to these questions? • Are you storing all of your logs for enough time to answer the question “What happened?” a week from now? How about a year from now? • Can you issue a single search across all your machine data - regardless of source or type? • Can you set an alert that would trigger from any source in your enterprise? • Do you analyze and correlate all events in your distributed microservice architecture? • What about predictive monitoring?
3. © Copyright 2000-2016 TIBCO Software Inc. Key Takeaways • Log Analytics is needed to monitor distributed microservice architectures • Consolidation of broad range of events is key to enabling business insights • Log Analytics is complementary to other Big Data components
4. © Copyright 2000-2016 TIBCO Software Inc. Agenda • Distributed Microservice Log Events • Introduction to Log Analytics • Market Overview • Relation to other Big Data Components
5. © Copyright 2000-2016 TIBCO Software Inc. Agenda • Distributed Microservice Log Events • Introduction to Log Analytics • Market Overview • Relation to other Big Data Components
6. © Copyright 2000-2016 TIBCO Software Inc. Scenarios for Distributed Log Events Infrastructure • Log Management – Applications – SOA – Microservices – Cloud Platforms – SaaS • Transaction Tracing • Root Cause Analysis • Visual Analytics on Machine Data Competitive Undermining • Filtering / Cost Avoidance Solution IT Operations • Troubleshooting Connectivity • Outage Troubleshooting • Application Monitoring / Tracking • Service Level Confirmation for IT Outsourcing Security • Centralized Log / Event Management Platform • Security • Fraud Detection Compliance • PCI Compliance • Retention Compliance • Service Level Confirmation for IT Outsourcing
7. © Copyright 2000-2016 TIBCO Software Inc. Distributed Microservice Architecture http://blogs.gartner.com/gary-olliffe/2015/01/30/microservices-guts-on-the-outside/ ”That complexity has moved and […] increased [to] the outer architecture.”
8. © Copyright 2000-2016 TIBCO Software Inc. 12 Factor Apps for Cloud Native Microservices Codebase One codebase tracked in revision control, many deploys. Dependencies Explicitly declare and isolate dependencies. Config Store config in the environment. Backing Services Treat backing services as attached resources. Build, Release, Run Strictly separate build and run stages. Processes Execute the app as one or more stateless processes. Port Binding Export services via port binding. Concurrency Scale out via the process model. Disposability Maximize robustness with fast startup and graceful shutdown. Dev / Prod Parity Keep dev, staging, and prod as similar as possible. Logs Treat logs as event streams. Admin Processes Run admin/mgmt tasks as one-off processes. https://12factor.net/
9. © Copyright 2000-2016 TIBCO Software Inc. Some Cloud Platforms (PaaS) with Support for 12 Factor Apps With or without such a cloud platform, you need a way to aggregate and analyze distributed microservice logs. … to treat logs as event streams. ! !
10. © Copyright 2000-2016 TIBCO Software Inc. Agenda • Distributed Microservice Log Events • Introduction to Log Analytics • Market Overview • Relation to other Big Data Components
11. © Copyright 2000-2016 TIBCO Software Inc. Distributed Microservice Architecture http://blogs.gartner.com/gary-olliffe/2015/01/30/microservices-guts-on-the-outside/ Microservices means… - distributed services - distributed infrastructure - different technologies - containers and cloud platforms - distributed log messages - unstructured / semi-structured data Log Analytics
12. © Copyright 2000-2016 TIBCO Software Inc. Operational Intelligence Platform for Log Analytics Log Analytics Platform ü Centralize and Store of Record ü Search, Auto-id, Parsing, Correlation ü Forensics and Alerts ü Reports Engine Logs Application Logs Microservices Monitoring Configuration Messaging Web UI API Analysis Tools Data Discovery Streaming Analytics Live Visualization
13. © Copyright 2000-2016 TIBCO Software Inc. How an Operation Intelligence Platform Works INGEST OPERATIONALIZE ANALYZE Collect Data from Any Source Device Logs Web Logs Application & DB Logs Configuration Files OS Metrics Sensor Data Microservice Events Make Unstructured Data Usable Normalize Enrich Transform Index Aggregate Gain Actionable Insight Search Report Alert Correlate Visualize
14. © Copyright 2000-2016 TIBCO Software Inc. Log Analytics Example • May 2 23:06:14 app-1 login[5130]: pam_unix(login:auth): authentication failure; logname=LOGIN uid=0 euid=0 tty=tty1 ruser= rhost= user=timothy • "<13>Feb 5 08:34:55 10.92.2.188 MSWinEventLog 0 Security 106236353 Fri Feb 05 08:33:15 2010 529 Security SYSTEM User Failure Audit OHAEPHQDC009 Logon/Logoff Logon Failure: Reason: Unknown user name or bad password User Name: timothy Domain: Logon Type: 3 Logon Process: CISCO Authentication Package: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0 Workstation Name: CISCO Caller User Name: portal Caller Domain: CORP Caller Logon ID: (0x0,0x63194519) Caller Process ID: 2972 Transited Services: - Source Network Address: - Source Port: - 1679136992" • Jun 11 10:51:04 10.0.0.244 Jun 11 10: 51:42 1,06/11 10:51:42,0001a100200,TRAFFIC,start,24,06/11 10:51:15,10.0.0.101,10.0.0.246,0.0.0.0,0.0.0.0,timothy,,,dns,vsys1,l2-lan-trust,l2- lan-untrust,ethernet1/12,ethernet1/11,Forward to Timothy,06/11 10:51:42,2074963,1,54604,53,0,0,0x0,udp,allow,80,80,80,1,06/11 10:51:16,0,any,0 Source Type User User Name SRC User Unix timothy Windows timothy Firewall timothy Source Type User Unix timothy Windows timothy Firewall timothy • Unix • Windows • Firewall
15. © Copyright 2000-2016 TIBCO Software Inc. Characteristics of Log Management Solutions Data Sources • Log information (standard protocols like TCP, UDP, File, Syslog, JMS) • All events (logs, messaging, streams, ...) • Extendable plugins (connectors, SDK, API) Features • Collect, parse, correlate, search, report, forward, etc. • Store and index • Query Language (SQL, Custom) à sliding windows, correlations, etc. • Retention • Compliance Templates Frequency • Historical data • Near Real Time Processing (seconds or minutes) Deployment Options • On-premise vs. Cloud (SaaS) • Open Source vs. Commercial • Software vs. Hardware Appliance Pricing • Free (open source) vs. CPU-based vs. Volume-based à Be careful here: IoT... Data grows exponentially
16. © Copyright 2000-2016 TIBCO Software Inc. Agenda • Distributed Microservice Log Events • Introduction to Log Analytics • Market Overview • Relation to other Big Data Components
17. © Copyright 2000-2016 TIBCO Software Inc. Market Analysis Segment CAGR Incumbents Challengers Log Management 15% Splunk, TIBCO LogLogic, etc. Open Source (Graylog, “ELK Stack”) SIEM RSA, ArcSight, LogRhythm Splunk, MSSPs (Managed Security Service Provider) ITOA (1.6B) 100% TIBCO LogLogic, Splunk, SumoLogic, AppDynamics, NewRelic APM (2.9B) 10% AppDynamics, NewRelic ITOM (19B) 4% IBM, CA, BMC, MS, HP AppDynamics, NewRelic, Chef, Puppet, Docker, CloudFoundry Rapidly Emerging and Evolving, Encompasses Many Segments Traditional: Log Management, IT Operations Monitoring (ITOM), Security (SIEM) Current: IT Operations Analytics (ITOA), Application Performance Management (APM) Future: DevOps and Continuous Improvement (2.9B)
18. © Copyright 2000-2016 TIBCO Software Inc. Security information and event management (SIEM) SIEM is a specific part of Log Analytics focusing on Security. • Threat management: Early detection of targeted attacks and data breaches • Compliance: Collect, store, analyze and report on log data for incident response, forensics and regulatory compliance • Aggregates event data produced by security devices, network infrastructures, systems and applications Log Analytics handles all kinds of use cases, not focusing on security. http://www.gartner.com/document/3097022 https://www-01.ibm.com/marketing/iwm/dre/signup?source=swg-WW_Security_Organic&S_PKG=ov37658&cm_mmc=Blog_SI-_-Sec_Int-_-Organic-_-IBM-is-a-leader-again-in-2015-gartner-magic-quadrant-for-SIEM SIEM is out-of-scope for this presentation!
19. © Copyright 2000-2016 TIBCO Software Inc. Alternatives for Log Analytics Time to Market Log Analytics Product Middleware Suite (includes Log Analytics Product) Slow Fast Log Analytics Framework Includes Includes
20. © Copyright 2000-2016 TIBCO Software Inc. Alternatives for Log Management (no complete list) Open Source Closed Source SaaS On Premise
21. © Copyright 2000-2016 TIBCO Software Inc. “Cloud washing is the purposeful and sometimes deceptive attempt by a vendor to rebrand an old product or service by associating the buzzword ‘cloud’ with it [and offering it via a public cloud infrastructure].” On Premise vs. Cloud Washing vs. Cloud Native / SaaS http://searchcloudstorage.techtarget.com/definition/cloud-washing !
22. © Copyright 2000-2016 TIBCO Software Inc. Alternatives for Log Management (no complete list) Open Source Closed Source SaaS On Premise Open Source Framework
23. © Copyright 2000-2016 TIBCO Software Inc. Alternatives for Log Analytics Time to Market Log Analytics Product Middleware Suite (includes Log Analytics Product) Slow Fast Log Analytics Framework Library (Java, .NET, Python) Operators (Collect, Filter, Sort, Aggregate, Alert) Scalability (Horizontal and Vertical, Fail Over) Connectivity (Standards, Technologies, Products) User Interface (Basic Monitoring and Reporting)
24. © Copyright 2000-2016 TIBCO Software Inc. ELK Stack (Logstash, Elasticsearch, Kibana) Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing Facts Combination of Open Source Frameworks • Quick getting started for developers with ”Hello World” example • More complex Enterprise setup and usage (coding and configuration) • AWS offering available for Elastic and Kibana, not Logstash) Targeted for developers • Mainly focused on helping developers detect and fix errors in their apps • Entirely open source, i.e. free to use • Commercial support available • Combination of different mature frameworks Less enterprise-focused • Very basic user interface • Based on ElasticSearch, Logstash and Kibana • Plenty of connectors + easy to extend (with coding) • Sufficient reporting (i.e. dashboards), but missing visual analytics
25. © Copyright 2000-2016 TIBCO Software Inc. Live Demo ELK Stack (Open Source) in Action…
26. © Copyright 2000-2016 TIBCO Software Inc. graylog Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing Facts Combination of Open Source Frameworks • Quick getting started for developers with ”Hello World” example • More complex Enterprise setup and usage (coding and configuration) Targeted for developers • Mainly focused on helping developers detect and fix errors in their apps • Entirely open source, i.e. free to use • Commercial support available • Young solution (1.0 GA in 2015) – not as mature as others yet Less enterprise-focused • Very basic user interface • Based on MongoDB, ElasticSearch and Apache Kafka • Marketplace for connectors + easy to extend (with coding) • Missing extensive reporting and analytics
27. © Copyright 2000-2016 TIBCO Software Inc. Alternatives for Log Management (no complete list) Open Source Closed Source SaaS On Premise SaaS Cloud Service
28. © Copyright 2000-2016 TIBCO Software Inc. papertrail Facts Easy setup and very simple to use • Targeted for developers • „Very small“ free version available (100MB/month) • Cheap pricing, e.g. 1GB/month: 5 USD; 1000GB/month: 875 USD Less enterprise-focused • Stripped down and basic log analyzer • Mostly text-based • User interface is very similar to looking at a log on your machine • No advanced integrations, predictive or reporting capabilities SaaS • Upload (masses of) data to the cloud • Worse latency than on-premise solutions • Efforts to anonymize sensitive data Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing
29. © Copyright 2000-2016 TIBCO Software Inc. Live Demo Papertrail (SaaS) in Action…
30. © Copyright 2000-2016 TIBCO Software Inc. loggly 30 Facts Easy setup and very simple to use • Custom performance and DevOps dashboards Targeted for developers and DevOps • Pricing from 50 USD to some thousand USD • Feature-limited free version available (200MB/day) Less enterprise-focused • Focus especially on logs from application servers • Anything beyond that has to be built • Find and fix operational problems • Primary use cases are for troubleshooting / customer support scenarios SaaS • Upload (masses of) data to the cloud • Worse latency than on-premise solutions • Efforts to anonymize sensitive data Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing
31. © Copyright 2000-2016 TIBCO Software Inc. Alternatives for Log Analytics Time to Market Log Analytics Product Middleware Suite (includes Log Analytics Product) Slow Fast Log Analytics Framework Library Operators Scalability Connectivity User Interface Visual Configuration (Analysis, Correlation, Alerting) Simulation (Feed Testing, Test Generation) User Interface (Advanced Monitoring, Reporting, Analytics) Maturity (product, 24h support, consulting)
32. © Copyright 2000-2016 TIBCO Software Inc. sumologic Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing Facts • Easy setup and simple to use • Targeted for developer, security teams, business – Pricing from 90 USD to some thousand USD – Feature-limited free version available (500MB/day) • Most enterprise-focused SaaS product – Founded as „Splunk for the Cloud“ – Most feature-rich SaaS solution – Many features of „enterprise grade solutions“ • SaaS – Upload (masses of) data to the cloud – Worse latency than on-premise solutions – Efforts to anonymize sensitive data
33. © Copyright 2000-2016 TIBCO Software Inc. Alternatives for Log Management (no complete list) Open Source Closed Source SaaS On Premise Enterprise Product
34. © Copyright 2000-2016 TIBCO Software Inc. Splunk Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing Facts • Complex setup (especially for larger scale) – SaaS Offering for getting started quickly in the public cloud • Simple to use for the end user • Targeted for all use cases (including SIEM) – Not just for log files, but also other events / messaging – „Enterprise Pricing“ - Very High pricing (for medium and high volume) – No access to your data if limit is reached! (contrary to other vendors) • Enterprise Class – Market leader – Most feature-rich solution – Moving into ITOA market – No hardware appliance (just via partner „SBOX“) – Just log analytics, no complete middleware suite
35. © Copyright 2000-2016 TIBCO Software Inc. Alternatives for Log Analytics Time to Market Log Analytics Product Middleware Suite (includes Log Analytics Product) Slow Fast Log Analytics Framework Library Operators Scalability Connectivity User Interface Visual Configuration Simulation Advanced User Interface Maturity Out-of-the-Box Integration and Support (Messaging, ESB, MDM, etc.)
36. © Copyright 2000-2016 TIBCO Software Inc. IBM QRadar Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing Facts • Complex setup • Simple to use for the end user • Targeted for all use cases (including SIEM) – Not just for log files, but also other events / messaging – „Enterprise Pricing“ - High pricing (for medium and high volume) • Enterprise Class – Part of a complete middleware suite – Very feature-rich solution – Available as SaaS offering – Available as hardware appliance – Moving into ITOA market
37. © Copyright 2000-2016 TIBCO Software Inc. TIBCO LogLogic 37 © Copyright 2000-2015 TIBCO Characteristics • Data Sources • Features • Frequency • Deployment Options • Pricing Facts • Easy setup (small and large scale) • Simple to use for the end user – Powerful user interface – Not as powerful as Splunk or IBM QRadar • Targeted for all use cases – Not just for log files, but also other events / messaging – „Enterprise Pricing“ - Low costs compared to competitors – „Always on“ – even after limit is reached • Enterprise Class – Part of a complete middleware suite – Most advanced analytics (via TIBCO Spotfire add-on) – Available as hardware appliance
38. © Copyright 2000-2016 TIBCO Software Inc. Live Demo TIBCO LogLogic (Enterprise) in Action…
39. © Copyright 2000-2016 TIBCO Software Inc. Message Pattern Generation with TIBCO LogLogic Web UI Discover Unstructured Data à Generate Pattern à Validate à Apply Pattern for Structured Data
40. © Copyright 2000-2016 TIBCO Software Inc. Spoilt for Choice Does it make sense to combine different Log Analytics solutions?
41. © Copyright 2000-2016 TIBCO Software Inc. Example: TIBCO LogLogic à „A Splunk Management Solution“ http://www.tibco.de/assets/blt0da0bc2ea7d5b9b7/solution-brief-tibco-loglogic-splunk-management-solution.pdf
42. © Copyright 2000-2016 TIBCO Software Inc. Conclusion - Market Analysis Log Management • SaaS à Easy to setup and use, but cloud cons (not flexible, public cloud) • Open Source à Free and extendable, but coding / config instead of tooling • Enterprise à Most feature-rich and powerful tooling, but more expensive IT Operations Analytics (ITOA) • Enterprise vendors entering this market these days à Extending existing solutions • Focus on more complex correlations, real time processing, predictive monitoring
43. © Copyright 2000-2016 TIBCO Software Inc. Market Analysis Segment CAG R Incumbents Challengers Log Management 15% Splunk, TIBCO LogLogic, etc. Open Source (Graylog, “ELK Stack”) SIEM RSA, ArcSight, LogRhythm Splunk, MSSPs (Managed Security Service Provider) ITOA (1.6B) 100% TIBCO LogLogic, Splunk, SumoLogic, AppDynamics, NewRelic APM (2.9B) 10% AppDynamics, NewRelic ITOM (19B) 4% IBM, CA, BMC, MS, HP AppDynamics, NewRelic, Chef, Puppet, Docker, CloudFoundry (2.9B) Rapidly Emerging and Evolving, Encompasses Many Segments Traditional: Log Management, IT Operations Monitoring (ITOM), Security (SIEM) Current: IT Operations Analytics (ITOA), Application Performance Management (APM) Future: DevOps & Continuous Improvement
44. © Copyright 2000-2016 TIBCO Software Inc. Log Analytics is a very stable and established market. ITOA enhances Log Analytics to allow more powerful real time correlation.
45. © Copyright 2000-2016 TIBCO Software Inc. Agenda • Distributed Microservice Log Events • Introduction to Log Analytics • Market Overview • Relation to other Big Data Components
46. © Copyright 2000-2016 TIBCO Software Inc. When to use Log Analytics Time of Action Historical Data Near Real Time Real Time Predictive IT Operations Analytics (ITOA) Log Management Data Warehouse Streaming Analytics Data Discovery „Data Lake“ (Various Apache Hadoop Frameworks) Log Analytics Visual Real Time Analytics (There is some overlapping!)
47. © Copyright 2000-2016 TIBCO Software Inc. Streaming Analytics: Act on Critical Business Moments
48. © Copyright 2000-2016 TIBCO Software Inc. Streaming Analytics Voltage Temperature Vibration Device history Temporal analytic: “If vibration spike is followed by temp spike then voltage spike [within 4 hours] then flag high severity alert.”
49. © Copyright 2000-2016 TIBCO Software Inc. Live Demo Apache Flink (Open Source), StreamSets (SaaS) and TIBCO StreamBase / Live Datamart (Enterprise) in Action…
50. © Copyright 2000-2016 TIBCO Software Inc. Log Management / ITOA vs. Hadoop and Log Collectors Why not use just a Data Lake (Apache Hadoop)? You can also store and analyze all data on its cluster! Why not just use Log Collectors and forward data directly without Log Analytics “in the middle”? • In general: Fluentd, Logstash • Apache Hadoop specific: Apache Flume or Apache Kafka DIFFERENTIATORS OF LOG MANAGEMENT / IT OPERATIONS ANALYTICS • Integrated solution for data analysis (tooling, consulting, support) • Built exactly for these use cases (Log Management, ITOA) • Involves data indexing, data processing (querying) and data visualization by means of dashboards and other tools • Tooling for Ease-of-Use and Time-to-Market • Graphical user interface for operational intelligence • There is no “one size fits all” tool to solve all your problems
51. © Copyright 2000-2016 TIBCO Software Inc. Relation to other Big Data Components • Data Warehouse – Historical data – Only structured data – Reporting • Apache Hadoop – Historical and near real time data – All data – Storage and Analytics (e.g. MapReduce, Spark) • NoSQL – Specific Storage (graph, document, key/value, ...) – Search (e.g. ElasticSearch) • Stream Processing – Especially real time data • Predictive Analytics – R, Machine Learning, SAS, etc. – Combined with the others! Log Analytics Forward Forward Parse, Filter, Structure, Forward Parse, Filter, Structure, Forward Parse, Filter, Structure, Forward
52. © Copyright 2000-2016 TIBCO Software Inc. Trend: Machine Learning applied to Log Analytics “… when the log-data patterns cannot be precisely defined in advance, unsupervised and reinforcement learning may be appropriate [to find outliers or anomalies].” http://www.infoworld.com/article/2608064/big-data/big-data-log-analysis-thrives-on-machine-learning.html “… They combined the aggregation of log data, the metadata that is created any time IT systems are used, along with high-level analytics and machine learning tools … … give context to the ’needle in a haystack’ problem …” http://www.forbes.com/sites/benkepes/2015/03/27/using-log-data-and-machine-learning-to-weed-out-the-bad- guys
53. © Copyright 2000-2016 TIBCO Software Inc. Key Takeaways • Log Analytics is needed to monitor distributed microservice architectures • Consolidation of broad range of events is key to enabling business insights • Log Analytics is complementary to other Big Data components
54. Questions? Please contact me! Kai Waehner Technology Evangelist kontakt@kai-waehner.de @KaiWaehner www.kai-waehner.de LinkedIn