Efficient logs with Elastic Stack

As a developer you have to do with fails and unexpected behaviors. And what you want to do in such situations is to be highly efficient. Browsing through multiple log files, in different formats trying to match timestamps and so on is not efficient. Elastic Stack is the tool that will bring order to the chaos and help you react as quick as possible. But as any tool it can be used in many ways - we will dive deeper into this so that you know what are your options when required.

http://codingideas.blog/elastic

Transcript

1. 1 EFFICIENT LOGS, WITH ELASTIC STACK APRIL 2019
2. 2 2 AGENDA  VIEWING LOGS THE OLD WAY  ELASTIC STACK COMPONENTS:  FILEBEAT  LOGSTASH  ELASTICSTACK  KIBANA  DEMO + LOGGING PATTERS  Q&A
3. 3 3 3 WHAT IS THIS ABOUT “Progress isn't made by early risers. It's made by lazy men trying to find easier ways to do something.” ― Robert Heinlein (sci-fi writer) We will talk about efficiency fighting complexity. Let’s start with a quote.
4. 4 HOW DO YOU ANALYZE LOGS?
5. 5 5 5 VIEWING LOGS TECHNIQUES cat command • simply displays contents of a file tail -f command • prints the tail of a file – last 10 lines of a file with a “forever” loop (actually that is a follow) less +F • similar to tail -f but allows for switching to navigation mode by clicking Ctrl-C and back to watch mode by clicking F ; also search by using /, go to the end by G
6. 6 6 6 MONITORING MANY LOGS
7. 7 7 7 BUILD A TIMELINE Checking through all log files we build a timeline of events
8. 8 8 8 MORPHEUS KNOWS BETTER
9. 9 9 9 A GLIMPSE INTO THE FUTURE - KIBANA
10. 10 10 10 ELASTIC EVOLUTION The open source, distributed, RESTful, JSON-based search engine.
11. 11 11 11 ELASTIC EVOLUTION BEATS – THE NEW KID IN TOWN!
12. 12 12 12 Beats are open source data shippers that you install as agents on your servers to send operational data to Elasticsearch. BEATS STACK COMPONENTS - BEATS
13. 13 13 13 Logstash is an open source data collection engine with real-time pipelining capabilities. LOGSTASH STACK COMPONENTS - LOGSTASH
14. 14 14 14 Elasticsearch is a highly scalable open-source full-text search and analytics engine. It allows you to store, search, and analyze big volumes of data quickly and in near real time. ELASTICSEARCH STACK COMPONENTS - ELASTICSEARCH
15. 15 15 15 A NodeJS application; On the backend it connects to Elasticsearch. It visualizes data in a bunch of different ways. KIBANA STACK COMPONENTS - KIBANA
16. 16 16 16 ~ FREE ELASTIC STACK FEATURES (X-PACK) ~ PAID
17. 17 17 17 ELASTIC STACK BEATS LOGSTASH ELASTICSEARCH KIBANA FEATURES (FORMERLY X-PACK)
18. 18 THIS STACK IS USED FOR BIG AMOUNDS OF DATA
19. 19 19 ELASTIC STACK USE CASES
20. 20 20 20 BEATS TYPES
21. 21 21 21 INTERACTION WITH BEATS
22. 22 22 22 ELASTIC LOGGING SETUP
23. 23 23 23 Kind of a tail –f over the network. Tails the file and sends its contents over the network. FILEBEAT Filebeat
24. 24 24 24 A harvester - reading the content of a single file. The harvester reads each file, line by line, and sends the content to the output. HARVESTERS FILEBEAT CONCEPTS An input - managing harvesters & finding all sources to read from. INPUTS REGISTRY FILES Registry - stores last offset a harvester was reading from and to ensure all log lines are sent
25. 25 25 25 Perform basic processing of events before they are sent to the output. PROCESSORS FILEBEAT CONCEPTS Templates that simplify the collection, parsing, and visualization of common log formats. (ex. Apache, MySQL, IIS etc.) MODULES OUTPUTS One of the many destinations Filebeat can deliver data to.
26. 26 26 26 FILEBEAT CONFIG Sample Docker config:
27. 27 27 27 FILEBEAT INPUTS AND OUTPUTS Events Stdin events Docker container logs BSD syslog events
28. 28 LET’S SEE IT LIVE!
29. 29 29 29 Aggregate, “ tail -f ” & Search FILEBEAT
30. 30 30 30 LOGSTASH WHAT IS LOGSTASH? Logstash is an open source, server-side data processing pipeline that ingests data from a multitude of sources simultaneously, transforms it, and then sends it to your favorite “stash.”
31. 31 WHY DO WE NEED IT? IF WE HAVE BEATS
32. 32 32 32 LOGSTASH INPUTS INGEST DATA OF ALL SHAPES, SIZES, AND SOURCES Logstash supports a variety of inputs that pull in events from a multitude of common sources, all at the same time. Easily ingest from your logs, metrics, web applications, data stores, and various AWS services, all in continuous, streaming fashion.
33. 33 BUT STILL … WHY?
34. 34 34 34 LOGSTASH INPUTS and more …
35. 35 35 35 LOGSTASH FILTERS PARSE & TRANSFORM YOUR DATA ON THE FLY As data travels from source to store, Logstash filters parse each event, identify named fields to build structure
36. 36 36 36 LOGSTASH FILTERS SOME INTERESTING ONES: aggregate - Aggregates information from several events originating with a single task cidr - Checks IP addresses against a list of network blocks elapsed - Calculates the elapsed time between a pair of events grok - Parses unstructured event data into fields throttle - Throttles the number of events uuid - Adds a UUID to events
37. 37 37 37 LOGSTASH OUTPUTS CHOOSE YOUR STASH, TRANSPORT YOUR DATA Logstash has a variety of outputs that let you route data where you want
38. 38 38 38 LOGSTASH CONFIG A SAMPLE DOCKER CONFIG Here we can see al the 3 components of a configuration file.
39. 39 39 39 MAIN PIECE OF ELASTIC STACK An open-source search and analytics engine, written in Java; built on Apache Lucene Distributed: Scales to thousands of nodes High availability: Multiple replicas for each shard may be used RESTful API: CRUD, monitoring etc. by JSON HTTP calls Powerful Query DSL: express complex queries simply No schema: index data with no explicit schema (no data types and corresponding fields names required, before indexing and parsing)
40. 40 40 40 An Elasticsearch cluster is a collection of 1 or more nodes. Can scale to thousands of them. Nodes store & index data, allow for searching it. ELASTICSEARCH CLUSTER ELASTICSEARCH BASIC CONCEPTS A document is a basic unit of information that can be indexed. Documents are expressed in JSON format. DOCUMENT
41. 41 41 41 ELASTICSEARCH BASIC CONCEPTS SHARDS & REPLICAS Shards (aka primary shards) are pieces of a subdivided index. For failover scenarios Elasticsearch allows you to make one or more copies of your index’s shards into what are called replica shards (aka replicas). INDEXES & DOCUMENTS Documents make up an index. Documents are expressed in JSON. Separate indexes are used for different logically related documents. Indexes number is not limited.
42. 43 43 43 LET’S TRY IT!
43. 44 44 44 We can use SLF4J as a logging façade. Additionally Logback will be added BUT it will be enabled only in case we take care to add the Logback-related binding from SLF4J. All 3 things may be pulled using a single dependency: SLF4J + BINDING + LOGBACK SAMPLE SETUP <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> <version>1.0.13</version> </dependency>
44. 45 45 45 In the demo 4 different logging flows are presented DEMO SETUP LOGGING PATTERS
45. 46 46 46 DOCKER COMPOSE Compose is a tool for defining and running multi-container Docker applications.
46. 47 47 47 REAL PROJECT Screenshot from real project 54.589 events logged in 15 mins
47. 48 THANK YOU VITALIE BORDINIUC vitalie.bordiniuc@endava.com SENIOR JAVA DEVELOPER http://codingideas.blog/ https://github.com/vitalieb