Loggen mit dem Elastic Stack

Das aktuell wahrscheinlich verbreitetste und beliebteste Werkzeug für Log-Monitoring und -Analyse ist der Elastic Stack:

* Elasticsearch um große Datenmengen zu analysieren und zu durchsuchen.
* Logstash und Beats alle relevanten Informationen einzusammeln.
* Kibana zur leistungsstarken und flexiblen Visualisierung.

Dieser Vortrag gibt einen Überblick über die vier Technologien, wie sie zusammenarbeiten und wie sie Ihre Probleme lösen können.

Transcript

1. Loggen mit dem Stack Philipp Krenn
   
   
   
   
   
   
   
   
   
   
   
   @xeraa
3. Infrastructure | Developer Advocate
4. Who uses 5+ servers or containers?
5. How are you logging?
6. You Know, for Search
8. Search is our core business
10. ELK Stack
15. Elastic Stack
20. Filebeat
21. tail -f
    
    
22. tail -f over the network
    
23. tail -f over the network on !
24. Parse & Enrich Logstash or Ingest-Node
25. 127.0.0.1 - - [27/Jun/2017:00:56:54 +0000] "GET / HTTP/1.1" 200 612 "-" "Go-http-client/1.1" "referrer": "-", "response_code": "200", "remote_ip": "127.0.0.1", "method": "GET", "user_name": "-", "http_version": "1.1", "body_sent": { "bytes": "612" }, "url": "/", "user_agent": { "os": "Other", "name": "Other", "os_name": "Other", "device": "Other" }
26. "remote_ip": "34.253.145.46", "geoip": { "continent_name": "North America", "city_name": "Houston", "country_iso_code": "US", "region_name": "Texas", "location": { "lon": -95.5858, "lat": 29.6997 } }
27. At-Least-Once Backpressure Graceful Downtime
31. Filtering include_lines
    
    exclude_lines
    
    exclude_files filebeat.prospectors: - input_type: log paths: - /var/log/myapp/*.log include_lines: ["^ERR", "^WARN"]
32. Multiline Exception in thread "main" java.lang.IllegalStateException: A book has a null property at com.example.myproject.Author.getBookIds(Author.java:38) at com.example.myproject.Bootstrap.main(Bootstrap.java:14) Caused by: java.lang.NullPointerException at com.example.myproject.Book.getId(Book.java:22) at com.example.myproject.Author.getBookIds(Author.java:35) ... 1 more multiline.pattern: '^[[:space:]]+|^Caused by:' multiline.negate: false multiline.match: after
33. JSON Decode
34. Docker Metadata 6.0.0-beta1 - input_type: log paths: - /var/lib/docker/containers/*/*-json.log document_type: docker json.message_key: log processors: - add_docker_metadata: ~
35. Kubernetes Metadata 6.0.0-alpha2 processors: - add_kubernetes_metadata: in_cluster: true
36. Filebeat Modules
37. Metricbeat
38. Metricbeat System
39. Metricbeat Service
40. Docker Metadata 6.0.0-alpha2 processors: - add_docker_metadata: match_fields: ["system.process.cgroup.id"] host: "unix:///var/run/docker.sock"
41. Kubernetes Metadata 6.0.0-alpha2 processors: - add_kubernetes_metadata: in_cluster: true
42. Kubernetes Metrics 6.0.0-alpha2 - module: kubelet metricsets: ["node", "container", "volume", "pod", "system"] hosts: ["localhost:10255"]
43. Packetbeat
44. Protocols
45. Flows Application layer: Unsupported or encrypted protocols IP / TCP / UDP Number of packets & bytes Retransmissions Temporal flow
46. Heartbeat
47. Winlogbeat
48. libbeat https://github.com/elastic/beats/tree/master/generate/beat
50. --- version: '2' services: kibana: image: docker.elastic.co/kibana/kibana:5.4.2 links: - elasticsearch ports: - 5601:5601 elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:5.4.2 cap_add: - IPC_LOCK volumes: - esdata1:/usr/share/elasticsearch/data ports: - 9200:9200 volumes: esdata1: driver: local
51. Demo https://github.com/xeraa/vagrant-elastic-stack
52. Conclusion
55. Thanks! Questions? Stickers Philipp Krenn
    
    
    
    
    
    
    
    
    
    
    
    @xeraa
56. Elk horn: https://www.theexplora.com/the-irish-elk- megaloceros-giganteus/ Container ship: https://flic.kr/p/hjxW62 Wooden logs: https://flic.kr/p/9vvbKE Axe https://flic.kr/p/pBU2VD Files https://flic.kr/p/2EFcQ Metric https://flic.kr/p/9g5h3f Packages https://flic.kr/p/cJFDLN Windows https://flic.kr/p/94Z6y Library https://flic.kr/p/fiXcBj Beats https://flic.kr/p/bWopW7