Beatsシリーズでお手軽 メトリック収集可視化

JJUG CCC 2016 Springで発表した時の資料です。

Transcript

1. ‹#› 2016/05/21 Jun Ohtani / @johtani BeatsγϦʔζͰ͓खܰ
 ϝτϦοΫऩूՄࢹԽ
2. about • Me, Jun Ohtani / Technical Adovocate ‒ lucene-gosenίϛολʔ ‒ ElasticSearch Server೔ຊޠ൛ͷ຋༁ ‒ http://blog.johtani.info
 
 • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats
 Marvel, Shield, Watcher, Graph ‒ Professional services: Support & development subscriptions ‒ Trainings, Consultings 2
3. 3 What's Beats?
4. Lightweight shipper • Small application • Install as agent on your servers • Written in Golang • No runtime dependencies • Single purpose 4 https://www.flickr.com/photos/8barbikes/17256970434/
5. Examples of operational data 5 wire data system stats logs Packetbeat Topbeat Filebeat Winlogbeat
6. Captures insights from network packets 6 Packetbeat
7. Sniffing the network traffic 7 Client Server sniff sniff • Copy traffic at OS or hardware level • Is completely passive • ZERO latency overhead • Not in the request/response path, cannot break your application
8. Sniffing use cases 8 • Security • Intrusion Detection Systems • Troubleshooting network issues • Troubleshooting applications • Performance analysis
9. Check out the demo on Our web site 9
10. Like the Unix top command but sends the output periodically to Elasticsearch. Also works on Windows. 10 Topbeat
11. Topbeat: Exported data 11 • system load • total CPU usage • CPU usage per core • Swap, memory usage System wide • state • name • command line • pid • CPU usage • memory usage Per process • available disks • used, free space • mounted points Disk usage
12. Forwards log lines to Elasticsearch 12 Filebeat
13. Filebeat: Never lose a log line 13 line line line line line read pointer Filebeat Logstash Back-pressure sensitive protocol Yo Filebeat, slow it down a bit, pls K buddy line The original log lines act like a queue
14. Filebeat: Parse logs with Logstash 14 • Filebeat sends out unparsed log lines • Use filters from Logstash to parse the log lines • Flexible, with conditionals & custom filters • Forward data to other systems using the Logstash output plugins Filebeat Elasticsearch Logstash Other systems
15. Filebeat: Parse logs with Ingest Node 15 • Upcoming in 5.0 • Filebeat sends out unparsed log lines directly to Elasticsearch • Use Ingest Node processors to parse the log lines • Easier to setup Filebeat Elasticsearch Don’t miss the Ingest Node presentation tomorrow at 2:15 p.m.
16. ‹#› Multiline 16 multiline: # Sticks together all lines # that don’t start with a [ pattern: ^\[ negate: true match: after Filebeat extra power • Sticks together related log lines in a single event • For all those long exceptions • Can also be done by Logstash, but it’s sometimes easier to configure the patterns closer to the source
17. Forwards Windows Event logs to Elasticsearch 17 Winlogbeat
18. Winlogbeat overview 18 • Supports Windows versions starting with XP • It remembers how far it read, so it never loses log events • Winlogbeat sends out unparsed Windows event logs • Use Ingest Node or Logstash to parse the Windows event logs
19. ‹#› DEMOʁ
20. ‹#› Community Beats
21. 21 1 Apachebeat 2 Dockerbeat 3 Elasticbeat 4 Execbeat 5 Factbeat 6 Hsbeat 14 COMMUNITY BEATS Sending all sorts of data to Logstash and Elasticsearch 7 Httpbeat 8 Nagioscheckbeat 9 Nginxbeat 10 Phpfpmbeat 11 Pingbeat 13 Unifiedbeat 12 Redisbeat 14 Uwsgibeat
22. Community Beats: libbeat 22 libbeat Community Beats Elastic Beats Elasticsearch Logstash • Golang library • Outputs for Elasticsearch and Logstash • At least once guarantees • Encryption & authentication • Common code for configuration files, logging, daemonizing, CLI flags, etc.
23. ‹#› How can we make it even easier to create a new Beat? 23
24. Beat generator Quickly get started with the development of a new Beat 24 $ pip install cookiecutter $ cookiecutter https://github.com/elastic/beat-generator.git project_name [Examplebeat]: Mybeat github_name [your-github-name]: tsg beat [examplebeat]: mybeat beat_path [github.com/your-github-name]: github.com/tsg full_name [Firstname Lastname]: Tudor Golubenco
25. 25 • Cross-compiles to all our supported platforms • Produces RPMs, DEBs, • Same tools that we use to build the official Elastic Beats • Can be executed from Travis CI Beats Packer
26. Overview about libbeat and custom logic 26
27. Develop your Beater Quickly get started with the development of a new Beat 27 type Beater interface { Config(*Beat) error Setup(*Beat) error Run(*Beat) error Cleanup(*Beat) error Stop() }
28. ‹#› Beat-generator DEMO! 28
29. ೔ຊޠ৘ใ • ElasticBeatsΛಋೖͯ͠Έͨ࿩/Go Conference 2016 Spring by Daichi Hirata • https://speakerdeck.com/daic_h/go-conference-2016-spring • beat-generatorʹΑΔBeats։ൃ - Developer.IO • http://dev.classmethod.jp/server-side/elasticsearch/develop-beats-by-beats- generator/ • Acroquest TechnologyגࣜձࣾͷΤϯδχΞ͕ॻٕ͘ज़ϒϩά • http://acro-engineer.hatenablog.com/archive/category/Beats 29