elasticsearchもうちょっと入門

「サーバ/インフラエンジニア養成読本 ログ収集〜可視化編」（
http://gihyo.jp/book/2014/978-4-7741-6983-5?ard=1410281447）
の出版記念イベント（http://eventdots.jp/event/137658）で使用したスライド。

Transcript

1. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Jun Ohtani @johtani elasticsearch.΋͏ͪΐͬͱೖ໳
2. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited about • Me, Jun Ohtani / Technical Adovocate lucene-gosenίϛολʔ ElasticSearch Server೔ຊޠ൛ͷ຋༁ elasticsearch-extended-analysisͷ։ൃ http://blog.johtani.info • Elasticsearch, founded in 2012 Products: Elasticsearch, Logstash, Kibana, Marvel
 Professional services: Support & development subscriptions Trainings
3. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearchʁ
4. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ϑϦʔϫʔυݕࡧ
5. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ߜΓࠐΈ
6. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ϋΠϥΠτ
7. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ιʔτ
8. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ϖʔδϯά
9. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited ूܭ
10. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited αδΣετ
11. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ
12. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Logstash
13. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Logstash in 10 seconds • Managing events and logs • Collect, parse, enrich, store data • Modular: many, many inputs and outputs • Apache License 2.0 • Ruby app (JRuby) • Part of Elasticsearch family
14. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Logstash architecture Logstash Input Output Filter ? ? collect and split alter and enrich store and visualize
15. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Inputs • Monitoring: collectd, graphite, ganglia, snmptrap, zenoss • Datastores: elasticsearch, redis, sqlite, s3 • Queues: rabbitmq, zeromq • Logging: eventlog, lumberjack, gelf, log4j, relp, syslog, varnish log • Platforms: drupal_dblog, gemfire, heroku, sqs, s3, twitter • Local: exec, generator, file, stdin, pipe, unix • Protocol: imap, irc, stomp, tcp, udp, websocket, wmi, xmpp
16. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Filters • alter, anonymize, checksum, csv, drop, multiline • dns, date, extractnumbers, geoip, i18n, kv, noop, ruby, range • json, urldecode, useragent • metrics, sleep • … many, many more …
17. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Outputs • Store: elasticsearch, gemfire, mongodb, redis, riak, rabbitmq • Monitoring: ganglia, graphite, graphtastic, nagios, opentsdb, statsd, zabbix • Notification: email, hipchat, irc, pagerduty, sns • Protocol: gelf, http, lumberjack, metriccatcher, stomp, tcp, udp, websocket, xmpp • External Monitoring: boundary, circonus, cloudwatch, datadog, librato • External service: google big query, google cloud storage, jira, loggly, riemann, s3, sqs, syslog, zeromq • Local: csv, exec, file, pipe, stdout, null
18. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Installation • Ruby application, but Java required (JRuby) • Download single tgz, deb, RPM (also repositories) No gem/dependency nightmares! • Puppet module
19. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Simple example • Download, create config and run input {! stdin {}! }! ! output {! stdout { debug => true }! } echo foo | bin/logstash -f simple.conf! {! "message" => "foo",! "@version" => "1",! "@timestamp" => "2014-01-20T13:30:59.648Z",! "host" => "kryptic.fritz.box"! } simple.conf
20. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited CLF log files {! "message" => "193.99.144.85 - - [23/Jan/2014:17:11:55 +0000] \"GET / HTTP/1.1\" 200 140 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/ 535.19\"",! "@version" => "1",! "@timestamp" => "2014-01-24T07:56:02.460Z",! "host" => "kryptic.local",! "clientip" => "193.99.144.85",! "ident" => "-",! "auth" => "-",! "timestamp" => "23/Jan/2014:17:11:55 +0000",! "verb" => "GET",! "request" => "/",! "httpversion" => "1.1",! "response" => "200",! "bytes" => "140",! "referrer" => "\"-\"",! "agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/ 535.19\""! }
21. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Write to elasticsearch input { stdin {} }! ! filter {! grok {! match => [ message, "%{COMBINEDAPACHELOG}" ]! }! }! ! output {! elasticsearch_http {}! }
22. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Deploying ELK for scale Shipper Logstash Store/Search Visualize
23. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Add a broker Shipper Logstash Store/Search Visualize Broker Brokers help with scale and stability by buffering the input and protecting against output downtime. ! Tip: set limits on broker queue to push back on source as well.
24. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Scale out the shipper Shipper Logstash Store/Search Visualize Broker Shipper Shipper
25. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Scale out the broker Shipper Logstash Store/Search Visualize Broker Shipper Shipper Broker Broker
26. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Scale out Logstash Shipper Logstash Store/Search Visualize Broker Shipper Shipper Broker Broker Logstash Logstash
27. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Scale out Elasticsearch Shipper Logstash Store/Search Visualize Broker Shipper Shipper Broker Broker Logstash Logstash Store/Search
28. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited aggregation
29. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Aggregationͱ͸ • 1.0͔Βར༻Մೳ • FacetΑΓ΋ڧྗͳूܭͳͲ͕Մೳ ֊૚తͳूܭɺάϧʔϓԽ
 ಈతͳूܭɺάϧʔϓԽ • େ͖͘2छྨ BucketɹυΩϡϝϯτͷ͋Δ஋͝ͱʹ݁ՌΛάϧʔϐϯά Metricɹ υΩϡϝϯτͷ࣋ͭ஋Λूܭ
30. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Aggregationͱ͸ • SQLͩͱ ! ! • GROUP BY colorɿBucket • COUNT(color)ɿMetric SELECT  COUNT(color)     FROM  table   GROUP  BY  color
31. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Bucket • Bucketͷྫɿ • Ϣʔβͷੑผɿʮஉੑʯʮঁੑʯ • πΠʔτͷ৔ॴɿʮ౦ژʯʮژ౎ʯ • πΠʔτͷݴޠɿʮ೔ຊޠʯʮӳޠʯ • ϒϥ΢βɿʮIEʯʮChromeʯ
32. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Metric • Metricͷྫɿ • Bucket಺ͷυΩϡϝϯτͷ਺ • πΠʔτจࣈ਺ͷฏۉ • ച্ߴͷ࠷େ஋ • ϦΫΤετॲཧ࣌ؒͷ࠷େ஋ɺ95ύʔηϯλΠϧ
33. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited αϯϓϧ { "price": 10000, "color": "red", "make": "honda", ɹɹ"sold": "2014-10-28" } { "price": 20000, "color": "red", "make": "honda", ɹɹ"sold": "2014-11-05" } { "price": 30000, "color": "green", "make": "ford", ɹɹ"sold": "2014-05-18" } { "price": 15000, "color": "blue", "make": "toyota", ɹɹ"sold": "2014-07-02" } { "price": 12000, "color": "green", "make": "toyota", ɹɹ"sold": "2014-08-19" }...
34. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫ:৭Ͱूܭ » GET /cars/transactions/_search?search_type=count { "aggs" : { "popular_colors" : { "terms" : { "field" : "color" } } } }
35. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫ:৭Ͱूܭ "aggregations": { "popular_colors": { "buckets": [ { "key": "red", "doc_count": 4 },{ "key": "blue", "doc_count": 2 }, { "key": "green", "doc_count": 2 }…
36. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫ:৭Ͱ෼ྨͯ͠Ձ֨ͷฏۉ » GET /cars/transactions/_search?search_type=count { "aggs": { "colors": { "terms": { "field": "color" }, "aggs": { "avg_price": { "avg": { "field": "price" } } } } } }
37. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫ:৭Ͱ෼ྨͯ͠Ձ֨ͷฏۉ "aggregations": { "colors": { "buckets": [ { "key": "red", "doc_count": 4, "avg_price": { "value": 32500 } }, {"key": "blue", "doc_count": 2, "avg_price": { "value": 20000 } },,…
38. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫ:ϝʔΧʔΛ௥Ճ » GET /cars/transactions/_search?search_type=count { "aggs": { "colors": { "terms": { "field": “color" }, "aggs": { "avg_price": { "avg": { "field": “price" } }, "make": { "terms": { "field": "make" } } } } }}
39. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫ:ϝʔΧʔΛ௥Ճ "aggregations": { "colors": { "buckets": [ { "key": "red", "doc_count": 4, "make": { "buckets": [ { "key": "honda", "doc_count": 3 }, { "key": "bmw", "doc_count": 1 } ] }, "avg_price": { "value": 32500 }…
40. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫ:ϝʔΧʔ͝ͱͷ࠷খ/࠷େՁ֨ » GET /cars/transactions/_search?search_type=count { "aggs": { "colors": { "terms": { "field": “color” }, "aggs": { "avg_price": { "avg": { "field": "price" } }, "make" : { "terms" : { "field" : “make" }, "aggs" : { "min_price" : { "min": { "field": "price"} }, "max_price" : { "max": { "field": "price"} } } } } } } }
41. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ྫ:ϝʔΧʔ͝ͱͷ࠷খ/࠷େՁ֨ { "key": "red", "doc_count": 4, "make": { "buckets": [ { "key": "honda", "doc_count": 3, "min_price": { "value": 10000 }, "max_price": { "value": 20000 } }, { "key": “bmw”, … } ] }, "avg_price": { "value": 32500 }…
42. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited bucket • filter • filters (>=1.4.0) • missing • nested • reverse nested (>=1.2.0) • children • terms • significant terms (>=1.1.0) • range • date range • ipv4 range • histogram • date histogram • geo distance • geohash grid
43. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited metric • min • max • sum • avg • stats • extended_stats • value count • percentiles (>=1.1.0) • percentile ranks (>=1.3) • cardinality (>=1.1.0) • geo bounds (>=1.3.0) • top_hits (>=1.3.0) • scripted metric 
 (>= 1.4.0)
44. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ·ͱΊ • ࢒೦ͳ͕ΒKibana3Ͱ͸ར༻Ͱ͖ͳ͍ • REST APIΛར༻͢Ε͹ूܭՄೳ • ଟஈͰूܭՄೳ • ϝϞϦͷফඅྔʹ஫ҙ
45. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited elasticsearchษڧձ • ୈ6ճelasticsearchษڧձ 
 ɹɹ#elasticsearch #elasticsearchjp • ೔࣌ɿ2014/09/16ɹ19:30ʙ • ৔ॴɿϦΫϧʔτςΫϊϩδʔζ41Fʢ౦ژӺʣ • URLɿhttp://elasticsearch.doorkeeper.jp/events/ 13917
46. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited ࢀߟ • Github https://github.com/elasticsearch • ΨΠυ http://www.elasticsearch.org/guide/ • αϙʔτ http://www.elasticsearch.com/support/