Slide 1
Slide 1 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Jun Ohtani
@johtani
elasticsearch.͏ͪΐͬͱೖ
Slide 2
Slide 2 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
about
• Me, Jun Ohtani / Technical Adovocate
lucene-gosenίϛολʔ
ElasticSearch Serverຊޠ൛ͷ༁
elasticsearch-extended-analysisͷ։ൃ
http://blog.johtani.info
• Elasticsearch, founded in 2012
Products: Elasticsearch, Logstash, Kibana, Marvel
Professional services: Support & development subscriptions
Trainings
Slide 3
Slide 3 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Elasticsearchʁ
Slide 4
Slide 4 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
ϑϦʔϫʔυݕࡧ
Slide 5
Slide 5 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
ߜΓࠐΈ
Slide 6
Slide 6 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
ϋΠϥΠτ
Slide 7
Slide 7 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
ιʔτ
Slide 8
Slide 8 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
ϖʔδϯά
Slide 9
Slide 9 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
ूܭ
Slide 10
Slide 10 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
αδΣετ
Slide 11
Slide 11 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Elasticsearch in 10 seconds
• εΩʔϚϑϦʔɺࢄυΩϡϝϯτετΞɺREST
& JSON
• Φʔϓϯιʔε: Apache License 2.0
• ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ
• JavaͰ࣮ɻ֦ு༰қ
Slide 12
Slide 12 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Logstash
Slide 13
Slide 13 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Logstash in 10 seconds
• Managing events and logs
• Collect, parse, enrich, store data
• Modular: many, many inputs and outputs
• Apache License 2.0
• Ruby app (JRuby)
• Part of Elasticsearch family
Slide 14
Slide 14 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Logstash architecture
Logstash
Input Output
Filter
? ?
collect and split alter and enrich store and visualize
Slide 15
Slide 15 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Inputs
• Monitoring: collectd, graphite, ganglia, snmptrap,
zenoss
• Datastores: elasticsearch, redis, sqlite, s3
• Queues: rabbitmq, zeromq
• Logging: eventlog, lumberjack, gelf, log4j, relp,
syslog, varnish log
• Platforms: drupal_dblog, gemfire, heroku, sqs, s3,
twitter
• Local: exec, generator, file, stdin, pipe, unix
• Protocol: imap, irc, stomp, tcp, udp, websocket,
wmi, xmpp
Slide 16
Slide 16 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Filters
• alter, anonymize, checksum, csv, drop, multiline
• dns, date, extractnumbers, geoip, i18n, kv, noop,
ruby, range
• json, urldecode, useragent
• metrics, sleep
• … many, many more …
Slide 17
Slide 17 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Outputs
• Store: elasticsearch, gemfire, mongodb, redis, riak,
rabbitmq
• Monitoring: ganglia, graphite, graphtastic, nagios,
opentsdb, statsd, zabbix
• Notification: email, hipchat, irc, pagerduty, sns
• Protocol: gelf, http, lumberjack, metriccatcher, stomp,
tcp, udp, websocket, xmpp
• External Monitoring: boundary, circonus, cloudwatch,
datadog, librato
• External service: google big query, google cloud
storage, jira, loggly, riemann, s3, sqs, syslog, zeromq
• Local: csv, exec, file, pipe, stdout, null
Slide 18
Slide 18 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Installation
• Ruby application, but Java required (JRuby)
• Download single tgz, deb, RPM (also repositories)
No gem/dependency nightmares!
• Puppet module
Slide 19
Slide 19 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Simple example
• Download, create config and run
input {!
stdin {}!
}!
!
output {!
stdout { debug => true }!
}
echo foo | bin/logstash -f simple.conf!
{!
"message" => "foo",!
"@version" => "1",!
"@timestamp" => "2014-01-20T13:30:59.648Z",!
"host" => "kryptic.fritz.box"!
}
simple.conf
Slide 20
Slide 20 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
CLF log files
{!
"message" => "193.99.144.85 - - [23/Jan/2014:17:11:55 +0000]
\"GET / HTTP/1.1\" 200 140 \"-\" \"Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/
535.19\"",!
"@version" => "1",!
"@timestamp" => "2014-01-24T07:56:02.460Z",!
"host" => "kryptic.local",!
"clientip" => "193.99.144.85",!
"ident" => "-",!
"auth" => "-",!
"timestamp" => "23/Jan/2014:17:11:55 +0000",!
"verb" => "GET",!
"request" => "/",!
"httpversion" => "1.1",!
"response" => "200",!
"bytes" => "140",!
"referrer" => "\"-\"",!
"agent" => "\"Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/
535.19\""!
}
Slide 21
Slide 21 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Write to elasticsearch
input { stdin {} }!
!
filter {!
grok {!
match => [ message, "%{COMBINEDAPACHELOG}" ]!
}!
}!
!
output {!
elasticsearch_http {}!
}
Slide 22
Slide 22 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Deploying ELK for scale
Shipper Logstash Store/Search Visualize
Slide 23
Slide 23 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Add a broker
Shipper Logstash Store/Search
Visualize
Broker
Brokers help with scale and stability by buffering the input and
protecting against output downtime.
!
Tip: set limits on broker queue to push back on source as well.
Slide 24
Slide 24 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Scale out the shipper
Shipper Logstash Store/Search
Visualize
Broker
Shipper
Shipper
Slide 25
Slide 25 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Scale out the broker
Shipper Logstash Store/Search
Visualize
Broker
Shipper
Shipper
Broker
Broker
Slide 26
Slide 26 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Scale out Logstash
Shipper Logstash Store/Search
Visualize
Broker
Shipper
Shipper
Broker
Broker
Logstash
Logstash
Slide 27
Slide 27 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Scale out Elasticsearch
Shipper Logstash Store/Search
Visualize
Broker
Shipper
Shipper
Broker
Broker
Logstash
Logstash
Store/Search
Slide 28
Slide 28 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
aggregation
Slide 29
Slide 29 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Aggregationͱ
• 1.0͔Βར༻Մೳ
• FacetΑΓڧྗͳूܭͳͲ͕Մೳ
֊తͳूܭɺάϧʔϓԽ
ಈతͳूܭɺάϧʔϓԽ
• େ͖͘2छྨ
BucketɹυΩϡϝϯτͷ͋Δ͝ͱʹ݁ՌΛάϧʔϐϯά
Metricɹ υΩϡϝϯτͷ࣋ͭΛूܭ
Slide 30
Slide 30 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Aggregationͱ
• SQLͩͱ
!
!
• GROUP BY colorɿBucket
• COUNT(color)ɿMetric
SELECT
COUNT(color)
FROM
table
GROUP
BY
color
Slide 31
Slide 31 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Bucket
• Bucketͷྫɿ
• Ϣʔβͷੑผɿʮஉੑʯʮঁੑʯ
• πΠʔτͷॴɿʮ౦ژʯʮژʯ
• πΠʔτͷݴޠɿʮຊޠʯʮӳޠʯ
• ϒϥβɿʮIEʯʮChromeʯ
Slide 32
Slide 32 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
Metric
• Metricͷྫɿ
• BucketͷυΩϡϝϯτͷ
• πΠʔτจࣈͷฏۉ
• ച্ߴͷ࠷େ
• ϦΫΤετॲཧ࣌ؒͷ࠷େɺ95ύʔηϯλΠϧ
Slide 33
Slide 33 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
αϯϓϧ
{ "price": 10000, "color": "red", "make": "honda",
ɹɹ"sold": "2014-10-28" }
{ "price": 20000, "color": "red", "make": "honda",
ɹɹ"sold": "2014-11-05" }
{ "price": 30000, "color": "green", "make": "ford",
ɹɹ"sold": "2014-05-18" }
{ "price": 15000, "color": "blue", "make": "toyota",
ɹɹ"sold": "2014-07-02" }
{ "price": 12000, "color": "green", "make": "toyota",
ɹɹ"sold": "2014-08-19" }...
Slide 34
Slide 34 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
ྫ:৭Ͱूܭ
» GET /cars/transactions/_search?search_type=count
{
"aggs" : {
"popular_colors" : {
"terms" : {
"field" : "color"
}
}
}
}
Slide 35
Slide 35 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
ྫ:৭Ͱूܭ
"aggregations": {
"popular_colors": {
"buckets": [
{
"key": "red",
"doc_count": 4
},{
"key": "blue",
"doc_count": 2
}, {
"key": "green",
"doc_count": 2
}…
Slide 36
Slide 36 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
ྫ:৭Ͱྨͯ͠Ձ֨ͷฏۉ
» GET /cars/transactions/_search?search_type=count
{
"aggs": {
"colors": {
"terms": {
"field": "color"
},
"aggs": {
"avg_price": {
"avg": {
"field": "price"
}
} } } } }
Slide 37
Slide 37 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
ྫ:৭Ͱྨͯ͠Ձ֨ͷฏۉ
"aggregations": {
"colors": {
"buckets": [
{ "key": "red",
"doc_count": 4,
"avg_price": {
"value": 32500
}
}, {"key": "blue",
"doc_count": 2,
"avg_price": {
"value": 20000
}
},,…
Slide 38
Slide 38 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
ྫ:ϝʔΧʔΛՃ
» GET /cars/transactions/_search?search_type=count
{
"aggs": {
"colors": {
"terms": { "field": “color" },
"aggs": {
"avg_price": {
"avg": { "field": “price" }
},
"make": {
"terms": { "field": "make" }
}
}
}
}}
Slide 39
Slide 39 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
ྫ:ϝʔΧʔΛՃ
"aggregations": {
"colors": {
"buckets": [
{ "key": "red",
"doc_count": 4,
"make": {
"buckets": [
{ "key": "honda",
"doc_count": 3 },
{ "key": "bmw",
"doc_count": 1 } ] },
"avg_price": {
"value": 32500
}…
Slide 40
Slide 40 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
ྫ:ϝʔΧʔ͝ͱͷ࠷খ/࠷େՁ֨
» GET /cars/transactions/_search?search_type=count
{
"aggs": {
"colors": {
"terms": { "field": “color” },
"aggs": {
"avg_price": { "avg": { "field": "price" } },
"make" : {
"terms" : { "field" : “make" },
"aggs" : {
"min_price" : { "min": { "field": "price"} },
"max_price" : { "max": { "field": "price"} }
} } } } }
}
Slide 41
Slide 41 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
ྫ:ϝʔΧʔ͝ͱͷ࠷খ/࠷େՁ֨
{ "key": "red",
"doc_count": 4,
"make": {
"buckets": [
{ "key": "honda",
"doc_count": 3,
"min_price": { "value": 10000 },
"max_price": { "value": 20000 }
}, {
"key": “bmw”,
…
} ] },
"avg_price": {
"value": 32500
}…
Slide 42
Slide 42 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
bucket
• filter
• filters (>=1.4.0)
• missing
• nested
• reverse nested
(>=1.2.0)
• children
• terms
• significant terms
(>=1.1.0)
• range
• date range
• ipv4 range
• histogram
• date histogram
• geo distance
• geohash grid
Slide 43
Slide 43 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
metric
• min
• max
• sum
• avg
• stats
• extended_stats
• value count
• percentiles (>=1.1.0)
• percentile ranks (>=1.3)
• cardinality (>=1.1.0)
• geo bounds (>=1.3.0)
• top_hits (>=1.3.0)
• scripted metric
(>= 1.4.0)
Slide 44
Slide 44 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
·ͱΊ
• ೦ͳ͕ΒKibana3Ͱར༻Ͱ͖ͳ͍
• REST APIΛར༻͢ΕूܭՄೳ
• ଟஈͰूܭՄೳ
• ϝϞϦͷফඅྔʹҙ
Slide 45
Slide 45 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
elasticsearchษڧձ
• ୈ6ճelasticsearchษڧձ
ɹɹ#elasticsearch #elasticsearchjp
• ࣌ɿ2014/09/16ɹ19:30ʙ
• ॴɿϦΫϧʔτςΫϊϩδʔζ41Fʢ౦ژӺʣ
• URLɿhttp://elasticsearch.doorkeeper.jp/events/
13917
Slide 46
Slide 46 text
Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
ࢀߟ
• Github
https://github.com/elasticsearch
• ΨΠυ
http://www.elasticsearch.org/guide/
• αϙʔτ
http://www.elasticsearch.com/support/