OSC 2018 Tokyo/Fall での発表資料になります。
!12018/10/27Community Engineer @Elastic Jun Ohtani @johtani༷ʑͳϝτϦΫεϩάΛूΊͯγεςϜղੳ - Elastic Stackͷೖͱ׆༻ -
View Slide
!2ΞδΣϯμ• ϝτϦοΫʗϩάͱʁ• γεςϜϝτϦΫεղੳɺϩάղੳΛࢼ͠ʹͬͯΈΑ͏• Beats - Elasticsearch - KibanaͰղੳ• ຊ֨తʹղੳΛΔʹʁ• LogstashͰϩάϝτϦΫεΛதܧɾू• ͞Βʹ৭ʑࢼͯ͠ΈΔʹʁ
!3about• Me, Jun Ohtani / Community Engineer‒ lucene-gosenίϛολʔ‒ σʔλੳج൫ߏஙೖ ڞஶ‒ http://blog.johtani.info • Elastic, founded in 2012‒ Products: Elasticsearch, Logstash, Kibana, Beats Elastic APM, Elastic Cloud, Swiftype Professional services: Support & development subscriptions Trainings, Consulting, SaaS
!4ͲΜͳϝτϦοΫɺ ϩάΛूΊ͍ͯ·͔͢ʁ
!5ϝτϦοΫ• CPUɺϝϞϦ༻ɺσΟεΫ༻• ΞΫηεɺωοτϫʔΫసૹྔ• Ԡ࣌ؒ• ίωΫγϣϯ• τϥϯβΫγϣϯɺച্• ίϯςφͷ্ͷ֤छϝτϦΫε
!6ϩά• ೝূϩά• γεςϜϩά• ΞϓϦέʔγϣϯϩά• Slow log• ΞΫηεϩά• ίϯςφͷதͷϩά
!7Ͱ͖ΕϩάͱϝτϦοΫΛ ·ͱΊͯ1ͭͷը໘Ͱ ݟ͍ͨͰ͢ΑͶʁ
!8Elastic Stack
ElasticStackอଘɺݕࡧɺੳElasticsearchՄࢹԽɺཧKibanaBeats ΠϯδΣετLogstash
MetricsLoggingAPMSite SearchApplicationSearchBusiness AnalyticsEnterprise SearchSecurity AnalyticsFuture ιϦϡʔγϣϯอଘɺݕࡧɺੳՄࢹԽɺཧΠϯδΣετKibanaElasticsearchBeats LogstashElasticStack
MetricsLoggingAPMSite SearchApp SearchBusiness AnalyticsEnterprise SearchSecurity AnalyticsFuture ιϦϡʔγϣϯSaaSElastic CloudSelf ManagedElastic Cloud Enterprise StandaloneσϓϩΠอଘɺݕࡧɺੳՄࢹԽɺཧΠϯδΣετKibanaElasticsearchBeats LogstashElasticStack
อଘɺݕࡧɺੳElasticsearchՄࢹԽɺཧKibanaBeats ΠϯδΣετLogstashMetricsLoggingAPMSite SearchApplicationSearchBusiness AnalyticsEnterprise SearchSecurity AnalyticsFuture ιϦϡʔγϣϯSaaSElastic CloudSelf ManagedElastic Cloud Enterprise StandaloneσϓϩΠElasticStack
!13ఆܕͷϝτϦΫε/ϩάղੳΛElastic StackͰ
!14ϝτϦοΫɾϩάੳʢ؆қ൛ʣBeatsLogFilesMetricsWireDataKibanaInstancesElasticsearchNodes
!15
16Beatsܰྔσʔλγούʔιʔε͔ΒσʔλΛసૹ సૹ͠Elasticsearchʹू มͱύʔεͷͨΊLogstashʹసૹElastic CloudʹసૹLibbeat: ΧελϜbeatsͷͨΊͷAPIϑϨʔϜϫʔΫ30Ҏ্ͷίϛϡχςΟbeats
The Beats familyHeartbeatUptime monitoringFilebeatLog filesWinlogbeatWindows Event LogsPacketbeatNetwork data+40communityBeatsMetricbeatMetricsAuditbeatAudit data
Collect systemand applicationmetricsMetricbeat
lots of modulesMetricbeat
!20Metricbeat Ϟδϡʔϧ● Aerospike module● Apache module● Ceph module● Couchbase module● Docker module● Dropwizard module● Elasticsearch module● Etcd module● Golang module● Graphite module● HAProxy module● HTTP module● Jolokia module● Kafka module● Kibana module● Kubernetes module● kvm module● Logstash module● Memcached module● MongoDB module● Munin module● MySQL module● Nginx module●● PHP_FPM module● PostgreSQL module● Prometheus module● RabbitMQ module● Redis module● System module● uwsgi module● vSphere module● Windows module● ZooKeeper module
tail log fromfileFilebeat
many modulesFilebeat
Filebeat modules - v6.4.2• Apache2 module• Auditd module• Icinga module• IIS module• Kafka module• Logstash module• MongoDB module• MySQL module• Nginx module• Osquery module• PostgreSQL module• Redis module• System module• Traefik module
Capture thePacketPacketbeat
Welcometo 1998winlogbeat
Nowwinlogbeat
!28
29ElasticsearchHeart of the Elastic Stackࢄܕɺεέʔϥϒϧ ߴՄ༻ੑ Ϛϧνςφϯτ։ൃऀϑϨϯυϦʔ ϦΞϧλΠϜɺશจݕࡧ ΞάϦήʔγϣϯ
Elasticsearchͱʁ
ϑϦʔϫʔυݕࡧ!31
ߜΓࠐΈ!32
ϋΠϥΠτ!33
ιʔτ!34
ϖʔδϯά!35
ूܭ!36
αδΣετ!37
Elasticsearch in 10 seconds• εΩʔϚϑϦʔɺࢄυΩϡϝϯτετΞɺREST & JSON• Φʔϓϯιʔε: Apache License 2.0• ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ• JavaͰ࣮ɻ֦ு༰қ!38
؆୯ͳCRUD
σʔλొ40curl -XPUT localhost:9200/books/book/1 -d '{"title" : "Elasticsearch - The definitive guide","authors" : "Clinton Gormley","started" : "2013-02-04","pages" : 230}'
σʔλߋ৽41curl -XPUT localhost:9200/books/book/1 -d '{"title" : "Elasticsearch - The definitive guide","authors" : [ "Clinton Gormley", "Zachary Tong" ],"started" : "2013-02-04","pages" : 230}'
σʔλআ!42curl -X DELETE localhost:9200/books/book/1σʔλͷऔಘcurl —X GET localhost:9200/books/book/1curl —X GET localhost:9200/books/book/1/_source
ݕࡧ - Query DSL!43curl -XGET ‘localhost:9200/books/doc/_search' -d '{"query": {"bool": {"must": [{ "match": { "title": "Search" }},{ "match": { "content": "Elasticsearch" }}],"filter": [{ "term": { "status": "published" }},{ "range": { "publish_date": { "gte": "2015-01-01" }}}]}}}'
ࢄߏɺ εέʔϧ
Basic terms• ΠϯσοΫε‒ σʔλͷཧతͳू߹ɻ RDBͷσʔλϕʔεͷΑ͏ͳͷLogical• ϨϓϦέʔγϣϯ• ಡΈࠐΈͷεέʔϥϏϦςΟ্• SPOFͷղফ• γϟʔσΟϯά• ෳϚγϯσʔλΛׂ ॻ͖ࠐΈͷεέʔϥϏϦςΟ্ σʔλϑϩʔ੍ޚ!45
γϟʔυͱϨϓϦΧ!46node 1ordersproducts141 223curl -X PUT localhost:9200/orders -d '{"settings.index.number_of_shards" : 4"settings.index.number_of_replicas" : 1}'curl -X PUT localhost:9200/products -d '{"settings.index.number_of_shards" : 2"settings.index.number_of_replicas" : 0}'
γϟʔυͱϨϓϦΧ!47node 1ordersproducts141node 2ordersproducts223 41 23
ࣗಈతͳࢄ!48node 1ordersproducts2141node 2ordersproducts22node 3ordersproducts3 413
શจݕࡧͱʁ
શจݕࡧͱʁ• શจݕࡧʢFull text searchʣͱɺίϯϐϡʔλʹ͓͍ͯɺෳͷจॻʢϑΝΠϧʣ͔ΒಛఆͷจࣈྻΛݕࡧ͢Δ͜ͱɻʮϑΝΠϧ໊ݕࡧʯʮ୯ҰϑΝΠϧͷจࣈྻݕࡧʯͱҟͳΓɺʮෳจॻʹ·͕ͨͬͯɺจॻʹؚ·ΕΔશจΛରͱͨ͠ݕࡧʯͱ͍͏ҙຯͰ༻͞ΕΔɻ ʢWikipediaΑΓʣ!50
༻ޠ• ΠϯσοΫεݕࡧΤϯδϯ͕ݕࡧʹ༻͢Δσʔλͷอଘઌ• υΩϡϝϯτʢจॻʣ‒ ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ• ϑΟʔϧυ‒ υΩϡϝϯτʹؚ·ΕΔଐੑ• ΫΤϦ‒ ݕࡧ݅ɺݕࡧࣜ!51
༻ޠ• εΩʔϚ‒ υΩϡϝϯτͷߏΛఆٛ͢Δͷ• λʔϜʢTermʣɺτʔΫϯʢTokenʣ‒ ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ‒ จষΛҰఆͷ๏ଇͰ۠ͬͨ୯ޠ‒ ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲؚΉ!52
υΩϡϝϯτͷొ!5312ΧπΦαβΤͷఋαβΤϫΧϝͷ࢞υΩϡϝϯτͷొ
υΩϡϝϯτͷొ!5412ΧπΦαβΤͷఋαβΤϫΧϝͷ࢞12ΧπΦαβΤͷͷαβΤϫΧϝఋ࢞υΩϡϝϯτͷొ୯ޠʹׂ
υΩϡϝϯτͷొ!5512ΧπΦαβΤͷఋαβΤϫΧϝͷ࢞12ΧπΦαβΤͷͷαβΤϫΧϝఋ࢞ΧπΦαβΤ11 2 ͷ࢞ϫΧϝ 21 21 21ఋ2υΩϡϝϯτͷొ୯ޠʹׂ୯ޠ͔Βidͷྻ͕Ҿ͚ΔΑ͏ʹ
ݕࡧ!56ΧπΦαβΤ 2 ͷ࢞ϫΧϝ 21 21 21ఋ2ݕࡧ݅ೖྗΧπΦɹαβΤ11
ݕࡧ!57ΧπΦαβΤ 2 ͷ࢞ϫΧϝ 21 21 21ఋ2ΧπΦ αβΤANDݕࡧ݅ೖྗݕࡧ݅ͷύʔε ݕࡧΫΤϦԽΧπΦɹαβΤ11
ݕࡧ!58ΧπΦαβΤ 2 ͷ࢞ϫΧϝ 21 21 21ఋ2ΧπΦ αβΤANDݕࡧ݅ೖྗݕࡧ݅ͷύʔε ݕࡧΫΤϦԽΧπΦɹαβΤ11
ݕࡧ!59ΧπΦαβΤ 2 ͷ࢞ϫΧϝ 21 21 21ఋ2ΧπΦ αβΤANDݕࡧ݅ೖྗݕࡧ݅ͷύʔε ݕࡧΫΤϦԽΧπΦɹαβΤ11
ݕࡧ!60ΧπΦαβΤ11 2 ͷ࢞ϫΧϝ 21 21 21ఋ2ΧπΦ αβΤANDݕࡧ݅ೖྗݕࡧ݅ͷύʔε ݕࡧΫΤϦԽΧπΦɹαβΤ
ݕࡧ!61ΧπΦαβΤ11 2 ͷ࢞ϫΧϝ 21 21 21ఋ2ΧπΦ αβΤANDݕࡧ݅ೖྗݕࡧ݅ͷύʔε ݕࡧΫΤϦԽΧπΦɹαβΤ
ݕࡧ!62ΧπΦαβΤ11 2 ͷ࢞ϫΧϝ 21 21 21ఋ2ΧπΦ αβΤANDݕࡧ݅ೖྗݕࡧ݅ͷύʔε ݕࡧΫΤϦԽΧπΦɹαβΤ
୯ޠͷ۠Γํ• ӳޠͷ߹I am speaking Introduction Elasticsearch. • ຊޠͷ߹ࢲೖElasticsearchʹ͍͍ͭͯͯ͠Δɻ !63
୯ޠͷ۠Γํ• ӳޠͷ߹I am speaking Introduction Elasticsearch. εϖʔε͕ΕͱΘ͔Δ• ຊޠͷ߹ࢲೖElasticsearchʹ͍͍ͭͯͯ͠Δɻ Ͳ͜Ͱ۠ΕΑ͍ʁ64
N-Gramͱܗଶૉղੳ• సஔΠϯσοΫεͷΩʔͷ࡞Γํ‒ ຊޠ୯ޠͷΕ͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷΩʔओʹ࣍ͷ̎ͭͷख๏Ͱ࡞• N-Gram‒ NจࣈͣͭจষΛ۠Δ• ܗଶૉղੳ‒ ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠Δ!65
ܗଶૉղੳ• ϝϦοτɿ‒ ҙຯͷ͋Δ୯ޠͷΕ ࢺใΛݩʹՃॲཧ͕ՄೳʢޠװมͳͲʣ• σϝϦοτɿ‒ ৽ޠʢະޠʣʹऑ͍→ࣙॻϕʔεͷ߹ɺࣙॻʹͳ͍୯ޠݕग़ෆೳɻ!66ΧπΦαβΤͷఋΧπΦ ͷαβΤ ఋ
N-Gram• ϝϦοτɿ‒ ະޠʹରԠՄೳ• σϝϦοτɿ‒ ΠϯσοΫεංେԽ‒ ࢺใʹجͮ͘ॲཧ͕ෆՄೳ!67ΧπΦαβΤͷఋΧπ πΦ Φ α αβ βΤ Τͷ ͷఋ
ͦͷଞͷػೳ
elasticsearch͞·͟·ͳܗࣜͷσʔλͰGeoݕࡧՄೳ ҢܦɺGeoHashɺGeoShape…GEO
Ecosystem• Plugins‒ ϓϥάΠϯʹΑΔػೳͷՃ• ΫϥΠΞϯτϥΠϒϥϦ• Java, Ruby, python, php, perl, javascript, .NET• Scala, clojure, go!70
Elasticsearch - The Definitive guide http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html71ৄ͘͠Γ͍ͨํ
!72
73KibanaWindow into the Elastic StackՄࢹԽͱੳ ཧۭؒ ΧελϚΠζͱϨϙʔτͷڞ༗άϥϑ୳ࡧ Elastic StackͷηΩϡΞͳΞΫηεͱཧΧελϜAppsͷ࡞
!74Kibana 6
!75σϞσʔλೖ͔ΒՄࢹԽ·Ͱ
!76ຊ֨తʹղੳΛߦ͏ʹʁ
!77Elastic StackͷߏBeatsLogFilesMetricsWireDataKibanaInstancesElasticsearchNodes
!78Elastic StackͷߏBeatsLogFilesMetricsWireDatayour{beat}KibanaInstancesKafkaDistributedMessageQueueNotificationQueues Storage MetricsDataStoreWebAPIsSocial SensorsElasticsearchNodesLogstashNodes
!79
80LogstashσʔλՃύΠϓϥΠϯશͯͷܗࣜɺαΠζͱσʔλιʔεͷೖύʔεͱಈతͳσʔλม͋ΒΏΔग़ྗʹσʔλసૹ҆શͰ҉߸Խ͞Εͨ σʔλೖྗಠࣗͷύΠϓϥΠϯॲཧͷ࡞200Ҏ্ͷϓϥάΠϯ
Logstash in 10 seconds• ϩάɾσʔλͷऩूɾཧ• ऩूɺύʔεɾՃɺૹग़• ΦʔϓϯιʔεɿApache License 2.0• Ruby app (JRuby)!81
Logstash architecture!82Input OutputFilter? ?collect and split alter and enrich store and visualize
ઃఆ83input {…}filter {…}output {…}
ઃఆɿinput84input {file {path => “/Users/johtani/sample/*_log"start_position => "beginning"}}
1ߦ1σʔλ189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101Firefox/5.0"85
ઃఆɿfilter86filter {grok {match => { "message" => "%{COMBINEDAPACHELOG}" }break_on_match => false}date {match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]locale => en}geoip { source => ["clientip"] }useragent {source => "agent"target => "useragent"}}
ύʔε!87189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0"{…"@timestamp": "2015-04-10T09:07:49.325Z","clientip": "189.120.xx.xx","ident": "-","auth": "-","timestamp": "02/Dec/2014:12:18:29 +0900","verb": "GET","request": "/manager/html",…"agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
ઃఆɿfilter!88filter {grok {match => { "message" => "%{COMBINEDAPACHELOG}" }break_on_match => false}date {match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]locale => en}geoip { source => ["clientip"] }useragent {source => "agent"target => "useragent"}}
ͷύʔε89{…"@timestamp": "2015-04-10T09:07:49.325Z",…"timestamp": "02/Dec/2014:12:18:29 +0900",…}{…"@timestamp": "2014-12-02T03:18:29.000Z",…"timestamp": "02/Dec/2014:12:18:29 +0900",…}
ઃఆɿfilter!90filter {grok {match => { "message" => "%{COMBINEDAPACHELOG}" }break_on_match => false}date {match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]locale => en}geoip { source => ["clientip"] }useragent {source => "agent"target => "useragent"}}
IP͔ΒҢܦͳͲ༩91"clientip": "189.120.xx.xx","clientip": "189.120.xx.xx","geoip": {"ip": “189.120.xxx.xxx”,…"country_name": "Brazil","continent_code": "SA","region_name": "27","city_name": "São Paulo","latitude":
ઃఆɿfilter!92filter {grok {match => { "message" => "%{COMBINEDAPACHELOG}" }break_on_match => false}date {match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]locale => en}geoip { source => ["clientip"] }useragent {source => "agent"target => "useragent"}}
ϢʔβΤʔδΣϯτͷύʔε93"agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0\"""agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0\"""useragent": {"name": "Firefox","os": "Windows XP","os_name": "Windows XP","device": "Other","major": "5","minor": "0"
ઃఆɿoutput94output {elasticsearch {hosts => ["localhost"]index => “demo_access_log-%{+YYYY.MM.dd}”}}
!95͞Βʹ׆༻͢Δʹʁ
!96elasticsearch-hadoop-• D E H• PD ecdER• g D• CH• Ca M DMSD FERC
!97
!98
!99
ͦͷଞͷ͍ํ!100
!101σʔλͷొํ๏• Kibanaͷαϯϓϧσʔλʢ6.4͔Βʣ• LogstashͰJDBC input• LogstashͰCSV• FilebeatͰΞΫηεϩά• MetricbeatͰϝτϦοΫ• PacketbeatͰMySQL/PostgreSQLͷύέοτղੳ
!102Kibanaͷαϯϓϧσʔλʢ>= 6.4.0ʣ
!103ϫϯΫϦοΫͰσʔλొ
!104LogstashͰJDBC InputKibanaInstancesDataStoreElasticsearchNodesLogstashNodes
!105JDBC Input
!106LogstashͰCSVKibanaInstancesCSV FileElasticsearchNodesLogstashNodes
!107CSV filter
!108FilebeatͰΞΫηεϩάBeatsLogFilesKibanaInstancesElasticsearchNodes
• 2ͭͷElasticsearchϓϥάΠϯΛΠϯετʔϧͯ͠ElasticsearchΛىಈ• Filebeatͷapache2ϞδϡʔϧΛ༗ޮԽ• modules.d/apache2.ymlʹΞΫηεϩάͷύεΛઃఆ• setupίϚϯυΛ࣮ߦ͔ͯ͠ΒFilebeatΛىಈ!109FilebeatͰΞΫηεϩά
MetricbeatͰϝτϦοΫBeatsMetricsKibanaInstancesElasticsearchNodes
• MetricbeatͷsystemϞδϡʔϧΛ༗ޮԽ• setupίϚϯυΛ࣮ߦ͔ͯ͠ΒFilebeatΛىಈ!111MetricbeatͰϝτϦοΫ
!112PacketbeatͰMySQLɺPostgreSQLͷύέοτղੳBeatsWireDataKibanaInstancesElasticsearchNodes
!113ࢀߟจݙ• Elasticsearch - The Definitive guide‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html• ॻ੶ʢຊޠʣ‒ σʔλੳج൫ߏஙೖ‒ Elasticsearch࣮ફΨΠυ
!114ࢀߟαΠτ• Ϣʔεέʔε• https://www.elastic.co/use-cases• DiscussʢWebϑΥʔϥϜʣ• https://discuss.elastic.co• Elastic{ON}ͷϏσΦͱࢿྉ• https://www.elastic.co/elasticon/videos• αϙʔτϝχϡʔ• https://www.elastic.co/subscriptions
Thank you!● Web : https://www.elastic.co/jp/● Forums : https://discuss.elastic.co/● Twitter : @johtani
johtani
keio_smilab
mitsuakitohei
nishiharatsubasa
adavis
huffon
karawoo
yosuke_furukawa
mixi_engineers
junjunjunk
lapis2411
shotatsuge
whywaita
keavy
mojombo
eileencodes
addyosmani
chriscoyier
dougneiner
ddemaree
sachag
cassininazir
quramy
62gerente
denniskardys