OSC Hokkaido 2018での発表資料です。
!12018/07/06Developer Advocate at ElasticJun Ohtani @johtani༷ʑͳϝτϦΫεϩάΛूΊͯγεςϜղੳ - Elastic Stackͷೖͱ׆༻ -
View Slide
!2
!3ΞδΣϯμ• ϝτϦΫεʗϩάͱʁ• γεςϜϝτϦΫεղੳɺϩάղੳΛࢼ͠ʹͬͯΈΑ͏• Beats - Elasticsearch - KibanaͰղੳ• ຊ֨తʹղੳΛΔʹʁ• LogstashͰϩάϝτϦΫεΛதܧɾू• ͞Βʹ৭ʑࢼͯ͠ΈΔʹʁ
!4about• Me, Jun Ohtani / Developer Advocate‒ lucene-gosenίϛολʔ‒ σʔλੳج൫ߏஙೖ ڞஶ‒ http://blog.johtani.info • Elasticsearch, founded in 2012‒ Products: Elasticsearch, Logstash, Kibana, Beats Elastic APM, X-Pack, Elastic Cloud, Swiftype Professional services: Support & development subscriptions Trainings, Consulting, SaaS
!5ͲΜͳϝτϦΫεɺ ϩάΛूΊ͍ͯ·͔͢ʁ
!6ϝτϦΫε• CPUɺϝϞϦ༻ɺσΟεΫ༻• ΞΫηεɺωοτϫʔΫసૹྔ• Ԡ࣌ؒ• ίωΫγϣϯ• τϥϯβΫγϣϯɺച্• ίϯςφͷ্ͷ֤छϝτϦΫε
!7ϩά• ೝূϩά• γεςϜϩά• ΞϓϦέʔγϣϯϩά• Slow log• ΞΫηεϩά• ίϯςφͷதͷϩά
!8Ͱ͖ΕϩάͱϝτϦΫεΛ ·ͱΊͯ1ͭͷը໘Ͱ ݟ͍ͨͰ͢ΑͶʁ
!9Elastic Stack
10Elastic Stack100% ΦʔϓϯιʔεʮΤϯλʔϓϥΠζ൛ʯແ͠όʔδϣϯ 5.0Ͱશ౷Ұ
!11X-Pack؆୯ʹΠϯετʔϧElastic StackΛ֦ுαϒεΫϦϓγϣϯʹؚ·ΕΔSecurityAlertingMonitoringReportingGraphMachine Learning
ఆܕͷϝτϦΫε/ϩάղੳΛElastic StackͰ
!13ϝτϦΫεɾϩάͷੳʢ؆қ൛ʣσʔλ Import Parse/ Store/SearchVisualize
!14
15Beatsܰྔσʔλγούʔιʔε͔ΒσʔλΛసૹ సૹ͠Elasticsearchʹू มͱύʔεͷͨΊLogstashʹసૹElastic CloudʹసૹLibbeat: ΧελϜbeatsͷͨΊͷAPIϑϨʔϜϫʔΫ30Ҏ্ͷίϛϡχςΟbeats
The Beats familyHeartbeatUptime monitoringFilebeatLog filesWinlogbeatWindows Event LogsPacketbeatNetwork data+40communityBeatsMetricbeatMetricsAuditbeatAudit data
Collect systemand applicationmetricsMetricbeat
lots of modulesMetricbeat
tail log fromfileFilebeat
many modulesFilebeat
Capture thePacketPacketbeat
Welcometo 1998winlogbeat
Nowwinlogbeat
!25• Kubernetes module in Metricbeat‒ CPU, memory, ωοτϫʔΫసૹྔͳͲ• add_docker_metadataϓϩηοα‒ Container ID, name, image, labels• add_kubernetes_metadataϓϩηοα‒ Pod name, pod namespace, container name, podlabelsBeats <3 ίϯςφDockerKubernetesͰͷσϓϩΠΛ؆୯ʹ
!26
27ElasticsearchHeart of the Elastic Stackࢄܕɺεέʔϥϒϧ ߴՄ༻ੑ Ϛϧνςφϯτ։ൃऀϑϨϯυϦʔ ϦΞϧλΠϜɺશจݕࡧ ΞάϦήʔγϣϯ
Elasticsearchͱʁ
!29ϑϦʔϫʔυݕࡧ
!30ߜΓࠐΈ
!31ϋΠϥΠτ
!32ιʔτ
!33ϖʔδϯά
!34ूܭ
!35αδΣετ
!36Elasticsearch in 10 seconds• εΩʔϚϑϦʔɺࢄυΩϡϝϯτετΞɺREST & JSON• Φʔϓϯιʔε: Apache License 2.0• ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ• JavaͰ࣮ɻ֦ு༰қ
؆୯ͳCRUD
σʔλొ38curl -XPUT localhost:9200/books/doc/1 -d '{"title" : "Elasticsearch - The definitive guide","authors" : "Clinton Gormley","started" : "2013-02-04","pages" : 230}'
σʔλߋ৽39curl -XPUT localhost:9200/books/doc/1 -d '{"title" : "Elasticsearch - The definitive guide","authors" : [ "Clinton Gormley", "Zachary Tong" ],"started" : "2013-02-04","pages" : 230}'
σʔλআ!40curl -X DELETE localhost:9200/books/doc/1σʔλͷऔಘcurl —X GET localhost:9200/books/doc/1curl —X GET localhost:9200/books/doc/1/_source
ݕࡧ - Query DSL!41curl -XGET ‘localhost:9200/books/doc/_search' -d '{"query": {"bool": {"must": [{ "match": { "title": "Search" }},{ "match": { "content": "Elasticsearch" }}],"filter": [{ "term": { "status": "published" }},{ "range": { "publish_date": { "gte": "2015-01-01" }}}]}}}'
ࢄߏɺ εέʔϧ
Basic terms• ΠϯσοΫε‒ σʔλͷཧతͳू߹ɻ RDBͷσʔλϕʔεͷΑ͏ͳͷLogical• ϨϓϦέʔγϣϯ• ಡΈࠐΈͷεέʔϥϏϦςΟ্• SPOFͷղফ• γϟʔσΟϯά• ෳϚγϯσʔλΛׂ ॻ͖ࠐΈͷεέʔϥϏϦςΟ্ σʔλϑϩʔ੍ޚ!43
γϟʔυͱϨϓϦΧ!44node 1ordersproducts141 223curl -X PUT localhost:9200/orders -d '{"settings.index.number_of_shards" : 4"settings.index.number_of_replicas" : 1}'curl -X PUT localhost:9200/products -d '{"settings.index.number_of_shards" : 2"settings.index.number_of_replicas" : 0}'
γϟʔυͱϨϓϦΧ!45node 1ordersproducts141node 2ordersproducts223 41 23
ࣗಈతͳࢄ!46node 1ordersproducts2141node 2ordersproducts22node 3ordersproducts3 413
ͦͷଞͷػೳ
elasticsearch͞·͟·ͳܗࣜͷσʔλͰGeoݕࡧՄೳ ҢܦɺGeoHashɺGeoShape…GEO
Ecosystem• Plugins‒ ϓϥάΠϯʹΑΔػೳͷՃ• ΫϥΠΞϯτϥΠϒϥϦ• Java, Ruby, python, php, perl, javascript, .NET• Scala, clojure, go!49
Elasticsearch - The Definitive guide http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html50ৄ͘͠Γ͍ͨํ
!51
52KibanaWindow into the Elastic StackՄࢹԽͱੳ ཧۭؒ ΧελϚΠζͱϨϙʔτͷڞ༗άϥϑ୳ࡧ Elastic StackͷηΩϡΞͳΞΫηεͱཧΧελϜAppsͷ࡞
!53Kibana 6
!54σϞσʔλೖ͔ΒՄࢹԽ·Ͱ
ຊ֨తʹղੳΛߦ͏ʹʁ
!56
57LogstashσʔλՃύΠϓϥΠϯશͯͷܗࣜɺαΠζͱσʔλιʔεͷೖύʔεͱಈతͳσʔλม͋ΒΏΔग़ྗʹσʔλసૹ҆શͰ҉߸Խ͞Εͨ σʔλೖྗಠࣗͷύΠϓϥΠϯॲཧͷ࡞200Ҏ্ͷϓϥάΠϯ
Logstash architecture!58Input OutputFilter? ?collect and split alter and enrich store and visualize
ઃఆ59input {…}filter {…}output {…}
1ߦ1σʔλ189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101Firefox/5.0"60
ઃఆɿfilter61filter {grok {match => { "message" => "%{COMBINEDAPACHELOG}" }break_on_match => false}date {match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]locale => en}geoip { source => ["clientip"] }useragent {source => "agent"target => "useragent"}}
ύʔε!62189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0"{…"@timestamp": "2015-04-10T09:07:49.325Z","clientip": "189.120.xx.xx","ident": "-","auth": "-","timestamp": "02/Dec/2014:12:18:29 +0900","verb": "GET","request": "/manager/html",…"agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
ઃఆɿfilter!63filter {grok {match => { "message" => "%{COMBINEDAPACHELOG}" }break_on_match => false}date {match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]locale => en}geoip { source => ["clientip"] }useragent {source => "agent"target => "useragent"}}
ͷύʔε64{…"@timestamp": "2015-04-10T09:07:49.325Z",…"timestamp": "02/Dec/2014:12:18:29 +0900",…}{…"@timestamp": "2014-12-02T03:18:29.000Z",…"timestamp": "02/Dec/2014:12:18:29 +0900",…}
ઃఆɿfilter!65filter {grok {match => { "message" => "%{COMBINEDAPACHELOG}" }break_on_match => false}date {match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]locale => en}geoip { source => ["clientip"] }useragent {source => "agent"target => "useragent"}}
IP͔ΒҢܦͳͲ༩66"clientip": "189.120.xx.xx","clientip": "189.120.xx.xx","geoip": {"ip": “189.120.xxx.xxx”,…"country_name": "Brazil","continent_code": "SA","region_name": "27","city_name": "São Paulo","latitude":
ઃఆɿfilter!67filter {grok {match => { "message" => "%{COMBINEDAPACHELOG}" }break_on_match => false}date {match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]locale => en}geoip { source => ["clientip"] }useragent {source => "agent"target => "useragent"}}
ϢʔβΤʔδΣϯτͷύʔε68"agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0\"""agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0\"""useragent": {"name": "Firefox","os": "Windows XP","os_name": "Windows XP","device": "Other","major": "5","minor": "0"
ͦͷ΄͔ʹʁ
!70elasticsearch-hadoop-• D E H• PD ecdER• g D• CH• Ca M DMSD FERC
!71
!72X-Pack؆୯ʹΠϯετʔϧElastic StackΛ֦ுαϒεΫϦϓγϣϯʹؚ·ΕΔSecurityAlertingMonitoringReportingGraphMachine Learning
!73
!74ࢀߟจݙ• Elasticsearch - The Definitive guide‒ http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html• ॻ੶ʢຊޠʣ‒ σʔλੳج൫ߏஙೖ‒ Elasticsearch࣮ફΨΠυ
!75ࢀߟαΠτ• Ϣʔεέʔε• https://www.elastic.co/use-cases• DiscussʢWebϑΥʔϥϜʣ• https://discuss.elastic.co• Elastic{ON}ͷϏσΦͱࢿྉ• https://www.elastic.co/elasticon/videos• αϙʔτϝχϡʔ• https://www.elastic.co/subscriptions
Thanks for listening!Q & AWe’re hiring!https://www.elastic.co/about/careers/We’re helping!https://www.elastic.co/subscriptionshttp://training.elastic.co
johtani
kneath
holman
chrislema
trallard
rmw
zakiwarfel
tmm1
robhawkes
maltzj
thekraken
tammielis
jlugia