第24回Elasticsearch勉強会「入門編」で使用した資料です。 #elasticsearchjp
!1Jun OhtaniCommunity Engineer @Elastic Twitter: @johtaniElastic Stackೖ
View Slide
!2about• Me, Jun Ohtani / Community Engineer‒ lucene-gosenίϛολʔ‒ σʔλੳج൫ߏஙೖ ڞஶ‒ http://blog.johtani.info • Elastic, founded in 2012‒ Products: Elasticsearch, Logstash, Kibana, Beats Elastic APM, X-Pack, Elastic Cloud, Swiftype Professional services: Support & development subscriptions Trainings, Consulting, SaaS
!3ΞδΣϯμ• Ξϯέʔτ• Ϣʔεέʔεհ• ϓϩμΫτհ• σϞ ˍ QA
Ϣʔεέʔε!4
Search andanalytics, it allstarted here!5More than 60% of ourcustomers have a searchor analytics use case
!6
Logs Logs Logs, many devices, many systemsMore than 40% of our customers use our products for operational log analysis!7
ΠϯϑϥετϥΫνϟɺWebαʔόʔɺΞϓϦέʔγϣϯ͔Βຖ ऩू͢Δϩά1.2TB Ҏ্ɹ8
Sniff sniff sniff, find the bad actors in your data200% YoY growth insecurity use cases withour products!9
We mine and analyze4 billion events every day todetect security hacks and threats.!10
75% of our customersuse our products formultiple use casesSEARCHSECURITYCUSTOM APPSMETRICSOPERATIONAL ANALYTICSLOG ANALYTICS!11
!121,000+ developers use theElastic Stack for use casesfrom trade tracking to creatingnew HR and compliance apps.
!13Elastic Stack
ElasticStackอଘɺݕࡧɺੳElasticsearchՄࢹԽɺཧKibanaBeats ΠϯδΣετLogstash
MetricsLoggingAPMSite SearchApplicationSearchBusiness AnalyticsEnterprise SearchSecurity AnalyticsFuture ιϦϡʔγϣϯElasticStackอଘɺݕࡧɺੳՄࢹԽɺཧΠϯδΣετKibanaElasticsearchBeats Logstash
MetricsLoggingAPMSite SearchApp SearchBusiness AnalyticsEnterprise SearchSecurity AnalyticsFuture ιϦϡʔγϣϯSaaSElastic CloudSelf ManagedElastic Cloud Enterprise StandaloneσϓϩΠElasticStackอଘɺݕࡧɺੳՄࢹԽɺཧΠϯδΣετKibanaElasticsearchBeats Logstash
ElasticStackอଘɺݕࡧɺੳElasticsearchՄࢹԽɺཧKibanaBeats ΠϯδΣετLogstashMetricsLoggingAPMSite SearchApplicationSearchBusiness AnalyticsEnterprise SearchSecurity AnalyticsFuture ιϦϡʔγϣϯSaaSElastic CloudSelf ManagedElastic Cloud Enterprise StandaloneσϓϩΠ
!18Elastic StackͷߏBeatsLogFilesMetricsWireDatayour{beat}KibanaInstancesKafkaDistributedMessageQueueNotificationQueues Storage MetricsDataStoreWebAPIsSocial SensorsElasticsearchNodesLogstashNodes
!19
20Beatsܰྔσʔλγούʔιʔε͔ΒσʔλΛసૹ సૹ͠Elasticsearchʹू มͱύʔεͷͨΊLogstashʹసૹElastic CloudʹసૹLibbeat: ΧελϜbeatsͷͨΊͷAPIϑϨʔϜϫʔΫ30Ҏ্ͷίϛϡχςΟbeats
The Beats familyHeartbeatUptime monitoringFilebeatLog filesWinlogbeatWindows Event LogsPacketbeatNetwork data+40communityBeatsMetricbeatMetricsAuditbeatAudit data
Collect systemand applicationmetricsMetricbeat
lots of modulesMetricbeat
tail log fromfileFilebeat
many modulesFilebeat
Capture thePacketPacketbeat
Welcometo 1998winlogbeat
Nowwinlogbeat
!30
31LogstashσʔλՃύΠϓϥΠϯશͯͷܗࣜɺαΠζͱσʔλιʔεͷೖύʔεͱಈతͳσʔλม͋ΒΏΔग़ྗʹσʔλసૹ҆શͰ҉߸Խ͞Εͨ σʔλೖྗಠࣗͷύΠϓϥΠϯॲཧͷ࡞200Ҏ্ͷϓϥάΠϯ
Logstash in 10 seconds• ϩάɾσʔλͷऩूɾཧ• ऩूɺύʔεɾՃɺૹग़• ΦʔϓϯιʔεɿApache License 2.0• Ruby app (JRuby)!32
Logstash architecture!33Input OutputFilter? ?collect and split alter and enrich store and visualize
ઃఆ34input {…}filter {…}output {…}
ઃఆɿinput35input {file {path => “/Users/johtani/sample/*_log"start_position => "beginning"}}
1ߦ1σʔλ189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101Firefox/5.0"36
ઃఆɿfilter37filter {grok {match => { "message" => "%{COMBINEDAPACHELOG}" }break_on_match => false}date {match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]locale => en}geoip { source => ["clientip"] }useragent {source => "agent"target => "useragent"}}
ύʔε!38189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/1.1"404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0"{…"@timestamp": "2015-04-10T09:07:49.325Z","clientip": "189.120.xx.xx","ident": "-","auth": "-","timestamp": "02/Dec/2014:12:18:29 +0900","verb": "GET","request": "/manager/html",…"agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/
ઃఆɿfilter!39filter {grok {match => { "message" => "%{COMBINEDAPACHELOG}" }break_on_match => false}date {match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]locale => en}geoip { source => ["clientip"] }useragent {source => "agent"target => "useragent"}}
ͷύʔε40{…"@timestamp": "2015-04-10T09:07:49.325Z",…"timestamp": "02/Dec/2014:12:18:29 +0900",…}{…"@timestamp": "2014-12-02T03:18:29.000Z",…"timestamp": "02/Dec/2014:12:18:29 +0900",…}
ઃఆɿfilter!41filter {grok {match => { "message" => "%{COMBINEDAPACHELOG}" }break_on_match => false}date {match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]locale => en}geoip { source => ["clientip"] }useragent {source => "agent"target => "useragent"}}
IP͔ΒҢܦͳͲ༩42"clientip": "189.120.xx.xx","clientip": "189.120.xx.xx","geoip": {"ip": “189.120.xxx.xxx”,…"country_name": "Brazil","continent_code": "SA","region_name": "27","city_name": "São Paulo","latitude":
ઃఆɿfilter!43filter {grok {match => { "message" => "%{COMBINEDAPACHELOG}" }break_on_match => false}date {match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"]locale => en}geoip { source => ["clientip"] }useragent {source => "agent"target => "useragent"}}
ϢʔβΤʔδΣϯτͷύʔε44"agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0\"""agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0\"""useragent": {"name": "Firefox","os": "Windows XP","os_name": "Windows XP","device": "Other","major": "5","minor": "0"
ઃఆɿoutput45output {elasticsearch {hosts => ["localhost"]index => “demo_access_log-%{+YYYY.MM.dd}”}}
!46
47ElasticsearchHeart of the Elastic Stackࢄܕɺεέʔϥϒϧ ߴՄ༻ੑ Ϛϧνςφϯτ։ൃऀϑϨϯυϦʔ ϦΞϧλΠϜɺશจݕࡧ ΞάϦήʔγϣϯ
Elasticsearchͱʁ
ϑϦʔϫʔυݕࡧ!49
ߜΓࠐΈ!50
ϋΠϥΠτ!51
ιʔτ!52
ϖʔδϯά!53
ूܭ!54
αδΣετ!55
Elasticsearch in 10 seconds• εΩʔϚϑϦʔɺࢄυΩϡϝϯτετΞɺREST & JSON• Φʔϓϯιʔε: Apache License 2.0• ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ• JavaͰ࣮ɻ֦ு༰қ!56
Powerful Search at Scale!57
؆୯ͳCRUD
σʔλొ59curl -XPUT localhost:9200/books/book/1 -d '{"title" : "Elasticsearch - The definitive guide","authors" : "Clinton Gormley","started" : "2013-02-04","pages" : 230}'
σʔλߋ৽60curl -XPUT localhost:9200/books/book/1 -d '{"title" : "Elasticsearch - The definitive guide","authors" : [ "Clinton Gormley", "Zachary Tong" ],"started" : "2013-02-04","pages" : 230}'
σʔλআ!61curl -X DELETE localhost:9200/books/book/1σʔλͷऔಘcurl —X GET localhost:9200/books/book/1curl —X GET localhost:9200/books/book/1/_source
ݕࡧ!62curl -XGET localhost:9200/books/_search?q=elasticsearch{"took" : 2, "timed_out" : false,"_shards" : { "total" : 5, "successful" : 5, "failed" : 0 },"hits" : {"total" : 1, "max_score" : 0.076713204,"hits" : [ {"_index" : “books", "_type" : “book", "_id" : "1","_score" : 0.076713204, "_source" : {"title" : "Elasticsearch - The definitive guide","authors" : [ "Clinton Gormley", "Zachary Tong" ],"started" : “2013-02-04", "pages" : 230}} ]
ݕࡧ - Query DSL!63curl -XGET ‘localhost:9200/books/book/_search' -d '{"query": {"filtered" : {"query" : {"match": {"text" : {"query" : “To Be Or Not To Be","cutoff_frequency" : 0.01}}},"filter" : {"range": {"price": {"gte": 20.0"lte": 50.0
ࢄߏɺ εέʔϧ
Basic terms• ΠϯσοΫε‒ σʔλͷཧతͳू߹ɻ RDBͷσʔλϕʔεͷΑ͏ͳͷLogical• ϨϓϦέʔγϣϯ• ಡΈࠐΈͷεέʔϥϏϦςΟ্• SPOFͷղফ• γϟʔσΟϯά• ෳϚγϯσʔλΛׂ ॻ͖ࠐΈͷεέʔϥϏϦςΟ্ σʔλϑϩʔ੍ޚ!65
γϟʔυͱϨϓϦΧ!66node 1ordersproducts141 223curl -X PUT localhost:9200/orders -d '{"settings.index.number_of_shards" : 4"settings.index.number_of_replicas" : 1}'curl -X PUT localhost:9200/products -d '{"settings.index.number_of_shards" : 2"settings.index.number_of_replicas" : 0}'
γϟʔυͱϨϓϦΧ!67node 1ordersproducts141node 2ordersproducts223 41 23
ࣗಈతͳࢄ!68node 1ordersproducts2141node 2ordersproducts22node 3ordersproducts3 413
શจݕࡧͱʁ
શจݕࡧͱʁ• શจݕࡧʢFull text searchʣͱɺίϯϐϡʔλʹ͓͍ͯɺෳͷจॻʢϑΝΠϧʣ͔ΒಛఆͷจࣈྻΛݕࡧ͢Δ͜ͱɻʮϑΝΠϧ໊ݕࡧʯʮ୯ҰϑΝΠϧͷจࣈྻݕࡧʯͱҟͳΓɺʮෳจॻʹ·͕ͨͬͯɺจॻʹؚ·ΕΔશจΛରͱͨ͠ݕࡧʯͱ͍͏ҙຯͰ༻͞ΕΔɻ ʢWikipediaΑΓʣ!70
༻ޠ• ΠϯσοΫεݕࡧΤϯδϯ͕ݕࡧʹ༻͢Δσʔλͷอଘઌ• υΩϡϝϯτʢจॻʣ‒ ݕࡧΤϯδϯʹอଘ͞Εͨσʔλ• ϑΟʔϧυ‒ υΩϡϝϯτʹؚ·ΕΔଐੑ• ΫΤϦ‒ ݕࡧ݅ɺݕࡧࣜ!71
༻ޠ• εΩʔϚ‒ υΩϡϝϯτͷߏΛఆٛ͢Δͷ• λʔϜʢTermʣɺτʔΫϯʢTokenʣ‒ ΠϯσοΫεͷΩʔʹͳΔ୯ޠʢจࣈྻʣ‒ จষΛҰఆͷ๏ଇͰ۠ͬͨ୯ޠ‒ ୯ޠ͚ͩͰͳ͘ɺ୯ޠͷҐஔͳͲؚΉ!72
υΩϡϝϯτͷొ!7312ΧπΦαβΤͷఋαβΤϫΧϝͷ࢞υΩϡϝϯτͷొ
υΩϡϝϯτͷొ!7412ΧπΦαβΤͷఋαβΤϫΧϝͷ࢞12ΧπΦαβΤͷͷαβΤϫΧϝఋ࢞υΩϡϝϯτͷొ୯ޠʹׂ
υΩϡϝϯτͷొ!7512ΧπΦαβΤͷఋαβΤϫΧϝͷ࢞12ΧπΦαβΤͷͷαβΤϫΧϝఋ࢞ΧπΦαβΤ11 2 ͷ࢞ϫΧϝ 21 21 21ఋ2υΩϡϝϯτͷొ୯ޠʹׂ୯ޠ͔Βidͷྻ͕Ҿ͚ΔΑ͏ʹ
ݕࡧ!76ΧπΦαβΤ11 2 ͷ࢞ϫΧϝ 21 21 21ఋ2ݕࡧ݅ೖྗΧπΦɹαβΤ
ݕࡧ!77ΧπΦαβΤ11 2 ͷ࢞ϫΧϝ 21 21 21ఋ2ΧπΦ αβΤANDݕࡧ݅ೖྗݕࡧ݅ͷύʔε ݕࡧΫΤϦԽΧπΦɹαβΤ
ݕࡧ!78ΧπΦαβΤ11 2 ͷ࢞ϫΧϝ 21 21 21ఋ2ΧπΦ αβΤANDݕࡧ݅ೖྗݕࡧ݅ͷύʔε ݕࡧΫΤϦԽΧπΦɹαβΤ
ݕࡧ!79ΧπΦαβΤ11 2 ͷ࢞ϫΧϝ 21 21 21ఋ2ΧπΦ αβΤANDݕࡧ݅ೖྗݕࡧ݅ͷύʔε ݕࡧΫΤϦԽΧπΦɹαβΤ
ݕࡧ!80ΧπΦαβΤ11 2 ͷ࢞ϫΧϝ 21 21 21ఋ2ΧπΦ αβΤANDݕࡧ݅ೖྗݕࡧ݅ͷύʔε ݕࡧΫΤϦԽΧπΦɹαβΤ
ݕࡧ!81ΧπΦαβΤ11 2 ͷ࢞ϫΧϝ 21 21 21ఋ2ΧπΦ αβΤANDݕࡧ݅ೖྗݕࡧ݅ͷύʔε ݕࡧΫΤϦԽΧπΦɹαβΤ
ݕࡧ!82ΧπΦαβΤ11 2 ͷ࢞ϫΧϝ 21 21 21ఋ2ΧπΦ αβΤANDݕࡧ݅ೖྗݕࡧ݅ͷύʔε ݕࡧΫΤϦԽΧπΦɹαβΤ
୯ޠͷ۠Γํ• ӳޠͷ߹I am speaking Introduction Elasticsearch. • ຊޠͷ߹ࢲೖElasticsearchʹ͍͍ͭͯͯ͠Δɻ !83
୯ޠͷ۠Γํ• ӳޠͷ߹I am speaking Introduction Elasticsearch. εϖʔε͕ΕͱΘ͔Δ• ຊޠͷ߹ࢲೖElasticsearchʹ͍͍ͭͯͯ͠Δɻ Ͳ͜Ͱ۠ΕΑ͍ʁ84
N-Gramͱܗଶૉղੳ• సஔΠϯσοΫεͷΩʔͷ࡞Γํ‒ ຊޠ୯ޠͷΕ͕Θ͔Βͳ͍ͷͰɺసஔΠϯσοΫεͷΩʔओʹ࣍ͷ̎ͭͷख๏Ͱ࡞• N-Gram‒ NจࣈͣͭจষΛ۠Δ• ܗଶૉղੳ‒ ࣙॻͳͲΛ༻͍ͯҙຯͷ͋Δ୯ޠͰ۠Δ!85
ܗଶૉղੳ• ϝϦοτɿ‒ ҙຯͷ͋Δ୯ޠͷΕ ࢺใΛݩʹՃॲཧ͕ՄೳʢޠװมͳͲʣ• σϝϦοτɿ‒ ৽ޠʢະޠʣʹऑ͍→ࣙॻϕʔεͷ߹ɺࣙॻʹͳ͍୯ޠݕग़ෆೳɻ!86ΧπΦαβΤͷఋΧπΦ ͷαβΤ ఋ
N-Gram• ϝϦοτɿ‒ ະޠʹରԠՄೳ• σϝϦοτɿ‒ ΠϯσοΫεංେԽ‒ ࢺใʹجͮ͘ॲཧ͕ෆՄೳ!87ΧπΦαβΤͷఋΧπ πΦ Φ α αβ βΤ Τͷ ͷఋ
ͦͷଞͷػೳ
elasticsearch͞·͟·ͳܗࣜͷσʔλͰGeoݕࡧՄೳ ҢܦɺGeoHashɺGeoShape…GEO
Ecosystem• Plugins‒ ϓϥάΠϯʹΑΔػೳͷՃ• ΫϥΠΞϯτϥΠϒϥϦ• Java, Ruby, python, php, perl, javascript, .NET• Scala, clojure, go!90
Elasticsearch - The Definitive guide http://www.elastic.co/guide/en/elasticsearch/guide/current/index.html91ৄ͘͠Γ͍ͨํ
!92
93KibanaWindow into the Elastic StackՄࢹԽͱੳ ཧۭؒ ΧελϚΠζͱϨϙʔτͷڞ༗άϥϑ୳ࡧ Elastic StackͷηΩϡΞͳΞΫηεͱཧΧελϜAppsͷ࡞
σʔλͷొํ๏!94
!95σʔλͷొํ๏• Kibanaͷαϯϓϧσʔλʢ6.4͔Βʣ• LogstashͰJDBC input• LogstashͰCSV• FilebeatͰΞΫηεϩά• MetricbeatͰϝτϦοΫ• PacketbeatͰMySQL/PostgreSQLͷύέοτղੳ
!96Kibanaͷαϯϓϧσʔλʢ>= 6.4.0ʣ
!97ϫϯΫϦοΫͰσʔλొ
!98LogstashͰJDBC InputKibanaInstancesDataStoreElasticsearchNodesLogstashNodes
!99JDBC Input
!100LogstashͰCSVKibanaInstancesCSV FileElasticsearchNodesLogstashNodes
!101CSV filter
!102FilebeatͰΞΫηεϩάBeatsLogFilesKibanaInstancesElasticsearchNodes
• 2ͭͷElasticsearchϓϥάΠϯΛΠϯετʔϧͯ͠ElasticsearchΛىಈ• Filebeatͷapache2ϞδϡʔϧΛ༗ޮԽ• modules.d/apache2.ymlʹΞΫηεϩάͷύεΛઃఆ• setupίϚϯυΛ࣮ߦ͔ͯ͠ΒFilebeatΛىಈ!103FilebeatͰΞΫηεϩά
MetricbeatͰϝτϦοΫBeatsMetricsKibanaInstancesElasticsearchNodes
• MetricbeatͷsystemϞδϡʔϧΛ༗ޮԽ• setupίϚϯυΛ࣮ߦ͔ͯ͠ΒFilebeatΛىಈ!105MetricbeatͰϝτϦοΫ
!106PacketbeatͰMySQLɺPostgreSQLͷύέοτղੳBeatsWireDataKibanaInstancesElasticsearchNodes
σϞ!107
Thank you!● Web : https://www.elastic.co/jp/● Forums : https://discuss.elastic.co/ˇ● Twitter : @johtani
johtani
csswizardry
eileencodes
jnunemaker
chriscoyier
jasonvnalue
keathley
keithpitt
caitiem20
eitanlees
gr2m
rocio