Kibana4でいろんなデータを見てみよう

͍ΖΜͳσʔλΛKibana4ͰݟͯΈΑ͏ Jun Ohtani @johtani 2015/04/11

www.elastic.co Copyright Elastic 2015 Copying, publishing and/or distributing without written
permission is strictly prohibited 3 about • Me, Jun Ohtani / Technical Adovocate – lucene-gosenίϛολʔ – ElasticSearch Server೔ຊޠ൛ͷ຋༁ – elasticsearch-extended-analysisͷ։ൃ – http://blog.johtani.info • Elasticsearch, founded in 2012 – Products: Elasticsearch, Logstash, Kibana, Marvel  Professional services: Support & development subscriptions – Trainings

permission is strictly prohibited 4 ΞδΣϯμ • ELK stack঺հ • Logstash • Elasticsearch • Kibana • ϢʔεέʔεʴσϞ • ΞΫηεϩά • Twitter Stream • …

permission is strictly prohibited 5 ELK stack ELK stack

permission is strictly prohibited 6 ELK stack Data Import/Parse/ Export Store/Search Visualize

permission is strictly prohibited 7 ELK stack Import/Parse/ Export Store/Search Visualize Data

permission is strictly prohibited 8 Logstash Logstash

permission is strictly prohibited 9 Logstash in 10 seconds • ϩάɾσʔλͷऩूɾ؅ཧ • ऩूɺύʔεɾՃ޻ɺૹग़ • ΦʔϓϯιʔεɿApache License 2.0 • Ruby app (JRuby) • ࠷৽൛ɿ1.4.2 or 1.5.0 RC2

permission is strictly prohibited 10 Logstash architecture Logstash Input Output Filter ? ? collect and split alter and enrich store and visualize

permission is strictly prohibited 11 ઃఆαϯϓϧ input { file { path => "/Users/johtani/demo_access_log/*/*.log" } } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } } output { elasticsearch { host => "localhost" } }

permission is strictly prohibited 15 μ΢ϯϩʔυͱىಈ $ wget https://download.elastic.co/... $ tar -xf logstash-1.5.0-rc2.tar.gz $ ./bin/logstash -e 'input { stdin{}} output {stdout{}}' ... Logstash startup completed ... Also puppet modules and RPM/DEB

permission is strictly prohibited 16 ELK stack Import/Parse/ Export Store/Search Visualize Data

permission is strictly prohibited 17 Elasticsearch Elasticsearch

permission is strictly prohibited 18 ϑϦʔϫʔυݕࡧ

permission is strictly prohibited 19 ߜΓࠐΈ

permission is strictly prohibited 20 ϋΠϥΠτ

permission is strictly prohibited 21 ιʔτ

permission is strictly prohibited 22 ϖʔδϯά

permission is strictly prohibited 23 ूܭ

permission is strictly prohibited 24 αδΣετ

permission is strictly prohibited 25 Elasticsearch in 10 seconds • εΩʔϚϑϦʔɺ෼ࢄυΩϡϝϯτετΞɺREST & JSON • Φʔϓϯιʔε: Apache License 2.0 • ઃఆͳ͠Ͱ؆୯ʹࢼ͢͜ͱ͕Մೳ • JavaͰ࣮૷ɻ֦ு΋༰қ

permission is strictly prohibited 26 μ΢ϯϩʔυͱىಈ $ wget https://download.elastic.co/... $ tar -xf elasticsearch-1.5.1.tar.gz $ ./elasticsearch-1.5.1/bin/elasticsearch ... [2015-04-10 10:02:17,278][INFO ][node] [Joseph] started ... Also puppet modules and RPM/DEB

permission is strictly prohibited 28 Kibana Kibana

permission is strictly prohibited 29 ࢖ͬͯ·͔͢ʁ Kibana3࢖ͬͯ·͔͢ʁ

permission is strictly prohibited 30 Kibana3

permission is strictly prohibited 31 Kibana3ʁ ·ͩKibana3࢖ͬͯΔΜͰ͔͢ʁ

permission is strictly prohibited 32 Kibana4 Kibana4

permission is strictly prohibited 33 ίϯηϓτ • σʔλΛ୳ࡧ • ୳ࡧͨ͠σʔλ͔ΒάϥϑΛ࡞੒ • άϥϑΛ૊Έ߹ΘͤͯμογϡϘʔυ࡞੒

permission is strictly prohibited 34 μ΢ϯϩʔυˍىಈ $ wget https://download.elastic.co/kibana… $ tar -xf kibana-4.0.2-darwin-x64.tar.gz $ ./kibana-4.0.2-darwin-x64/bin/kibana ... {"@timestamp":"2015-04-10T13:26:53.673Z","level":"info", "message":"Found kibana index","node_env":"production"} {"@timestamp":"2015-04-10T13:26:53.785Z","level":"info", "message":"Listening on 0.0.0.0:5601","node_env":"production"} ...

permission is strictly prohibited 35 ॳظը໘ http://localhost:5601/ ʹΞΫηε

permission is strictly prohibited 36 Discover • σʔλͷ୳ࡧΛߦ͏ը໘ • ͲΜͳϑΟʔϧυ͕͋Δ͔ • ݕࡧର৅ͷϑΟʔϧυɺσʔλͷछผ͕Θ͔Δ • ݕࡧͨ݁͠Ռ͸ͲΜͳσʔλ͔ • ݕࡧ৚݅ͷอଘ • ݕࡧ݁ՌͷΤΫεϙʔτ

permission is strictly prohibited 37 Visualize • ݕࡧ݁ՌΛݩʹάϥϑΛ࡞੒ • άϥϑͷλΠϓΛબ୒ • Y࣠ɺX࣠ͷબ୒ • Sub Aggregationͷબ୒ • άϥϑͷอଘ

permission is strictly prohibited 38 Dashboard • อଘͨ͠άϥϑΛ഑ஔɺϦαΠζ • μογϡϘʔυͷอଘɺڞ༗ • άϥϑͷ֤छ৘ใͷදࣔ – Table – Request – Response – Statistics

permission is strictly prohibited 39 Kibana3ͱͷҧ͍ • ෳ਺ͷΠϯσοΫεͷάϥϑΛ1ͭͷμογϡϘʔυʹ • ҟͳΔϑΟʔϧυͷ஋ΛҰͭͷάϥϑʹ • σʔλͷΤΫεϙʔτ • ݕࡧ৚݅ͷอଘ • Aggregation͕ར༻Մೳ

permission is strictly prohibited 40 ΞΫηεϩά ϢʔεέʔεɿΞΫηεϩά

permission is strictly prohibited 41 Dashboard

permission is strictly prohibited 43 ΞΫηεϩάʁ 75.126.xx.xx - - [02/Dec/2014:01:40:35 +0900] "GET /phpMyAdmin/scripts/ setup.php HTTP/1.1" 404 290 "-" "ZmEu" 75.126.xx.xx - - [02/Dec/2014:01:40:35 +0900] "GET /pma/scripts/setup.php HTTP/1.1" 404 283 "-" "ZmEu" 75.126.xx.xx - - [02/Dec/2014:01:40:35 +0900] "GET /myadmin/scripts/ setup.php HTTP/1.1" 404 287 "-" "ZmEu" 75.126.xx.xx - - [02/Dec/2014:01:40:36 +0900] "GET /MyAdmin/scripts/ setup.php HTTP/1.1" 404 287 "-" "ZmEu" 62.210.xx.xx - - [02/Dec/2014:04:19:17 +0900] "GET /admin/config.php HTTP/1.1" 404 278 "-" "curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18 libssh2/1.4.2" 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0"

permission is strictly prohibited 53 ΞΫηεϩάɿLogstash Import/Parse/ Export Store/Search Visualize Data

permission is strictly prohibited 54 ઃఆɿinput input { file { path => “/Users/johtani/sample/*_log" start_position => "beginning" } }

permission is strictly prohibited 55 1ߦ1σʔλ 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0"

permission is strictly prohibited 56 ઃఆɿfilter filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } break_on_match => false } date { match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss Z"] locale => en } geoip { source => ["clientip"] } useragent { source => "agent" target => "useragent" } }

permission is strictly prohibited 57 ύʔε 189.120.xx.xx - - [02/Dec/2014:12:18:29 +0900] "GET /manager/html HTTP/ 1.1" 404 274 "-" "Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0" {… "@timestamp": "2015-04-10T09:07:49.325Z", "clientip": "189.120.xx.xx", "ident": "-", "auth": "-", "timestamp": "02/Dec/2014:12:18:29 +0900", "verb": "GET", "request": "/manager/html", … "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0\"" }

permission is strictly prohibited 59 ೔෇ͷύʔε {… "@timestamp": "2015-04-10T09:07:49.325Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", … } {… "@timestamp": "2014-12-02T03:18:29.000Z", … "timestamp": "02/Dec/2014:12:18:29 +0900", … }

permission is strictly prohibited 61 IP͔ΒҢ౓ܦ౓ͳͲ෇༩ {… "clientip": "189.120.xx.xx", …} {… "clientip": "189.120.xx.xx", … "geoip": { "ip": “189.120.xxx.xxx”, … "country_name": "Brazil", "continent_code": "SA", "region_name": "27", "city_name": "São Paulo", "latitude": -‐23.473299999999995, "longitude": -‐46.66579999999999, …

permission is strictly prohibited 63 ϢʔβΤʔδΣϯτͷύʔε {… "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0\"" …} {… "agent": "\"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0\"" … "useragent": { "name": "Firefox", "os": "Windows XP", "os_name": "Windows XP", "device": "Other", "major": "5", "minor": "0" } …

permission is strictly prohibited 64 ઃఆɿoutput output { elasticsearch { host => "localhost" index => “demo_access_log-‐%{+YYYY.MM.dd}” } }

permission is strictly prohibited 65 ΞΫηεϩάɿelasticsearch Import/Parse/ Export Store/Search Visualize Data

permission is strictly prohibited 66 Index template {… "order": 0, "template": "demo_access_log-*", "settings": { "index.number_of_replicas": "0", "index.number_of_shards": "2" }, "mappings": { … }, "aliases": {} }, …

permission is strictly prohibited 67 Index template {… "mappings": { "_default_": { "dynamic_templates": [ { "string_template": { "mapping": { "index": "not_analyzed", "type": "string" }, "match_mapping_type": "string", "match": "*" } } ], …

permission is strictly prohibited 68 σϞ for Kibana4 Access Log demo for Kibana4

permission is strictly prohibited 69 πΠʔτ ϢʔεέʔεɿπΠʔτ

permission is strictly prohibited 70 ઃఆɿinput input{ twitter{ consumer_key => “…….” consumer_secret => “……………………." oauth_token => “……..-‐…………………….” oauth_token_secret => “……………………" keywords => ["ࡩ", "։Ֆ", "Ֆݟ"] }

permission is strictly prohibited 71 ઃఆɿoutput output { elasticsearch { host => "localhost" index => “twitter-‐%{+YYYY.MM.dd}” } }

permission is strictly prohibited 72 σϞ for Kibana4 Twitter Stream demo for Kibana4

permission is strictly prohibited 73 Git log ϢʔεέʔεɿGit log

permission is strictly prohibited 74 git-log2es • GitHubͰެ։͞ΕͯͨͷͰɺར༻ͯ͠Έͨ • https://github.com/Etsukata/git-log2es

permission is strictly prohibited 75 ؾ৅ிͷؾ৅σʔλ ϢʔεέʔεɿؾԹσʔλ

permission is strictly prohibited 76 Elasticsearchͷϩά ϢʔεέʔεɿElasticsearchͷϩά

permission is strictly prohibited 77 Elasticsearchษڧձ

permission is strictly prohibited 78 ࢀߟจݙ • Elasticsearch - The Definitive guide – http://www.elasticsearch.com/guide/en/elasticsearch/ guide/current/index.html • ॻ੶ – ElasticSearchServer೔ຊޠ൛  αʔό/ΠϯϑϥΤϯδχΞ  ɹཆ੒ಡຊɹϩάऩू

permission is strictly prohibited 79 Q&A • Elasticsearch Document – http://www.elastic.co/guide/en/elasticsearch/ reference/current/index.html • Logstash Document – http://www.elastic.co/guide/en/logstash/current/ index.html • Kibana – http://www.elastic.co/guide/en/kibana/current/ index.html

Kibana4でいろんなデータを見てみよう

Kibana4でいろんなデータを見てみよう

More Decks by Jun Ohtani

Other Decks in Technology

Featured

Transcript