Share
Embed
Visualizing Your E-mail with Elastic Stack
警視庁の犯罪・防犯情報提供サービス「メールけいしちょう」で受信したメッセージを Elasticsearch でインデックスし、Kibana で可視化する方法を紹介します。
Kosho Owa
April 20, 2016
Transcript
14 slides
-
‹#› Kosho Owa, Solutions Architect, Elastic April 20th, 2016 Visualizing
Your E-mail ʮϝʔϧ͚͍ͪ͠ΐ͏ʯΛՄࢹԽ͢Δ -
ରσʔλ • ܯࢹிͷϝʔϧ͚͍ͪ͠ΐ͏(ొແྉ) http://www.keishicho.metro.tokyo.jp/about_mpd/joho/mail_info.html • ʮ൜ࡑൃੜใʯʮ൜ใʯΛϝʔϧ৴ • CC BY 2.1
JP Ͱఏڙ 2 Subject: ۄܯॺ(ࢠͲʢߦʣ) Body: 4݄16ʢʣɺޕޙ4࣌40͜Ζɺੈా୩۠Ԟ̍ஸͷ࿏্Ͱɺࣇಐ͕௨ߦதɺஉʹಥ ͖ඈ͞Ε·ͨ͠ɻʢ൜ਓʢஉʣͷಛʹ͍ͭͯɺ̑̌ࡀɺ170cm Ґɺதɺޱͻ ͛ɺ৭ͬΆ্͍ҥɺࠇ৭ͬΆ͍ζϘϯʣ ʲ߹ͤઌʳۄܯॺ 03-3705-0110ʢઢ2612ʣ -
Logstash Pipeline and Plugins ϓϥάΠϯՄೳͳΞʔΩςΫνϟʔͱɺ։ൃऀʹ༏͍͠ΤίγεςϜ 4 input {} filter {}
output {} beats, file, graphite, http, imap, kafka, rss, redis, stdin, sqlite, s3, syslog, zenoss and etc. csv, cloudwatch, email, elasticsearch, exec, file, graphite, http, kafka, mongodb, nagios, redis, s3, syslog, stdout, zabbix and etc. -
Input Plugin - imap 5 input { imap { host
=> "imap.gmail.com" port => 993 user => "_IMAP_USER_" password => "_IMAP_PASSWORD_" folder => "_IMAP_FOLDER_" type => "_TYPE_" check_interval => 300 codec => plain { charset => "ISO-2022-JP" } } } • ϝʔϧຊจͷΤϯίʔυΛcodecͰࢦఆ͢Δ • ͋Β͔͡ΊIMAPͷfolderΛ͚ • ෳͷλΠϓϝʔϧΛॲཧ͢Δ߹ʹλά(tags)ΛՃ͢Δ https://www.elastic.co/guide/en/logstash/current/plugins-inputs-imap.html • : ίϛϡχςΟϓϥάΠϯ -
Filter Plugin • ϝʔϧͷຊจ͔Βൈ͖ग़͢ϑΟʔϧυ: city, area, place • λΠτϧ͔Βൈ͖ग़͢ϑΟʔϧυ: police_station,
incident • λΠϜελϯϓͱͯ͠࠾༻: datetime 6 filter { grok { match => { "message" => "%{DATA:[@metadata][datetime]}͜Ζɺ%{NOTSPACE:city}(۠|ࢢ)% {NOTSPACE:area}(ͷ|ۙ)(%{NOTSPACE:place}|)Ͱɺ%{GREEDYDATA}" } } date { match => ["[@metadata][datetime]", "M݄dʢEʣɺaK࣌m"] locale => ja timezone => "Asia/Tokyo" } grok { match => { "subject" => "%{NOTSPACE:police_station}ܯॺ\(%{NOTSPACE:incident}\)" } } } -
ೖྗσʔλ grok ग़ྗ Filter Plugin - grok • ύλʔϯʹϚονͨ͠จࣈྻΛϑΟʔϧυʹؔ࿈͚ɺඇߏσʔλΛߏԽ͢Δ https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
7 “subject” => “ۄܯॺ(ࢠͲʢߦʣ)” grok { match => { "subject" => "%{NOTSPACE:police_station}ܯॺ\(%{NOTSPACE:incident}\)" } } “police_station” => "ۄ" “incident" => "(ࢠͲʢߦʣ)" -
ೖྗσʔλ date ग़ྗ Filter Plugin - date ϑΟʔϧυΛύʔε͠ɺLogstashͷΠϕϯτͱͯ͠༻ 8 "datetime"
=> “4݄16ʢʣɺޕલ7࣌40” "@timestamp" => "2016-04-16T07:40:00.000Z" • ͷऔಘʹࣦഊͨ͠߹ʹɺॲཧ͕࣌@timestampͱͯ͠࠾༻͞ΕΔ (tag_on_failure => true ݕ౼) https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html date { match => ["[@metadata][datetime]", "M݄dʢEʣɺaK࣌m"] locale => ja timezone => "Asia/Tokyo" } -
Output Plugin - elasticsearch 9 output { stdout { codec
=> dots } elasticsearch { hosts => ["http://127.0.0.1:9200/"] index => "mail-%{+YYYY.MM}" } } • stdout { codec => dots } ͰɺҰ݅ॲཧ͝ͱʹυοτΛग़ྗ͢Δ • ΠϯσοΫε͕దͳαΠζʹͳΔΑ͏ɺΠϯσοΫε໊Λݕ౼͢Δ https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html -
Logstash Tips • ग़ྗ࣌ʹύΠϓϥΠϯΛදࣔ • ϫʔΧʔΛదʹઃఆ͢Δ • ҟͳΔछྨͷσʔλɺLogstashͷೖྗલʹ͚͓ͯ͘ • grok
ϔϧύʔπʔϧΛ͏ http://grokdebug.herokuapp.com http://grokconstructor.appspot.com 10 output { stdout { codec => rubydebug } } $ logstash -w [NUMBER OF WORKERS] -f [PATH TO CONFIG] -
Elasticsearch - Mapping • text (analyzed strings), keyword(not_analyzed strings)ϑΟʔϧυ5.0͔Βಋೖ •
textϑΟʔϧυͷanalyzerʹkuromojiΛࢦఆ͢Δ • terms aggregationΛߦ͏ͨΊʹɺmulti-fieldػೳΛͬͯkeywordϑΟʔϧυΛࢦఆ͢Δ 11 PUT /_template/mail-1 { "template": "mail-*", "mappings": { "_default_": { "properties": { "message": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "analyzer": "kuromoji" },... }}}}