Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Visualizing Your E-mail with Elastic Stack
Search
Kosho Owa
April 20, 2016
Technology
2
320
Visualizing Your E-mail with Elastic Stack
警視庁の犯罪・防犯情報提供サービス「メールけいしちょう」で受信したメッセージを Elasticsearch でインデックスし、Kibana で可視化する方法を紹介します。
Kosho Owa
April 20, 2016
Tweet
Share
More Decks by Kosho Owa
See All by Kosho Owa
Introducing Machine Learning for the Elastic Stack
kosho
2
12k
Elastic Stack X-Pack 5.0 for IT Security Workshop
kosho
1
310
Elastic Stack X-Pack 5.0 for IT Ops Workshop
kosho
0
330
[Developers Summit 2017] Anomaly Detection with the Elastic Stack
kosho
1
710
Anomaly Detection with the Elastic Stack
kosho
1
1.8k
Getting Started with Elastic Cloud and Beats for Log Analytics
kosho
0
99
Elastic{ON} Seminar Tokyo 2016 Product Update
kosho
0
170
Introducing Elastic Cloud
kosho
0
76
Gearing Up for Elastic Stack, X-Pack 5.0 Releases
kosho
0
150
Other Decks in Technology
See All in Technology
認知戦の理解と、市民としての対抗策
hogehuga
0
370
浸透しなさいRFC 5322&7208
hinono
0
120
「守る」から「進化させる」セキュリティへ ~AWS re:Inforce 2025参加報告~ / AWS re:Inforce 2025 Participation Report
yuj1osm
1
140
Evolution on AI Agent and Beyond - AGI への道のりと、シンギュラリティの3つのシナリオ
masayamoriofficial
0
190
Browser
recruitengineers
PRO
5
670
Preferred Networks (PFN) とLLM Post-Training チームの紹介 / 第4回 関東Kaggler会 スポンサーセッション
pfn
PRO
1
250
我々は雰囲気で仕事をしている / How can we do vibe coding as well
naospon
2
220
現場が抱える様々な問題は “組織設計上” の問題によって生じていることがある / Team-oriented Organization Design 20250827
mtx2s
6
1.5k
Devinを使ったモバイルアプリ開発 / Mobile app development with Devin
yanzm
0
190
あなたの知らない OneDrive
murachiakira
0
240
JuniorからSeniorまで: DevOpsエンジニアの成長ロードマップ
yuriemori
1
240
Backboneとしてのtimm2025
yu4u
4
1.6k
Featured
See All Featured
Building a Modern Day E-commerce SEO Strategy
aleyda
43
7.5k
A Modern Web Designer's Workflow
chriscoyier
695
190k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
33
2.4k
The Cult of Friendly URLs
andyhume
79
6.5k
Designing Experiences People Love
moore
142
24k
Six Lessons from altMBA
skipperchong
28
4k
KATA
mclloyd
32
14k
jQuery: Nuts, Bolts and Bling
dougneiner
64
7.9k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
[RailsConf 2023] Rails as a piece of cake
palkan
56
5.8k
Agile that works and the tools we love
rasmusluckow
329
21k
Transcript
‹#› Kosho Owa, Solutions Architect, Elastic April 20th, 2016 Visualizing
Your E-mail ʮϝʔϧ͚͍ͪ͠ΐ͏ʯΛՄࢹԽ͢Δ
ରσʔλ • ܯࢹிͷϝʔϧ͚͍ͪ͠ΐ͏(ొແྉ) http://www.keishicho.metro.tokyo.jp/about_mpd/joho/mail_info.html • ʮ൜ࡑൃੜใʯʮ൜ใʯΛϝʔϧ৴ • CC BY 2.1
JP Ͱఏڙ 2 Subject: ۄܯॺ(ࢠͲʢߦʣ) Body: 4݄16ʢʣɺޕޙ4࣌40͜Ζɺੈా୩۠Ԟ̍ஸͷ࿏্Ͱɺࣇಐ͕௨ߦதɺஉʹಥ ͖ඈ͞Ε·ͨ͠ɻʢ൜ਓʢஉʣͷಛʹ͍ͭͯɺ̑̌ࡀɺ170cm Ґɺதɺޱͻ ͛ɺ৭ͬΆ্͍ҥɺࠇ৭ͬΆ͍ζϘϯʣ ʲ߹ͤઌʳۄܯॺ 03-3705-0110ʢઢ2612ʣ
ํ • ϝʔϧΛIMAPͰऔಘ • ϑΟʔϧυΛߏԽ͢Δ • λΠϓΛߟྀͯ͠ΠϯσοΫε • analyzed, not_analyzedϑΟʔϧυͦΕͧΕΛ༻ͯ͠ՄࢹԽ͢Δ
3
Logstash Pipeline and Plugins ϓϥάΠϯՄೳͳΞʔΩςΫνϟʔͱɺ։ൃऀʹ༏͍͠ΤίγεςϜ 4 input {} filter {}
output {} beats, file, graphite, http, imap, kafka, rss, redis, stdin, sqlite, s3, syslog, zenoss and etc. csv, cloudwatch, email, elasticsearch, exec, file, graphite, http, kafka, mongodb, nagios, redis, s3, syslog, stdout, zabbix and etc.
Input Plugin - imap 5 input { imap { host
=> "imap.gmail.com" port => 993 user => "_IMAP_USER_" password => "_IMAP_PASSWORD_" folder => "_IMAP_FOLDER_" type => "_TYPE_" check_interval => 300 codec => plain { charset => "ISO-2022-JP" } } } • ϝʔϧຊจͷΤϯίʔυΛcodecͰࢦఆ͢Δ • ͋Β͔͡ΊIMAPͷfolderΛ͚ • ෳͷλΠϓϝʔϧΛॲཧ͢Δ߹ʹλά(tags)ΛՃ͢Δ https://www.elastic.co/guide/en/logstash/current/plugins-inputs-imap.html • : ίϛϡχςΟϓϥάΠϯ
Filter Plugin • ϝʔϧͷຊจ͔Βൈ͖ग़͢ϑΟʔϧυ: city, area, place • λΠτϧ͔Βൈ͖ग़͢ϑΟʔϧυ: police_station,
incident • λΠϜελϯϓͱͯ͠࠾༻: datetime 6 filter { grok { match => { "message" => "%{DATA:[@metadata][datetime]}͜Ζɺ%{NOTSPACE:city}(۠|ࢢ)% {NOTSPACE:area}(ͷ|ۙ)(%{NOTSPACE:place}|)Ͱɺ%{GREEDYDATA}" } } date { match => ["[@metadata][datetime]", "M݄dʢEʣɺaK࣌m"] locale => ja timezone => "Asia/Tokyo" } grok { match => { "subject" => "%{NOTSPACE:police_station}ܯॺ\(%{NOTSPACE:incident}\)" } } }
ೖྗσʔλ grok ग़ྗ Filter Plugin - grok • ύλʔϯʹϚονͨ͠จࣈྻΛϑΟʔϧυʹؔ࿈͚ɺඇߏσʔλΛߏԽ͢Δ https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
7 “subject” => “ۄܯॺ(ࢠͲʢߦʣ)” grok { match => { "subject" => "%{NOTSPACE:police_station}ܯॺ\(%{NOTSPACE:incident}\)" } } “police_station” => "ۄ" “incident" => "(ࢠͲʢߦʣ)"
ೖྗσʔλ date ग़ྗ Filter Plugin - date ϑΟʔϧυΛύʔε͠ɺLogstashͷΠϕϯτͱͯ͠༻ 8 "datetime"
=> “4݄16ʢʣɺޕલ7࣌40” "@timestamp" => "2016-04-16T07:40:00.000Z" • ͷऔಘʹࣦഊͨ͠߹ʹɺॲཧ͕࣌@timestampͱͯ͠࠾༻͞ΕΔ (tag_on_failure => true ݕ౼) https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html date { match => ["[@metadata][datetime]", "M݄dʢEʣɺaK࣌m"] locale => ja timezone => "Asia/Tokyo" }
Output Plugin - elasticsearch 9 output { stdout { codec
=> dots } elasticsearch { hosts => ["http://127.0.0.1:9200/"] index => "mail-%{+YYYY.MM}" } } • stdout { codec => dots } ͰɺҰ݅ॲཧ͝ͱʹυοτΛग़ྗ͢Δ • ΠϯσοΫε͕దͳαΠζʹͳΔΑ͏ɺΠϯσοΫε໊Λݕ౼͢Δ https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
Logstash Tips • ग़ྗ࣌ʹύΠϓϥΠϯΛදࣔ • ϫʔΧʔΛదʹઃఆ͢Δ • ҟͳΔछྨͷσʔλɺLogstashͷೖྗલʹ͚͓ͯ͘ • grok
ϔϧύʔπʔϧΛ͏ http://grokdebug.herokuapp.com http://grokconstructor.appspot.com 10 output { stdout { codec => rubydebug } } $ logstash -w [NUMBER OF WORKERS] -f [PATH TO CONFIG]
Elasticsearch - Mapping • text (analyzed strings), keyword(not_analyzed strings)ϑΟʔϧυ5.0͔Βಋೖ •
textϑΟʔϧυͷanalyzerʹkuromojiΛࢦఆ͢Δ • terms aggregationΛߦ͏ͨΊʹɺmulti-fieldػೳΛͬͯkeywordϑΟʔϧυΛࢦఆ͢Δ 11 PUT /_template/mail-1 { "template": "mail-*", "mappings": { "_default_": { "properties": { "message": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }, "analyzer": "kuromoji" },... }}}}
Kibana - Visualize “Terms Aggregation” keywordϑΟʔϧυͰaggregation͢Δ 12
Kibana - Visualize “Filters Aggregation” analyzedϑΟʔϧυͰaggregation͢Δ 13
ؔ࿈ใ 14