Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Introducing Machine Learning for the Elastic Stack
Search
Kosho Owa
May 19, 2017
Technology
2
12k
Introducing Machine Learning for the Elastic Stack
Elastic Machine Learning Seminar held on May 19th, 2017
Kosho Owa
May 19, 2017
Tweet
Share
More Decks by Kosho Owa
See All by Kosho Owa
Elastic Stack X-Pack 5.0 for IT Security Workshop
kosho
1
290
Elastic Stack X-Pack 5.0 for IT Ops Workshop
kosho
0
310
[Developers Summit 2017] Anomaly Detection with the Elastic Stack
kosho
1
690
Anomaly Detection with the Elastic Stack
kosho
1
1.8k
Getting Started with Elastic Cloud and Beats for Log Analytics
kosho
0
91
Elastic{ON} Seminar Tokyo 2016 Product Update
kosho
0
160
Introducing Elastic Cloud
kosho
0
64
Gearing Up for Elastic Stack, X-Pack 5.0 Releases
kosho
0
130
Elastic Stack Hands-on Workshop (EN)
kosho
1
150
Other Decks in Technology
See All in Technology
1行のコードから社会課題の解決へ: EMの探究、事業・技術・組織を紡ぐ実践知 / EM Conf 2025
9ma3r
11
3.8k
偏光画像処理ライブラリを作った話
elerac
1
170
あなたが人生で成功するための5つの普遍的法則 #jawsug #jawsdays2025 / 20250301 HEROZ
yoshidashingo
2
290
生成AI “再”入門 2025年春@WIRED TUESDAY EDITOR'S LOUNGE
kajikent
0
110
短縮URLをお手軽に導入しよう
nakasho
0
150
ディスプレイ広告(Yahoo!広告・LINE広告)におけるバックエンド開発
lycorptech_jp
PRO
0
360
(機械学習システムでも) SLO から始める信頼性構築 - ゆる SRE#9 2025/02/21
daigo0927
0
270
Autonomous Database Serverless 技術詳細 / adb-s_technical_detail_jp
oracle4engineer
PRO
17
45k
AI Agent時代なのでAWSのLLMs.txtが欲しい!
watany
2
230
いまからでも遅くない!コンテナでWebアプリを動かしてみよう!コンテナハンズオン編
nomu
0
150
IAMポリシーのAllow/Denyについて、改めて理解する
smt7174
2
210
AIエージェント入門
minorun365
PRO
31
18k
Featured
See All Featured
Building Applications with DynamoDB
mza
93
6.2k
RailsConf 2023
tenderlove
29
1k
Music & Morning Musume
bryan
46
6.4k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.7k
Why Our Code Smells
bkeepers
PRO
336
57k
Designing on Purpose - Digital PM Summit 2013
jponch
117
7.1k
A Modern Web Designer's Workflow
chriscoyier
693
190k
How to train your dragon (web standard)
notwaldorf
91
5.9k
Git: the NoSQL Database
bkeepers
PRO
427
65k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
28
9.3k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
193
16k
Embracing the Ebb and Flow
colly
84
4.6k
Transcript
Machine Learning for the Elastic Stack Beta in 5.4.
GA coming soon May 2017 େྠ ߂ৄ | Kosho Owa Solutions Architect, Elastic
2 Elastic Stack 100% Φʔϓϯιʔε ʮΤϯλʔϓϥΠζ൛ʯແ͠ όʔδϣϯ 5.0Ͱશ౷Ұ
3 X-Pack ؆୯ʹΠϯετʔϧ Elastic StackΛ֦ு αϒεΫϦϓγϣϯʹؚ·ΕΔ Security Alerting Monitoring Reporting
Graph Machine Learning
4 Elastic Cloud Elasticsearch, Kibanaͷ ϚωʔδυαʔϏε X-Packͷػೳར༻Մೳ Available in AWS
today
5 Elastic Cloud Enterprise ෳͷElastic StackڥΛࣗࡏʹ࡞ Logging as a serviceΛࣗ৫ʹల։
Public beta; Expected GA Q1 2017
ҟৗͷൃݟ͕τϥϒϧͷஹީΛࣔ͢ 6 Spiked 404 errors Web attack IT Operational Analytics
Security Analytics Business Analytics Unusual DNS activity Data exfiltration Rare log messages Failing sensor
Operational Analytics • ΣϒαΠτͷΞΫηετϥϑΟοΫʹҟৗແ͍͔? • Ϙοτ߈ܸऀ͕๚Ε͍ͯͳ͍͔? • σʔλϕʔε͕ग़ྗ͍ͯ͠ΔErrorϩάରॲ͢Δඞཁ͕ ͋Δͷ͔? Use
Case
Security Analytics • ϚϧΣΞʹ৵ೖ͞Ε͍ͯͳ͍͔? • ෦ऀʹΑΔηΩϡϦςΟڴҖແ͍͔? • DNSͷϩάʹɺσʔλऔͷ͕ࠟͳ͍͔? Use Case
Telemetry / Sensors ▪ ISPͷωοτϫʔΫҰ࣌ःஅʹΑΔϨΠςϯγʔͷٸ ܹͳ૿Ճ? ▪ ଞͱҟͳΔӡసύλʔϯΛͱΔυϥΠόʔ? ▪ ಛҟͳΠϕϯτλΠϓηϯαʔͷނোΛ͔ࣔ͢?
Use Case
10 ҟৗͷൃݟࢥͬͨΑΓ͍͠ • σʔλෳࡶɺߴ࣍ݩɺߴʹมԽ • ਓؒͷࢹೝݱ࣮తʹෆՄೳ • ༰қʹݟಀ͢ Visual inspection
is not practical Where’s the anomaly?
11 ҟৗͷൃݟࢥͬͨΑΓ͍͠ • ੩తͳᮢʹΑΔʮਖ਼ৗʯͷఆٛࠔ • ϧʔϧσʔλΠϯϑϥͷมߋʹैͰ͖ͳ͍ • ༰қʹᷖճ͞Εͯ͠·͏ Rule-based alerts
are insufficient What’s the right threshold ?
X-Pack͕ࣗಈతͳҟৗݕͰղܾ 12 • ʮڭࢣͳ͠ʯػցֶशςΫχοΫʹΑΓ ▪ աڈͷσʔλ͔Βʮਖ਼ৗʯΛֶͼϞσϧΛ࡞Δ ▪ ਖ਼ৗͷൣғ͔Βҳͨ͠ࡍʹҟৗͱͯ͠ݕ
X-Pack͕ࣗಈతͳҟৗݕͰղܾ 13 • ڭࢣͳ͠ - खಈͰͷਖ਼ৗͷೖྗ͕ෆཁ • σʔλͷมԽʹै - ೖ͞ΕΔσʔλʹΑΓܧଓతʹϞσϧΛߋ৽
• ӨڹҼࢠಛఆ - ࠜຊݪҼղੳΛՃ
ҟͳΔछྨͷҟৗΛݕ 14 • ࣌ܥྻͷϝτϦοΫ Time series - single / multiple
• ͙Εऀ Outliers in population (using entity profiling) • ك༗ͳඇߏϝοηʔδ Rare / unusual rates in “categories” of events
࣌ܥྻσʔλͷҟৗ 15 Time Metric • Single (univariate) time series Example:
Is there unusual traffic on website ?
࣌ܥྻσʔλͷҟৗ 16 Time Metric USA UK France Japan • Multiple
time series ▪ ෳͷϝτϦοΫ ▪ FieldʹΑͬͯྨ͞ΕͨϝτϦοΫ • ͦΕͧΕ͕ಠཱͯ͠ଘࡏ͢Δ Example: Is there unusual web activity from any country?
͙Εऀ Outliers in population (using entity profiling) 17 • ूஂͷಛ(server,
user, IPͳͲ)͔ΒϓϩϑΝΠϧΛ࡞͢Δ • ͜ͷूஂ͔Βҳ͢ΔͷΛൃݟ͢Δ Example: • Which IP address is not like the others? (indication of a bot / attacker)
͙Εऀ Outliers in population (using entity profiling) 18 • ूஂͷಛ(server,
user, IPͳͲ)͔ΒϓϩϑΝΠϧΛ࡞͢Δ • ͜ͷूஂ͔Βҳ͢ΔͷΛൃݟ͢Δ Example: • Which IP address is not like the others? (indication of a bot / attacker)
ك༗ͳඇߏϝοηʔδͷมԽ Unusual or rare events (via log categorization) 19 •
ྨࣅੑʹج͍ͮͯΧςΰϦ͚ • ࣌ؒมԽʹΑΔසΛֶश • ϞσϧͱҟͳΕҟৗͱͯ͠ݕ Example: • Do my application logs contain unusual messages
X-Pack Machine Learning Elastic StackͱͷڧݻͳΠϯςάϨʔγϣϯ 20
• Elasticsearch • Kibana ༰қʹΠϯετʔϧ 21 $ elasticsearch-plugin install x-pack
$ kibana-plugin install x-pack
σϓϩΠϝϯτϞσϧ 22 Cluster Data node Apps Master node Data node
Data node Master node Master node Data node Data node ES clients, Kibana, Logstash, Beats, User apps and etc. ML node ML node # config/elasticsearch.yml xpack.ml.enabled: true node.ml: true
֎෦γεςϜͱͷଓ • API (anomaly_detectors, datafeeds, results, model_snapshots, validate) • ΠϯσοΫε
(.ml-anomalies-*)
Taking Action with X-Pack Alerting 24
Demo Single/Multiple Metrics: New York City Yellow Taxi Outliers in
Population: Web Server Log Rare Messages: DBMS Server Log 25
26 4JOHMF.FUSJD
27 .VMUJ.FUSJD
28 .VMUJ.FUSJD
29 0VUMJFSTJO1PQVMBUJPO
30 0VUMJFSTJO1PQVMBUJPO
31 3BSF.FTTBHFT
32 3BSF.FTTBHFT
࣍ͷεςοϓ 33 • Elastic StackΛ·ͩར༻͍ͯ͠ͳ͍ • ϋϯζΦϯϫʔΫγϣοϓ • Elastic StackɺX-PackΛΠϯετʔϧ
• αϯϓϧσʔλΛར༻ (ϒϩάࢀর) or ࣗͷσʔλΛೖ • MLδϣϒΛ࡞ • Elastic StackΛར༻த • X-PackΛΠϯετʔϧ (30ؒͷτϥΠΞϧ/ඇϓϩμΫγϣϯڥ) • MLδϣϒΛ࡞ (Ϩγϐ׆༻) • AlertingͰΞΫγϣϯ