Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Introducing Machine Learning for the Elastic Stack
Search
Kosho Owa
May 19, 2017
Technology
2
12k
Introducing Machine Learning for the Elastic Stack
Elastic Machine Learning Seminar held on May 19th, 2017
Kosho Owa
May 19, 2017
Tweet
Share
More Decks by Kosho Owa
See All by Kosho Owa
Elastic Stack X-Pack 5.0 for IT Security Workshop
kosho
1
300
Elastic Stack X-Pack 5.0 for IT Ops Workshop
kosho
0
320
[Developers Summit 2017] Anomaly Detection with the Elastic Stack
kosho
1
700
Anomaly Detection with the Elastic Stack
kosho
1
1.8k
Getting Started with Elastic Cloud and Beats for Log Analytics
kosho
0
96
Elastic{ON} Seminar Tokyo 2016 Product Update
kosho
0
160
Introducing Elastic Cloud
kosho
0
74
Gearing Up for Elastic Stack, X-Pack 5.0 Releases
kosho
0
140
Elastic Stack Hands-on Workshop (EN)
kosho
1
160
Other Decks in Technology
See All in Technology
データ戦略部門 紹介資料
sansan33
PRO
1
3.1k
Redmineの意外と知らない便利機能 (Redmine 6.0対応版)
vividtone
0
1.4k
Zero Data Loss Autonomous Recovery Service サービス概要
oracle4engineer
PRO
2
7.3k
20250514_未経験から Fintech実務参画まで。学生エンジニアの挑戦録
hideto1008
0
810
Digitization部 紹介資料
sansan33
PRO
1
4k
Information Architecture Recommoning: How Standardization Enables Differentiation
angioia
0
140
Flutterアプリを⾃然⾔語で操作する
yukisakai1225
0
200
メルカリにおけるデータアナリティクス AI エージェント「Socrates」と ADK 活用事例
na0
14
6.3k
ai bot got sick (abc 2025s version)
kojira
0
120
CloudBruteによる外部からのS3バケットの探索・公開の発見について / 20250605 Kumiko Henmi
shift_evolve
3
340
データプレーンプログラミングとは? DPU&スイッチASICの開発経験から語る
ebiken
PRO
1
300
Devin&Cursor、それぞれの「本質」から導く最適ユースケース戦略
empitsu
8
2.9k
Featured
See All Featured
YesSQL, Process and Tooling at Scale
rocio
172
14k
Raft: Consensus for Rubyists
vanstee
137
7k
StorybookのUI Testing Handbookを読んだ
zakiyama
30
5.8k
Site-Speed That Sticks
csswizardry
9
610
Docker and Python
trallard
44
3.4k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
50k
We Have a Design System, Now What?
morganepeng
52
7.6k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
280
13k
Designing for humans not robots
tammielis
253
25k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
42
2.3k
Become a Pro
speakerdeck
PRO
28
5.4k
How GitHub (no longer) Works
holman
314
140k
Transcript
Machine Learning for the Elastic Stack Beta in 5.4.
GA coming soon May 2017 େྠ ߂ৄ | Kosho Owa Solutions Architect, Elastic
2 Elastic Stack 100% Φʔϓϯιʔε ʮΤϯλʔϓϥΠζ൛ʯແ͠ όʔδϣϯ 5.0Ͱશ౷Ұ
3 X-Pack ؆୯ʹΠϯετʔϧ Elastic StackΛ֦ு αϒεΫϦϓγϣϯʹؚ·ΕΔ Security Alerting Monitoring Reporting
Graph Machine Learning
4 Elastic Cloud Elasticsearch, Kibanaͷ ϚωʔδυαʔϏε X-Packͷػೳར༻Մೳ Available in AWS
today
5 Elastic Cloud Enterprise ෳͷElastic StackڥΛࣗࡏʹ࡞ Logging as a serviceΛࣗ৫ʹల։
Public beta; Expected GA Q1 2017
ҟৗͷൃݟ͕τϥϒϧͷஹީΛࣔ͢ 6 Spiked 404 errors Web attack IT Operational Analytics
Security Analytics Business Analytics Unusual DNS activity Data exfiltration Rare log messages Failing sensor
Operational Analytics • ΣϒαΠτͷΞΫηετϥϑΟοΫʹҟৗແ͍͔? • Ϙοτ߈ܸऀ͕๚Ε͍ͯͳ͍͔? • σʔλϕʔε͕ग़ྗ͍ͯ͠ΔErrorϩάରॲ͢Δඞཁ͕ ͋Δͷ͔? Use
Case
Security Analytics • ϚϧΣΞʹ৵ೖ͞Ε͍ͯͳ͍͔? • ෦ऀʹΑΔηΩϡϦςΟڴҖແ͍͔? • DNSͷϩάʹɺσʔλऔͷ͕ࠟͳ͍͔? Use Case
Telemetry / Sensors ▪ ISPͷωοτϫʔΫҰ࣌ःஅʹΑΔϨΠςϯγʔͷٸ ܹͳ૿Ճ? ▪ ଞͱҟͳΔӡసύλʔϯΛͱΔυϥΠόʔ? ▪ ಛҟͳΠϕϯτλΠϓηϯαʔͷނোΛ͔ࣔ͢?
Use Case
10 ҟৗͷൃݟࢥͬͨΑΓ͍͠ • σʔλෳࡶɺߴ࣍ݩɺߴʹมԽ • ਓؒͷࢹೝݱ࣮తʹෆՄೳ • ༰қʹݟಀ͢ Visual inspection
is not practical Where’s the anomaly?
11 ҟৗͷൃݟࢥͬͨΑΓ͍͠ • ੩తͳᮢʹΑΔʮਖ਼ৗʯͷఆٛࠔ • ϧʔϧσʔλΠϯϑϥͷมߋʹैͰ͖ͳ͍ • ༰қʹᷖճ͞Εͯ͠·͏ Rule-based alerts
are insufficient What’s the right threshold ?
X-Pack͕ࣗಈతͳҟৗݕͰղܾ 12 • ʮڭࢣͳ͠ʯػցֶशςΫχοΫʹΑΓ ▪ աڈͷσʔλ͔Βʮਖ਼ৗʯΛֶͼϞσϧΛ࡞Δ ▪ ਖ਼ৗͷൣғ͔Βҳͨ͠ࡍʹҟৗͱͯ͠ݕ
X-Pack͕ࣗಈతͳҟৗݕͰղܾ 13 • ڭࢣͳ͠ - खಈͰͷਖ਼ৗͷೖྗ͕ෆཁ • σʔλͷมԽʹै - ೖ͞ΕΔσʔλʹΑΓܧଓతʹϞσϧΛߋ৽
• ӨڹҼࢠಛఆ - ࠜຊݪҼղੳΛՃ
ҟͳΔछྨͷҟৗΛݕ 14 • ࣌ܥྻͷϝτϦοΫ Time series - single / multiple
• ͙Εऀ Outliers in population (using entity profiling) • ك༗ͳඇߏϝοηʔδ Rare / unusual rates in “categories” of events
࣌ܥྻσʔλͷҟৗ 15 Time Metric • Single (univariate) time series Example:
Is there unusual traffic on website ?
࣌ܥྻσʔλͷҟৗ 16 Time Metric USA UK France Japan • Multiple
time series ▪ ෳͷϝτϦοΫ ▪ FieldʹΑͬͯྨ͞ΕͨϝτϦοΫ • ͦΕͧΕ͕ಠཱͯ͠ଘࡏ͢Δ Example: Is there unusual web activity from any country?
͙Εऀ Outliers in population (using entity profiling) 17 • ूஂͷಛ(server,
user, IPͳͲ)͔ΒϓϩϑΝΠϧΛ࡞͢Δ • ͜ͷूஂ͔Βҳ͢ΔͷΛൃݟ͢Δ Example: • Which IP address is not like the others? (indication of a bot / attacker)
͙Εऀ Outliers in population (using entity profiling) 18 • ूஂͷಛ(server,
user, IPͳͲ)͔ΒϓϩϑΝΠϧΛ࡞͢Δ • ͜ͷूஂ͔Βҳ͢ΔͷΛൃݟ͢Δ Example: • Which IP address is not like the others? (indication of a bot / attacker)
ك༗ͳඇߏϝοηʔδͷมԽ Unusual or rare events (via log categorization) 19 •
ྨࣅੑʹج͍ͮͯΧςΰϦ͚ • ࣌ؒมԽʹΑΔසΛֶश • ϞσϧͱҟͳΕҟৗͱͯ͠ݕ Example: • Do my application logs contain unusual messages
X-Pack Machine Learning Elastic StackͱͷڧݻͳΠϯςάϨʔγϣϯ 20
• Elasticsearch • Kibana ༰қʹΠϯετʔϧ 21 $ elasticsearch-plugin install x-pack
$ kibana-plugin install x-pack
σϓϩΠϝϯτϞσϧ 22 Cluster Data node Apps Master node Data node
Data node Master node Master node Data node Data node ES clients, Kibana, Logstash, Beats, User apps and etc. ML node ML node # config/elasticsearch.yml xpack.ml.enabled: true node.ml: true
֎෦γεςϜͱͷଓ • API (anomaly_detectors, datafeeds, results, model_snapshots, validate) • ΠϯσοΫε
(.ml-anomalies-*)
Taking Action with X-Pack Alerting 24
Demo Single/Multiple Metrics: New York City Yellow Taxi Outliers in
Population: Web Server Log Rare Messages: DBMS Server Log 25
26 4JOHMF.FUSJD
27 .VMUJ.FUSJD
28 .VMUJ.FUSJD
29 0VUMJFSTJO1PQVMBUJPO
30 0VUMJFSTJO1PQVMBUJPO
31 3BSF.FTTBHFT
32 3BSF.FTTBHFT
࣍ͷεςοϓ 33 • Elastic StackΛ·ͩར༻͍ͯ͠ͳ͍ • ϋϯζΦϯϫʔΫγϣοϓ • Elastic StackɺX-PackΛΠϯετʔϧ
• αϯϓϧσʔλΛར༻ (ϒϩάࢀর) or ࣗͷσʔλΛೖ • MLδϣϒΛ࡞ • Elastic StackΛར༻த • X-PackΛΠϯετʔϧ (30ؒͷτϥΠΞϧ/ඇϓϩμΫγϣϯڥ) • MLδϣϒΛ࡞ (Ϩγϐ׆༻) • AlertingͰΞΫγϣϯ