Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Introducing Machine Learning for the Elastic Stack
Search
Kosho Owa
May 19, 2017
Technology
2
12k
Introducing Machine Learning for the Elastic Stack
Elastic Machine Learning Seminar held on May 19th, 2017
Kosho Owa
May 19, 2017
Tweet
Share
More Decks by Kosho Owa
See All by Kosho Owa
Elastic Stack X-Pack 5.0 for IT Security Workshop
kosho
1
310
Elastic Stack X-Pack 5.0 for IT Ops Workshop
kosho
0
320
[Developers Summit 2017] Anomaly Detection with the Elastic Stack
kosho
1
700
Anomaly Detection with the Elastic Stack
kosho
1
1.8k
Getting Started with Elastic Cloud and Beats for Log Analytics
kosho
0
96
Elastic{ON} Seminar Tokyo 2016 Product Update
kosho
0
160
Introducing Elastic Cloud
kosho
0
76
Gearing Up for Elastic Stack, X-Pack 5.0 Releases
kosho
0
140
Elastic Stack Hands-on Workshop (EN)
kosho
1
160
Other Decks in Technology
See All in Technology
データプラットフォーム技術におけるメダリオンアーキテクチャという考え方/DataPlatformWithMedallionArchitecture
smdmts
5
660
本が全く読めなかった過去の自分へ
genshun9
0
640
AWS Organizations 新機能!マルチパーティ承認の紹介
yhana
1
180
LangChain Interrupt & LangChain Ambassadors meetingレポート
os1ma
2
130
BrainPadプログラミングコンテスト記念LT会2025_社内イベント&問題解説
brainpadpr
1
170
KubeCon + CloudNativeCon Japan 2025 Recap by CA
ponkio_o
PRO
0
210
なぜ私はいま、ここにいるのか? #もがく中堅デザイナー #プロダクトデザイナー
bengo4com
0
1.1k
生成AI開発案件におけるClineの業務活用事例とTips
shinya337
0
110
標準技術と独自システムで作る「つらくない」SaaS アカウント管理 / Effortless SaaS Account Management with Standard Technologies & Custom Systems
yuyatakeyama
3
1.3k
Lambda Web Adapterについて自分なりに理解してみた
smt7174
5
130
Oracle Cloud Infrastructure:2025年6月度サービス・アップデート
oracle4engineer
PRO
2
280
WordPressから ヘッドレスCMSへ! Storyblokへの移行プロセス
nyata
0
200
Featured
See All Featured
Building Flexible Design Systems
yeseniaperezcruz
328
39k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
5
230
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
7
710
VelocityConf: Rendering Performance Case Studies
addyosmani
331
24k
Measuring & Analyzing Core Web Vitals
bluesmoon
7
490
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
130
19k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
34
5.9k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.5k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
32
2.3k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
161
15k
Building an army of robots
kneath
306
45k
The Straight Up "How To Draw Better" Workshop
denniskardys
234
140k
Transcript
Machine Learning for the Elastic Stack Beta in 5.4.
GA coming soon May 2017 େྠ ߂ৄ | Kosho Owa Solutions Architect, Elastic
2 Elastic Stack 100% Φʔϓϯιʔε ʮΤϯλʔϓϥΠζ൛ʯແ͠ όʔδϣϯ 5.0Ͱશ౷Ұ
3 X-Pack ؆୯ʹΠϯετʔϧ Elastic StackΛ֦ு αϒεΫϦϓγϣϯʹؚ·ΕΔ Security Alerting Monitoring Reporting
Graph Machine Learning
4 Elastic Cloud Elasticsearch, Kibanaͷ ϚωʔδυαʔϏε X-Packͷػೳར༻Մೳ Available in AWS
today
5 Elastic Cloud Enterprise ෳͷElastic StackڥΛࣗࡏʹ࡞ Logging as a serviceΛࣗ৫ʹల։
Public beta; Expected GA Q1 2017
ҟৗͷൃݟ͕τϥϒϧͷஹީΛࣔ͢ 6 Spiked 404 errors Web attack IT Operational Analytics
Security Analytics Business Analytics Unusual DNS activity Data exfiltration Rare log messages Failing sensor
Operational Analytics • ΣϒαΠτͷΞΫηετϥϑΟοΫʹҟৗແ͍͔? • Ϙοτ߈ܸऀ͕๚Ε͍ͯͳ͍͔? • σʔλϕʔε͕ग़ྗ͍ͯ͠ΔErrorϩάରॲ͢Δඞཁ͕ ͋Δͷ͔? Use
Case
Security Analytics • ϚϧΣΞʹ৵ೖ͞Ε͍ͯͳ͍͔? • ෦ऀʹΑΔηΩϡϦςΟڴҖແ͍͔? • DNSͷϩάʹɺσʔλऔͷ͕ࠟͳ͍͔? Use Case
Telemetry / Sensors ▪ ISPͷωοτϫʔΫҰ࣌ःஅʹΑΔϨΠςϯγʔͷٸ ܹͳ૿Ճ? ▪ ଞͱҟͳΔӡసύλʔϯΛͱΔυϥΠόʔ? ▪ ಛҟͳΠϕϯτλΠϓηϯαʔͷނোΛ͔ࣔ͢?
Use Case
10 ҟৗͷൃݟࢥͬͨΑΓ͍͠ • σʔλෳࡶɺߴ࣍ݩɺߴʹมԽ • ਓؒͷࢹೝݱ࣮తʹෆՄೳ • ༰қʹݟಀ͢ Visual inspection
is not practical Where’s the anomaly?
11 ҟৗͷൃݟࢥͬͨΑΓ͍͠ • ੩తͳᮢʹΑΔʮਖ਼ৗʯͷఆٛࠔ • ϧʔϧσʔλΠϯϑϥͷมߋʹैͰ͖ͳ͍ • ༰қʹᷖճ͞Εͯ͠·͏ Rule-based alerts
are insufficient What’s the right threshold ?
X-Pack͕ࣗಈతͳҟৗݕͰղܾ 12 • ʮڭࢣͳ͠ʯػցֶशςΫχοΫʹΑΓ ▪ աڈͷσʔλ͔Βʮਖ਼ৗʯΛֶͼϞσϧΛ࡞Δ ▪ ਖ਼ৗͷൣғ͔Βҳͨ͠ࡍʹҟৗͱͯ͠ݕ
X-Pack͕ࣗಈతͳҟৗݕͰղܾ 13 • ڭࢣͳ͠ - खಈͰͷਖ਼ৗͷೖྗ͕ෆཁ • σʔλͷมԽʹै - ೖ͞ΕΔσʔλʹΑΓܧଓతʹϞσϧΛߋ৽
• ӨڹҼࢠಛఆ - ࠜຊݪҼղੳΛՃ
ҟͳΔछྨͷҟৗΛݕ 14 • ࣌ܥྻͷϝτϦοΫ Time series - single / multiple
• ͙Εऀ Outliers in population (using entity profiling) • ك༗ͳඇߏϝοηʔδ Rare / unusual rates in “categories” of events
࣌ܥྻσʔλͷҟৗ 15 Time Metric • Single (univariate) time series Example:
Is there unusual traffic on website ?
࣌ܥྻσʔλͷҟৗ 16 Time Metric USA UK France Japan • Multiple
time series ▪ ෳͷϝτϦοΫ ▪ FieldʹΑͬͯྨ͞ΕͨϝτϦοΫ • ͦΕͧΕ͕ಠཱͯ͠ଘࡏ͢Δ Example: Is there unusual web activity from any country?
͙Εऀ Outliers in population (using entity profiling) 17 • ूஂͷಛ(server,
user, IPͳͲ)͔ΒϓϩϑΝΠϧΛ࡞͢Δ • ͜ͷूஂ͔Βҳ͢ΔͷΛൃݟ͢Δ Example: • Which IP address is not like the others? (indication of a bot / attacker)
͙Εऀ Outliers in population (using entity profiling) 18 • ूஂͷಛ(server,
user, IPͳͲ)͔ΒϓϩϑΝΠϧΛ࡞͢Δ • ͜ͷूஂ͔Βҳ͢ΔͷΛൃݟ͢Δ Example: • Which IP address is not like the others? (indication of a bot / attacker)
ك༗ͳඇߏϝοηʔδͷมԽ Unusual or rare events (via log categorization) 19 •
ྨࣅੑʹج͍ͮͯΧςΰϦ͚ • ࣌ؒมԽʹΑΔසΛֶश • ϞσϧͱҟͳΕҟৗͱͯ͠ݕ Example: • Do my application logs contain unusual messages
X-Pack Machine Learning Elastic StackͱͷڧݻͳΠϯςάϨʔγϣϯ 20
• Elasticsearch • Kibana ༰қʹΠϯετʔϧ 21 $ elasticsearch-plugin install x-pack
$ kibana-plugin install x-pack
σϓϩΠϝϯτϞσϧ 22 Cluster Data node Apps Master node Data node
Data node Master node Master node Data node Data node ES clients, Kibana, Logstash, Beats, User apps and etc. ML node ML node # config/elasticsearch.yml xpack.ml.enabled: true node.ml: true
֎෦γεςϜͱͷଓ • API (anomaly_detectors, datafeeds, results, model_snapshots, validate) • ΠϯσοΫε
(.ml-anomalies-*)
Taking Action with X-Pack Alerting 24
Demo Single/Multiple Metrics: New York City Yellow Taxi Outliers in
Population: Web Server Log Rare Messages: DBMS Server Log 25
26 4JOHMF.FUSJD
27 .VMUJ.FUSJD
28 .VMUJ.FUSJD
29 0VUMJFSTJO1PQVMBUJPO
30 0VUMJFSTJO1PQVMBUJPO
31 3BSF.FTTBHFT
32 3BSF.FTTBHFT
࣍ͷεςοϓ 33 • Elastic StackΛ·ͩར༻͍ͯ͠ͳ͍ • ϋϯζΦϯϫʔΫγϣοϓ • Elastic StackɺX-PackΛΠϯετʔϧ
• αϯϓϧσʔλΛར༻ (ϒϩάࢀর) or ࣗͷσʔλΛೖ • MLδϣϒΛ࡞ • Elastic StackΛར༻த • X-PackΛΠϯετʔϧ (30ؒͷτϥΠΞϧ/ඇϓϩμΫγϣϯڥ) • MLδϣϒΛ࡞ (Ϩγϐ׆༻) • AlertingͰΞΫγϣϯ