Introducing Machine Learning for the Elastic Stack

Elastic Machine Learning Seminar held on May 19th, 2017

Transcript

1. Machine Learning for the Elastic Stack 
 Beta in 5.4. GA coming soon May 2017 େྠ ߂ৄ | Kosho Owa Solutions Architect, Elastic
2. 2 Elastic Stack 100% Φʔϓϯιʔε ʮΤϯλʔϓϥΠζ൛ʯ͸ແ͠ όʔδϣϯ 5.0Ͱ׬શ౷Ұ
3. 3 X-Pack ؆୯ʹΠϯετʔϧ Elastic StackΛ֦ு αϒεΫϦϓγϣϯʹؚ·ΕΔ Security Alerting Monitoring Reporting Graph Machine Learning
4. 4 Elastic Cloud Elasticsearch, Kibanaͷ ϚωʔδυαʔϏε X-Packͷػೳ΋ར༻Մೳ Available in AWS today
5. 5 Elastic Cloud Enterprise ෳ਺ͷElastic Stack؀ڥΛࣗࡏʹ࡞੒ Logging as a serviceΛࣗ૊৫ʹల։ Public beta; Expected GA Q1 2017
6. ҟৗͷൃݟ͕τϥϒϧͷஹީΛࣔ͢ 6 Spiked 404 errors Web attack IT Operational Analytics Security Analytics Business Analytics Unusual DNS activity Data exfiltration Rare log messages Failing sensor
7. Operational Analytics • ΢ΣϒαΠτ΁ͷΞΫηετϥϑΟοΫʹҟৗ͸ແ͍͔? • Ϙοτ΍߈ܸऀ͕๚Ε͍ͯͳ͍͔? • σʔλϕʔε͕ग़ྗ͍ͯ͠ΔErrorϩά͸ରॲ͢Δඞཁ͕ ͋Δͷ͔? Use Case
8. Security Analytics • Ϛϧ΢ΣΞʹ৵ೖ͞Ε͍ͯͳ͍͔? • ಺෦ऀʹΑΔηΩϡϦςΟڴҖ͸ແ͍͔? • DNSͷϩάʹ͸ɺσʔλ࠮औͷࠟ੻͕ͳ͍͔? Use Case
9. Telemetry / Sensors ▪ ISPͷωοτϫʔΫҰ࣌ःஅʹΑΔϨΠςϯγʔͷٸ ܹͳ૿Ճ͸? ▪ ଞͱ͸ҟͳΔӡసύλʔϯΛͱΔυϥΠόʔ͸? ▪ ಛҟͳΠϕϯτλΠϓ͸ηϯαʔͷނোΛ͔ࣔ͢? Use Case
10. 10 ҟৗͷൃݟ͸ࢥͬͨΑΓ΋೉͍͠ • σʔλ͸ෳࡶɺߴ࣍ݩɺߴ଎ʹมԽ • ਓؒͷࢹೝ͸ݱ࣮తʹෆՄೳ • ༰қʹݟಀ͢ Visual inspection is not practical Where’s the anomaly?
11. 11 ҟৗͷൃݟ͸ࢥͬͨΑΓ΋೉͍͠ • ੩తͳᮢ஋ʹΑΔʮਖ਼ৗʯͷఆٛ͸ࠔ೉ • ϧʔϧ͸σʔλ͸Πϯϑϥͷมߋʹ௥ैͰ͖ͳ͍ • ༰қʹᷖճ͞Εͯ͠·͏ Rule-based alerts are insufficient What’s the right threshold ?
12. X-Pack͕ࣗಈతͳҟৗݕ஌Ͱղܾ 12 • ʮڭࢣͳ͠ʯػցֶशςΫχοΫʹΑΓ ▪ աڈͷσʔλ͔Βʮਖ਼ৗʯΛֶͼϞσϧΛ࡞Δ ▪ ਖ਼ৗ஋ͷൣғ͔Βҳ୤ͨ͠ࡍʹҟৗͱͯ͠ݕ஌
13. X-Pack͕ࣗಈతͳҟৗݕ஌Ͱղܾ 13 • ڭࢣͳ͠ - खಈͰͷਖ਼ৗ஋ͷೖྗ͕ෆཁ • σʔλͷมԽʹ௥ै - ౤ೖ͞ΕΔσʔλʹΑΓܧଓతʹϞσϧΛߋ৽ • ӨڹҼࢠಛఆ - ࠜຊݪҼղੳΛՃ଎
14. ҟͳΔछྨͷҟৗΛݕ஌ 14 • ࣌ܥྻͷϝτϦοΫ Time series - single / multiple • ͸͙Εऀ Outliers in population (using entity profiling) • ك༗ͳඇߏ଄ϝοηʔδ Rare / unusual rates in “categories” of events
15. ࣌ܥྻσʔλͷҟৗ 15 Time Metric • Single (univariate) time series Example: Is there unusual traffic on website ?
16. ࣌ܥྻσʔλͷҟৗ 16 Time Metric USA UK France Japan • Multiple time series ▪ ෳ਺ͷϝτϦοΫ ▪ FieldʹΑͬͯ෼ྨ͞ΕͨϝτϦοΫ • ͦΕͧΕ͕ಠཱͯ͠ଘࡏ͢Δ Example: Is there unusual web activity from any country?
17. ͸͙Εऀ Outliers in population (using entity profiling) 17 • ूஂͷಛ௃(server, user, IPͳͲ)͔ΒϓϩϑΝΠϧΛ࡞੒͢Δ • ͜ͷूஂ͔Βҳ୤͢Δ΋ͷΛൃݟ͢Δ Example: • Which IP address is not like the others? (indication of a bot / attacker)

18. ͸͙Εऀ Outliers in population (using entity profiling) 18 • ूஂͷಛ௃(server, user, IPͳͲ)͔ΒϓϩϑΝΠϧΛ࡞੒͢Δ • ͜ͷूஂ͔Βҳ୤͢Δ΋ͷΛൃݟ͢Δ Example: • Which IP address is not like the others? (indication of a bot / attacker)

19. ك༗ͳඇߏ଄ϝοηʔδͷมԽ Unusual or rare events (via log categorization) 19 • ྨࣅੑʹج͍ͮͯΧςΰϦ෼͚ • ࣌ؒมԽʹΑΔස౓Λֶश • ϞσϧͱҟͳΕ͹ҟৗͱͯ͠ݕ஌ Example: • Do my application logs contain unusual messages
20. X-Pack Machine Learning Elastic StackͱͷڧݻͳΠϯςάϨʔγϣϯ 20
21. • Elasticsearch • Kibana ༰қʹΠϯετʔϧ 21 $ elasticsearch-plugin install x-pack $ kibana-plugin install x-pack
22. σϓϩΠϝϯτϞσϧ 22 Cluster Data node Apps Master node Data node Data node Master node Master node Data node Data node ES clients, Kibana, Logstash, Beats, User apps and etc. ML node ML node # config/elasticsearch.yml xpack.ml.enabled: true node.ml: true
23. ֎෦γεςϜͱͷ઀ଓ • API (anomaly_detectors, datafeeds, results, model_snapshots, validate) • ΠϯσοΫε (.ml-anomalies-*)
24. Taking Action with X-Pack Alerting 24
25. Demo Single/Multiple Metrics: New York City Yellow Taxi Outliers in Population: Web Server Log Rare Messages: DBMS Server Log 25
26. 26 4JOHMF
    .FUSJD
27. 27 .VMUJ
    .FUSJD
28. 28 .VMUJ
    .FUSJD
29. 29 0VUMJFST
    JO
    1PQVMBUJPO
30. 30 0VUMJFST
    JO
    1PQVMBUJPO
31. 31 3BSF
    .FTTBHFT
32. 32 3BSF
    .FTTBHFT
33. ࣍ͷεςοϓ 33 • Elastic StackΛ·ͩར༻͍ͯ͠ͳ͍ • ϋϯζΦϯϫʔΫγϣοϓ • Elastic StackɺX-PackΛΠϯετʔϧ • αϯϓϧσʔλΛར༻ (ϒϩάࢀর) or ࣗ਎ͷσʔλΛ౤ೖ • MLδϣϒΛ࡞੒ • Elastic StackΛར༻த • X-PackΛΠϯετʔϧ (30೔ؒͷτϥΠΞϧ/ඇϓϩμΫγϣϯ؀ڥ) • MLδϣϒΛ࡞੒ (Ϩγϐ΋׆༻) • AlertingͰΞΫγϣϯ