Introducing Machine Learning for the Elastic Stack

Transcript

1. Machine Learning for the Elastic Stack 
 Beta in 5.4.

   GA coming soon May 2017 େྠ ߂ৄ | Kosho Owa Solutions Architect, Elastic

2. 2 Elastic Stack 100% Φʔϓϯιʔε ʮΤϯλʔϓϥΠζ൛ʯ͸ແ͠ όʔδϣϯ 5.0Ͱ׬શ౷Ұ

3. 3 X-Pack ؆୯ʹΠϯετʔϧ Elastic StackΛ֦ு αϒεΫϦϓγϣϯʹؚ·ΕΔ Security Alerting Monitoring Reporting

   Graph Machine Learning

4. 4 Elastic Cloud Elasticsearch, Kibanaͷ ϚωʔδυαʔϏε X-Packͷػೳ΋ར༻Մೳ Available in AWS

   today

5. 5 Elastic Cloud Enterprise ෳ਺ͷElastic Stack؀ڥΛࣗࡏʹ࡞੒ Logging as a serviceΛࣗ૊৫ʹల։

   Public beta; Expected GA Q1 2017

6. ҟৗͷൃݟ͕τϥϒϧͷஹީΛࣔ͢ 6 Spiked 404 errors Web attack IT Operational Analytics

   Security Analytics Business Analytics Unusual DNS activity Data exfiltration Rare log messages Failing sensor

7. Operational Analytics • ΢ΣϒαΠτ΁ͷΞΫηετϥϑΟοΫʹҟৗ͸ແ͍͔? • Ϙοτ΍߈ܸऀ͕๚Ε͍ͯͳ͍͔? • σʔλϕʔε͕ग़ྗ͍ͯ͠ΔErrorϩά͸ରॲ͢Δඞཁ͕ ͋Δͷ͔? Use

   Case

8. Security Analytics • Ϛϧ΢ΣΞʹ৵ೖ͞Ε͍ͯͳ͍͔? • ಺෦ऀʹΑΔηΩϡϦςΟڴҖ͸ແ͍͔? • DNSͷϩάʹ͸ɺσʔλ࠮औͷࠟ੻͕ͳ͍͔? Use Case

9. Telemetry / Sensors ▪ ISPͷωοτϫʔΫҰ࣌ःஅʹΑΔϨΠςϯγʔͷٸ ܹͳ૿Ճ͸? ▪ ଞͱ͸ҟͳΔӡసύλʔϯΛͱΔυϥΠόʔ͸? ▪ ಛҟͳΠϕϯτλΠϓ͸ηϯαʔͷނোΛ͔ࣔ͢?

   Use Case

10. 10 ҟৗͷൃݟ͸ࢥͬͨΑΓ΋೉͍͠ • σʔλ͸ෳࡶɺߴ࣍ݩɺߴ଎ʹมԽ • ਓؒͷࢹೝ͸ݱ࣮తʹෆՄೳ • ༰қʹݟಀ͢ Visual inspection

    is not practical Where’s the anomaly?

11. 11 ҟৗͷൃݟ͸ࢥͬͨΑΓ΋೉͍͠ • ੩తͳᮢ஋ʹΑΔʮਖ਼ৗʯͷఆٛ͸ࠔ೉ • ϧʔϧ͸σʔλ͸Πϯϑϥͷมߋʹ௥ैͰ͖ͳ͍ • ༰қʹᷖճ͞Εͯ͠·͏ Rule-based alerts

    are insufficient What’s the right threshold ?

12. X-Pack͕ࣗಈతͳҟৗݕ஌Ͱղܾ 12 • ʮڭࢣͳ͠ʯػցֶशςΫχοΫʹΑΓ ▪ աڈͷσʔλ͔Βʮਖ਼ৗʯΛֶͼϞσϧΛ࡞Δ ▪ ਖ਼ৗ஋ͷൣғ͔Βҳ୤ͨ͠ࡍʹҟৗͱͯ͠ݕ஌

13. X-Pack͕ࣗಈతͳҟৗݕ஌Ͱղܾ 13 • ڭࢣͳ͠ - खಈͰͷਖ਼ৗ஋ͷೖྗ͕ෆཁ • σʔλͷมԽʹ௥ै - ౤ೖ͞ΕΔσʔλʹΑΓܧଓతʹϞσϧΛߋ৽

    • ӨڹҼࢠಛఆ - ࠜຊݪҼղੳΛՃ଎

14. ҟͳΔछྨͷҟৗΛݕ஌ 14 • ࣌ܥྻͷϝτϦοΫ Time series - single / multiple

    • ͸͙Εऀ Outliers in population (using entity profiling) • ك༗ͳඇߏ଄ϝοηʔδ Rare / unusual rates in “categories” of events

15. ࣌ܥྻσʔλͷҟৗ 15 Time Metric • Single (univariate) time series Example:

    Is there unusual traffic on website ?

16. ࣌ܥྻσʔλͷҟৗ 16 Time Metric USA UK France Japan • Multiple

    time series ▪ ෳ਺ͷϝτϦοΫ ▪ FieldʹΑͬͯ෼ྨ͞ΕͨϝτϦοΫ • ͦΕͧΕ͕ಠཱͯ͠ଘࡏ͢Δ Example: Is there unusual web activity from any country?

17. ͸͙Εऀ Outliers in population (using entity profiling) 17 • ूஂͷಛ௃(server,

    user, IPͳͲ)͔ΒϓϩϑΝΠϧΛ࡞੒͢Δ • ͜ͷूஂ͔Βҳ୤͢Δ΋ͷΛൃݟ͢Δ Example: • Which IP address is not like the others? (indication of a bot / attacker)


18. ͸͙Εऀ Outliers in population (using entity profiling) 18 • ूஂͷಛ௃(server,

    user, IPͳͲ)͔ΒϓϩϑΝΠϧΛ࡞੒͢Δ • ͜ͷूஂ͔Βҳ୤͢Δ΋ͷΛൃݟ͢Δ Example: • Which IP address is not like the others? (indication of a bot / attacker)


19. ك༗ͳඇߏ଄ϝοηʔδͷมԽ Unusual or rare events (via log categorization) 19 •

    ྨࣅੑʹج͍ͮͯΧςΰϦ෼͚ • ࣌ؒมԽʹΑΔස౓Λֶश • ϞσϧͱҟͳΕ͹ҟৗͱͯ͠ݕ஌ Example: • Do my application logs contain unusual messages

20. X-Pack Machine Learning Elastic StackͱͷڧݻͳΠϯςάϨʔγϣϯ 20

21. • Elasticsearch • Kibana ༰қʹΠϯετʔϧ 21 $ elasticsearch-plugin install x-pack

    $ kibana-plugin install x-pack

22. σϓϩΠϝϯτϞσϧ 22 Cluster Data node Apps Master node Data node

    Data node Master node Master node Data node Data node ES clients, Kibana, Logstash, Beats, User apps and etc. ML node ML node # config/elasticsearch.yml xpack.ml.enabled: true node.ml: true

23. ֎෦γεςϜͱͷ઀ଓ • API (anomaly_detectors, datafeeds, results, model_snapshots, validate) • ΠϯσοΫε

    (.ml-anomalies-*)

24. Taking Action with X-Pack Alerting 24

25. Demo Single/Multiple Metrics: New York City Yellow Taxi Outliers in

    Population: Web Server Log Rare Messages: DBMS Server Log 25

26. 26 4JOHMF
    .FUSJD

27. 27 .VMUJ
    .FUSJD

28. 28 .VMUJ
    .FUSJD

29. 29 0VUMJFST
    JO
    1PQVMBUJPO

30. 30 0VUMJFST
    JO
    1PQVMBUJPO

31. 31 3BSF
    .FTTBHFT

32. 32 3BSF
    .FTTBHFT

33. ࣍ͷεςοϓ 33 • Elastic StackΛ·ͩར༻͍ͯ͠ͳ͍ • ϋϯζΦϯϫʔΫγϣοϓ • Elastic StackɺX-PackΛΠϯετʔϧ

    • αϯϓϧσʔλΛར༻ (ϒϩάࢀর) or ࣗ਎ͷσʔλΛ౤ೖ • MLδϣϒΛ࡞੒ • Elastic StackΛར༻த • X-PackΛΠϯετʔϧ (30೔ؒͷτϥΠΞϧ/ඇϓϩμΫγϣϯ؀ڥ) • MLδϣϒΛ࡞੒ (Ϩγϐ΋׆༻) • AlertingͰΞΫγϣϯ