Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introducing Machine Learning for the Elastic Stack

Introducing Machine Learning for the Elastic Stack

Elastic Machine Learning Seminar held on May 19th, 2017

34cbde72de5f384380d5489543294dc5?s=128

Kosho Owa

May 19, 2017
Tweet

Transcript

  1. Machine Learning for the Elastic Stack 
 Beta in 5.4.

    GA coming soon May 2017 େྠ ߂ৄ | Kosho Owa Solutions Architect, Elastic
  2. 2 Elastic Stack 100% Φʔϓϯιʔε ʮΤϯλʔϓϥΠζ൛ʯ͸ແ͠ όʔδϣϯ 5.0Ͱ׬શ౷Ұ

  3. 3 X-Pack ؆୯ʹΠϯετʔϧ Elastic StackΛ֦ு αϒεΫϦϓγϣϯʹؚ·ΕΔ Security Alerting Monitoring Reporting

    Graph Machine Learning
  4. 4 Elastic Cloud Elasticsearch, Kibanaͷ ϚωʔδυαʔϏε X-Packͷػೳ΋ར༻Մೳ Available in AWS

    today
  5. 5 Elastic Cloud Enterprise ෳ਺ͷElastic Stack؀ڥΛࣗࡏʹ࡞੒ Logging as a serviceΛࣗ૊৫ʹల։

    Public beta; Expected GA Q1 2017
  6. ҟৗͷൃݟ͕τϥϒϧͷஹީΛࣔ͢ 6 Spiked 404 errors Web attack IT Operational Analytics

    Security Analytics Business Analytics Unusual DNS activity Data exfiltration Rare log messages Failing sensor
  7. Operational Analytics • ΢ΣϒαΠτ΁ͷΞΫηετϥϑΟοΫʹҟৗ͸ແ͍͔? • Ϙοτ΍߈ܸऀ͕๚Ε͍ͯͳ͍͔? • σʔλϕʔε͕ग़ྗ͍ͯ͠ΔErrorϩά͸ରॲ͢Δඞཁ͕ ͋Δͷ͔? Use

    Case
  8. Security Analytics • Ϛϧ΢ΣΞʹ৵ೖ͞Ε͍ͯͳ͍͔? • ಺෦ऀʹΑΔηΩϡϦςΟڴҖ͸ແ͍͔? • DNSͷϩάʹ͸ɺσʔλ࠮औͷࠟ੻͕ͳ͍͔? Use Case

  9. Telemetry / Sensors ▪ ISPͷωοτϫʔΫҰ࣌ःஅʹΑΔϨΠςϯγʔͷٸ ܹͳ૿Ճ͸? ▪ ଞͱ͸ҟͳΔӡసύλʔϯΛͱΔυϥΠόʔ͸? ▪ ಛҟͳΠϕϯτλΠϓ͸ηϯαʔͷނোΛ͔ࣔ͢?

    Use Case
  10. 10 ҟৗͷൃݟ͸ࢥͬͨΑΓ΋೉͍͠ • σʔλ͸ෳࡶɺߴ࣍ݩɺߴ଎ʹมԽ • ਓؒͷࢹೝ͸ݱ࣮తʹෆՄೳ • ༰қʹݟಀ͢ Visual inspection

    is not practical Where’s the anomaly?
  11. 11 ҟৗͷൃݟ͸ࢥͬͨΑΓ΋೉͍͠ • ੩తͳᮢ஋ʹΑΔʮਖ਼ৗʯͷఆٛ͸ࠔ೉ • ϧʔϧ͸σʔλ͸Πϯϑϥͷมߋʹ௥ैͰ͖ͳ͍ • ༰қʹᷖճ͞Εͯ͠·͏ Rule-based alerts

    are insufficient What’s the right threshold ?
  12. X-Pack͕ࣗಈతͳҟৗݕ஌Ͱղܾ 12 • ʮڭࢣͳ͠ʯػցֶशςΫχοΫʹΑΓ ▪ աڈͷσʔλ͔Βʮਖ਼ৗʯΛֶͼϞσϧΛ࡞Δ ▪ ਖ਼ৗ஋ͷൣғ͔Βҳ୤ͨ͠ࡍʹҟৗͱͯ͠ݕ஌

  13. X-Pack͕ࣗಈతͳҟৗݕ஌Ͱղܾ 13 • ڭࢣͳ͠ - खಈͰͷਖ਼ৗ஋ͷೖྗ͕ෆཁ • σʔλͷมԽʹ௥ै - ౤ೖ͞ΕΔσʔλʹΑΓܧଓతʹϞσϧΛߋ৽

    • ӨڹҼࢠಛఆ - ࠜຊݪҼղੳΛՃ଎
  14. ҟͳΔछྨͷҟৗΛݕ஌ 14 • ࣌ܥྻͷϝτϦοΫ Time series - single / multiple

    • ͸͙Εऀ Outliers in population (using entity profiling) • ك༗ͳඇߏ଄ϝοηʔδ Rare / unusual rates in “categories” of events
  15. ࣌ܥྻσʔλͷҟৗ 15 Time Metric • Single (univariate) time series Example:

    Is there unusual traffic on website ?
  16. ࣌ܥྻσʔλͷҟৗ 16 Time Metric USA UK France Japan • Multiple

    time series ▪ ෳ਺ͷϝτϦοΫ ▪ FieldʹΑͬͯ෼ྨ͞ΕͨϝτϦοΫ • ͦΕͧΕ͕ಠཱͯ͠ଘࡏ͢Δ Example: Is there unusual web activity from any country?
  17. ͸͙Εऀ Outliers in population (using entity profiling) 17 • ूஂͷಛ௃(server,

    user, IPͳͲ)͔ΒϓϩϑΝΠϧΛ࡞੒͢Δ • ͜ͷूஂ͔Βҳ୤͢Δ΋ͷΛൃݟ͢Δ Example: • Which IP address is not like the others? (indication of a bot / attacker)

  18. ͸͙Εऀ Outliers in population (using entity profiling) 18 • ूஂͷಛ௃(server,

    user, IPͳͲ)͔ΒϓϩϑΝΠϧΛ࡞੒͢Δ • ͜ͷूஂ͔Βҳ୤͢Δ΋ͷΛൃݟ͢Δ Example: • Which IP address is not like the others? (indication of a bot / attacker)

  19. ك༗ͳඇߏ଄ϝοηʔδͷมԽ Unusual or rare events (via log categorization) 19 •

    ྨࣅੑʹج͍ͮͯΧςΰϦ෼͚ • ࣌ؒมԽʹΑΔස౓Λֶश • ϞσϧͱҟͳΕ͹ҟৗͱͯ͠ݕ஌ Example: • Do my application logs contain unusual messages
  20. X-Pack Machine Learning Elastic StackͱͷڧݻͳΠϯςάϨʔγϣϯ 20

  21. • Elasticsearch • Kibana ༰қʹΠϯετʔϧ 21 $ elasticsearch-plugin install x-pack

    $ kibana-plugin install x-pack
  22. σϓϩΠϝϯτϞσϧ 22 Cluster Data node Apps Master node Data node

    Data node Master node Master node Data node Data node ES clients, Kibana, Logstash, Beats, User apps and etc. ML node ML node # config/elasticsearch.yml xpack.ml.enabled: true node.ml: true
  23. ֎෦γεςϜͱͷ઀ଓ • API (anomaly_detectors, datafeeds, results, model_snapshots, validate) • ΠϯσοΫε

    (.ml-anomalies-*)
  24. Taking Action with X-Pack Alerting 24

  25. Demo Single/Multiple Metrics: New York City Yellow Taxi Outliers in

    Population: Web Server Log Rare Messages: DBMS Server Log 25
  26. 26 4JOHMF.FUSJD

  27. 27 .VMUJ.FUSJD

  28. 28 .VMUJ.FUSJD

  29. 29 0VUMJFSTJO1PQVMBUJPO

  30. 30 0VUMJFSTJO1PQVMBUJPO

  31. 31 3BSF.FTTBHFT

  32. 32 3BSF.FTTBHFT

  33. ࣍ͷεςοϓ 33 • Elastic StackΛ·ͩར༻͍ͯ͠ͳ͍ • ϋϯζΦϯϫʔΫγϣοϓ • Elastic StackɺX-PackΛΠϯετʔϧ

    • αϯϓϧσʔλΛར༻ (ϒϩάࢀর) or ࣗ਎ͷσʔλΛ౤ೖ • MLδϣϒΛ࡞੒ • Elastic StackΛར༻த • X-PackΛΠϯετʔϧ (30೔ؒͷτϥΠΞϧ/ඇϓϩμΫγϣϯ؀ڥ) • MLδϣϒΛ࡞੒ (Ϩγϐ΋׆༻) • AlertingͰΞΫγϣϯ