Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introducing Machine Learning for the Elastic Stack

Introducing Machine Learning for the Elastic Stack

Elastic Machine Learning Seminar held on May 19th, 2017

Kosho Owa

May 19, 2017
Tweet

More Decks by Kosho Owa

Other Decks in Technology

Transcript

  1. Machine Learning for the Elastic Stack 
 Beta in 5.4.

    GA coming soon May 2017 େྠ ߂ৄ | Kosho Owa Solutions Architect, Elastic
  2. ҟৗͷൃݟ͕τϥϒϧͷஹީΛࣔ͢ 6 Spiked 404 errors Web attack IT Operational Analytics

    Security Analytics Business Analytics Unusual DNS activity Data exfiltration Rare log messages Failing sensor
  3. ҟͳΔछྨͷҟৗΛݕ஌ 14 • ࣌ܥྻͷϝτϦοΫ Time series - single / multiple

    • ͸͙Εऀ Outliers in population (using entity profiling) • ك༗ͳඇߏ଄ϝοηʔδ Rare / unusual rates in “categories” of events
  4. ࣌ܥྻσʔλͷҟৗ 16 Time Metric USA UK France Japan • Multiple

    time series ▪ ෳ਺ͷϝτϦοΫ ▪ FieldʹΑͬͯ෼ྨ͞ΕͨϝτϦοΫ • ͦΕͧΕ͕ಠཱͯ͠ଘࡏ͢Δ Example: Is there unusual web activity from any country?
  5. ͸͙Εऀ Outliers in population (using entity profiling) 17 • ूஂͷಛ௃(server,

    user, IPͳͲ)͔ΒϓϩϑΝΠϧΛ࡞੒͢Δ • ͜ͷूஂ͔Βҳ୤͢Δ΋ͷΛൃݟ͢Δ Example: • Which IP address is not like the others? (indication of a bot / attacker)

  6. ͸͙Εऀ Outliers in population (using entity profiling) 18 • ूஂͷಛ௃(server,

    user, IPͳͲ)͔ΒϓϩϑΝΠϧΛ࡞੒͢Δ • ͜ͷूஂ͔Βҳ୤͢Δ΋ͷΛൃݟ͢Δ Example: • Which IP address is not like the others? (indication of a bot / attacker)

  7. ك༗ͳඇߏ଄ϝοηʔδͷมԽ Unusual or rare events (via log categorization) 19 •

    ྨࣅੑʹج͍ͮͯΧςΰϦ෼͚ • ࣌ؒมԽʹΑΔස౓Λֶश • ϞσϧͱҟͳΕ͹ҟৗͱͯ͠ݕ஌ Example: • Do my application logs contain unusual messages
  8. σϓϩΠϝϯτϞσϧ 22 Cluster Data node Apps Master node Data node

    Data node Master node Master node Data node Data node ES clients, Kibana, Logstash, Beats, User apps and etc. ML node ML node # config/elasticsearch.yml xpack.ml.enabled: true node.ml: true
  9. Demo Single/Multiple Metrics: New York City Yellow Taxi Outliers in

    Population: Web Server Log Rare Messages: DBMS Server Log 25
  10. ࣍ͷεςοϓ 33 • Elastic StackΛ·ͩར༻͍ͯ͠ͳ͍ • ϋϯζΦϯϫʔΫγϣοϓ • Elastic StackɺX-PackΛΠϯετʔϧ

    • αϯϓϧσʔλΛར༻ (ϒϩάࢀর) or ࣗ਎ͷσʔλΛ౤ೖ • MLδϣϒΛ࡞੒ • Elastic StackΛར༻த • X-PackΛΠϯετʔϧ (30೔ؒͷτϥΠΞϧ/ඇϓϩμΫγϣϯ؀ڥ) • MLδϣϒΛ࡞੒ (Ϩγϐ΋׆༻) • AlertingͰΞΫγϣϯ