Introducing Machine Learning for the Elastic Stack

Slide 1

Slide 1 text

Machine Learning for the Elastic Stack   Beta in 5.4. GA coming soon May 2017 େྠ ߂ৄ | Kosho Owa Solutions Architect, Elastic

Slide 2

Slide 2 text

2 Elastic Stack 100% Φʔϓϯιʔε ʮΤϯλʔϓϥΠζ൛ʯ͸ແ͠ όʔδϣϯ 5.0Ͱ׬શ౷Ұ

Slide 3

Slide 3 text

3 X-Pack ؆୯ʹΠϯετʔϧ Elastic StackΛ֦ு αϒεΫϦϓγϣϯʹؚ·ΕΔ Security Alerting Monitoring Reporting Graph Machine Learning

Slide 4

Slide 4 text

4 Elastic Cloud Elasticsearch, Kibanaͷ ϚωʔδυαʔϏε X-Packͷػೳ΋ར༻Մೳ Available in AWS today

Slide 5

Slide 5 text

5 Elastic Cloud Enterprise ෳ਺ͷElastic Stack؀ڥΛࣗࡏʹ࡞੒ Logging as a serviceΛࣗ૊৫ʹల։ Public beta; Expected GA Q1 2017

Slide 6

Slide 6 text

ҟৗͷൃݟ͕τϥϒϧͷஹީΛࣔ͢ 6 Spiked 404 errors Web attack IT Operational Analytics Security Analytics Business Analytics Unusual DNS activity Data exfiltration Rare log messages Failing sensor

Slide 7

Slide 7 text

Operational Analytics • ΢ΣϒαΠτ΁ͷΞΫηετϥϑΟοΫʹҟৗ͸ແ͍͔? • Ϙοτ΍߈ܸऀ͕๚Ε͍ͯͳ͍͔? • σʔλϕʔε͕ग़ྗ͍ͯ͠ΔErrorϩά͸ରॲ͢Δඞཁ͕ ͋Δͷ͔? Use Case

Slide 8

Slide 8 text

Security Analytics • Ϛϧ΢ΣΞʹ৵ೖ͞Ε͍ͯͳ͍͔? • ಺෦ऀʹΑΔηΩϡϦςΟڴҖ͸ແ͍͔? • DNSͷϩάʹ͸ɺσʔλ࠮औͷࠟ੻͕ͳ͍͔? Use Case

Slide 9

Slide 9 text

Telemetry / Sensors ▪ ISPͷωοτϫʔΫҰ࣌ःஅʹΑΔϨΠςϯγʔͷٸ ܹͳ૿Ճ͸? ▪ ଞͱ͸ҟͳΔӡసύλʔϯΛͱΔυϥΠόʔ͸? ▪ ಛҟͳΠϕϯτλΠϓ͸ηϯαʔͷނোΛ͔ࣔ͢? Use Case

Slide 10

Slide 10 text

10 ҟৗͷൃݟ͸ࢥͬͨΑΓ΋೉͍͠ • σʔλ͸ෳࡶɺߴ࣍ݩɺߴ଎ʹมԽ • ਓؒͷࢹೝ͸ݱ࣮తʹෆՄೳ • ༰қʹݟಀ͢ Visual inspection is not practical Where’s the anomaly?

Slide 11

Slide 11 text

11 ҟৗͷൃݟ͸ࢥͬͨΑΓ΋೉͍͠ • ੩తͳᮢ஋ʹΑΔʮਖ਼ৗʯͷఆٛ͸ࠔ೉ • ϧʔϧ͸σʔλ͸Πϯϑϥͷมߋʹ௥ैͰ͖ͳ͍ • ༰қʹᷖճ͞Εͯ͠·͏ Rule-based alerts are insufficient What’s the right threshold ?

Slide 12

Slide 12 text

X-Pack͕ࣗಈతͳҟৗݕ஌Ͱղܾ 12 • ʮڭࢣͳ͠ʯػցֶशςΫχοΫʹΑΓ ▪ աڈͷσʔλ͔Βʮਖ਼ৗʯΛֶͼϞσϧΛ࡞Δ ▪ ਖ਼ৗ஋ͷൣғ͔Βҳ୤ͨ͠ࡍʹҟৗͱͯ͠ݕ஌

Slide 13

Slide 13 text

X-Pack͕ࣗಈతͳҟৗݕ஌Ͱղܾ 13 • ڭࢣͳ͠ - खಈͰͷਖ਼ৗ஋ͷೖྗ͕ෆཁ • σʔλͷมԽʹ௥ै - ౤ೖ͞ΕΔσʔλʹΑΓܧଓతʹϞσϧΛߋ৽ • ӨڹҼࢠಛఆ - ࠜຊݪҼղੳΛՃ଎

Slide 14

Slide 14 text

ҟͳΔछྨͷҟৗΛݕ஌ 14 • ࣌ܥྻͷϝτϦοΫ Time series - single / multiple • ͸͙Εऀ Outliers in population (using entity profiling) • ك༗ͳඇߏ଄ϝοηʔδ Rare / unusual rates in “categories” of events

Slide 15

Slide 15 text

࣌ܥྻσʔλͷҟৗ 15 Time Metric • Single (univariate) time series Example: Is there unusual traffic on website ?

Slide 16

Slide 16 text

࣌ܥྻσʔλͷҟৗ 16 Time Metric USA UK France Japan • Multiple time series ▪ ෳ਺ͷϝτϦοΫ ▪ FieldʹΑͬͯ෼ྨ͞ΕͨϝτϦοΫ • ͦΕͧΕ͕ಠཱͯ͠ଘࡏ͢Δ Example: Is there unusual web activity from any country?

Slide 17

Slide 17 text

͸͙Εऀ Outliers in population (using entity profiling) 17 • ूஂͷಛ௃(server, user, IPͳͲ)͔ΒϓϩϑΝΠϧΛ࡞੒͢Δ • ͜ͷूஂ͔Βҳ୤͢Δ΋ͷΛൃݟ͢Δ Example: • Which IP address is not like the others? (indication of a bot / attacker) 

Slide 18

Slide 18 text

͸͙Εऀ Outliers in population (using entity profiling) 18 • ूஂͷಛ௃(server, user, IPͳͲ)͔ΒϓϩϑΝΠϧΛ࡞੒͢Δ • ͜ͷूஂ͔Βҳ୤͢Δ΋ͷΛൃݟ͢Δ Example: • Which IP address is not like the others? (indication of a bot / attacker) 

Slide 19

Slide 19 text

ك༗ͳඇߏ଄ϝοηʔδͷมԽ Unusual or rare events (via log categorization) 19 • ྨࣅੑʹج͍ͮͯΧςΰϦ෼͚ • ࣌ؒมԽʹΑΔස౓Λֶश • ϞσϧͱҟͳΕ͹ҟৗͱͯ͠ݕ஌ Example: • Do my application logs contain unusual messages

Slide 20

Slide 20 text

X-Pack Machine Learning Elastic StackͱͷڧݻͳΠϯςάϨʔγϣϯ 20

Slide 21

Slide 21 text

• Elasticsearch • Kibana ༰қʹΠϯετʔϧ 21 $ elasticsearch-plugin install x-pack $ kibana-plugin install x-pack

Slide 22

Slide 22 text

σϓϩΠϝϯτϞσϧ 22 Cluster Data node Apps Master node Data node Data node Master node Master node Data node Data node ES clients, Kibana, Logstash, Beats, User apps and etc. ML node ML node # config/elasticsearch.yml xpack.ml.enabled: true node.ml: true

Slide 23

Slide 23 text

֎෦γεςϜͱͷ઀ଓ • API (anomaly_detectors, datafeeds, results, model_snapshots, validate) • ΠϯσοΫε (.ml-anomalies-*)

Slide 24

Slide 24 text

Taking Action with X-Pack Alerting 24

Slide 25

Slide 25 text

Demo Single/Multiple Metrics: New York City Yellow Taxi Outliers in Population: Web Server Log Rare Messages: DBMS Server Log 25

Slide 26

Slide 26 text

26 4JOHMF.FUSJD

Slide 27

Slide 27 text

27 .VMUJ.FUSJD

Slide 28

Slide 28 text

28 .VMUJ.FUSJD

Slide 29

Slide 29 text

29 0VUMJFSTJO1PQVMBUJPO

Slide 30

Slide 30 text

30 0VUMJFSTJO1PQVMBUJPO

Slide 31

Slide 31 text

31 3BSF.FTTBHFT

Slide 32

Slide 32 text

32 3BSF.FTTBHFT

Slide 33

Slide 33 text

࣍ͷεςοϓ 33 • Elastic StackΛ·ͩར༻͍ͯ͠ͳ͍ • ϋϯζΦϯϫʔΫγϣοϓ • Elastic StackɺX-PackΛΠϯετʔϧ • αϯϓϧσʔλΛར༻ (ϒϩάࢀর) or ࣗ਎ͷσʔλΛ౤ೖ • MLδϣϒΛ࡞੒ • Elastic StackΛར༻த • X-PackΛΠϯετʔϧ (30೔ؒͷτϥΠΞϧ/ඇϓϩμΫγϣϯ؀ڥ) • MLδϣϒΛ࡞੒ (Ϩγϐ΋׆༻) • AlertingͰΞΫγϣϯ