Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[Developers Summit 2017] Anomaly Detection with the Elastic Stack

34cbde72de5f384380d5489543294dc5?s=47 Kosho Owa
February 16, 2017

[Developers Summit 2017] Anomaly Detection with the Elastic Stack

2017/02/16 11:05 -
Developers Summit 2017「Elastic Stackを利用した異常検知」セッション資料

34cbde72de5f384380d5489543294dc5?s=128

Kosho Owa

February 16, 2017
Tweet

Transcript

  1. Prelert: Elastic StackΛར༻ͨ͠ҟৗݕ஌ February 16th, Developers Summit 2017 େྠ ߂ৄ

    | Kosho Owa Solutions Architect, Elastic
  2. 2 Elastic Stack 100% open source No enterprise edition All

    new versions with 5.0
  3. 3 Elasticsearch Heart of the Elastic Stack Distributed, Scalable High-availability

    Multi-tenancy Developer Friendly Real-time, Full-text Search Aggregations
  4. 4 Kibana Window into the Elastic Stack Visualize and analyze

    Geospatial Customize and Share Reports Graph Exploration UX to secure and manage the Elastic Stack Build Custom Apps
  5. 5

  6. 6

  7. 7 Beats Lightweight data shippers Ship data from the source

    Ship and centralize in Elasticsearch Ship to Logstash for transformation and parsing Ship to Elastic Cloud Libbeat: API framework to build custom beats 30+ community Beats
  8. 8 FILEBEAT Log Files METRICBEAT Metrics PACKETBEAT Network Data WINGLOGBEAT

    Window Events More than 30 community Beats and growing … Apachebeat, dockbeat, httpbeat, mysqlbeat, nginxbeat, redis beats, twitterbeat, and more
  9. 9 Logstash Data processing pipeline Ingest data of all shapes,

    sizes, and sources Parse and dynamically transform data Transport data to any output Secure and encrypt data inputs Build your own pipeline More than 200+ plugins
  10. 10 X-Pack Extensions for the Elastic Stack Security Alerting Monitoring

    Reporting Graph Analytics Single Install, included in Elastic Subscription
  11. 11 X-Pack Security • Username and password • Integrate with

    authentication systems • Create a custom realm to authenticate users AUTHENTICATION • Manage users and roles • Assign permissions and privileges AUTHORITIZATION • SSL/TLS encryption • IP filtering • Field and document level security • Audit logging ADDITIONAL CONTROLS
  12. 12 X-Pack Alerting • Create Watches to detect changes in

    your data • Trigger automatic notifications • Setup nested alerts • Store and track alert history SETUP ALERTS NOTIFY AND INTEGRATE • Email • Slack • Pagerduty • Hipchat or JIRA • Other monitoring systems
  13. 13 X-Pack Monitoring • Prebuilt Kibana dashboards to monitor the

    performance of the Elastic Stack • Get vital statistics at various levels -- cluster, node, and indices MONITOR CLUSTER HEALTH OPTIMIZE CLUSTER PERFORMANCE • Multicluster support to compare health and performance of multiple clusters • Analyze historical or real-time data for root cause analyses • Utilize analyses to proactively optimize and improve cluster performance • Configure data retention policy
  14. 14 X-Pack Reporting • Email recurring status updates daily, weekly,

    monthly, etc. • Combine reporting with X-Pack alerting capabilities to trigger conditional reports AUTOMATE SCHEDULING SHARE AND COLLABORATE • Export any Kibana visualization or dashboard • Print-optimized and PDF formatted • Download and share past reports
  15. 15 X-Pack Graph • Uses relevance capabilities of Elasticsearch •

    Discover linkages and connections • Leverage API and UI-drive tool A NEW WAY TO EXPLORE DATA EXTEND TO NEW USE CASES • Fraud discovery • Recommendations • Cyber security • Behavioral analyses
  16. 16

  17. σʔλ͔Β༗ҙٛͳ৘ใΛݟ͚ͭΔํ๏ 17 Search Aggregations Visualization Prelert Machine Learning

  18. 18 Prelert Behavioral analytics and unsupervised machine learning • Automatically

    detect anomalies • Advanced correlation and categorization • Identify root cause(s) • Expose early warning signs UNSUPERVISED MACHINE LEARNING ENABLE NEW USE CASES • Analyze time series data • Expand security, IT Ops, fraud, finance, and many more use cases • Currently beta; building a more native integration into the Elastic Stack
  19. *5ΦϖϨʔγϣϯ • ࣗ෼ͷγεςϜ͸ਖ਼ৗʹՔಇ͍ͯ͠Δ? • ͲͷΑ͏ʹᮢ஋Λ൑அ͢Δ? • ໰୊͕ൃੜͨ࣌͠ʹɺͲͷΑ͏ʹݪҼΛݟ͚ͭΔ? 19

  20. *5ηΩϡϦςΟ • Ϛϧ΢ΣΞʹ৵ೖ͞Ε͍ͯΔγεςϜ͸ແ͍͔? • Ϛϧ΢ΣΞ͕ͲͷΑ͏ʹײછΛ޿͔͛ͨ? • જࡏతʹڴҖͱͳΔ૊৫಺෦ͷϢʔβʔ͸୭͔? 20

  21. ͦͷଞ • ͲͷΑ͏ʹɺଟ͘ͷछྨͷ࣌ܥྻσʔλͱ޲͖߹͏͔? • ޻৔͸ਖ਼ৗʹՔಇ͍ͯ͠Δ? • Ͳͷަ௨ࣄނ͕࠷΋ौ଺ΛҾ͖ى͍ͯ͜͠Δ͔? 21

  22. Demo IT Operation, Security

  23. None
  24. None
  25. None
  26. None
  27. None
  28. None
  29. None
  30. None
  31. None
  32. None
  33. &MBTUJDͷ.BDIJOF-FBSOJOHςΫϊϩδʔ 33 σʔλʹજΉߦಈϞσϧΛ
 ࣗಈతʹڭࢣͳֶ͠श ݱࡏͷߦಈ͕༧ଌϞσϧͱ
 ݦஶʹҟͳΔ৔߹ʹ௨஌

  34. ϩʔυϚοϓ • ϕʔλ൛Λఏڙத (prelert.com) • Elastic StackͱͷΠϯςάϨʔγϣϯ͕ਐߦத • 2017೥্൒ظͷϦϦʔεΛ໨ඪ 34