Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[Developers Summit 2017] Anomaly Detection with the Elastic Stack

Kosho Owa
February 16, 2017

[Developers Summit 2017] Anomaly Detection with the Elastic Stack

2017/02/16 11:05 -
Developers Summit 2017「Elastic Stackを利用した異常検知」セッション資料

Kosho Owa

February 16, 2017
Tweet

More Decks by Kosho Owa

Other Decks in Technology

Transcript

  1. Prelert:
    Elastic StackΛར༻ͨ͠ҟৗݕ஌
    February 16th, Developers Summit 2017
    େྠ ߂ৄ | Kosho Owa
    Solutions Architect, Elastic

    View Slide

  2. 2
    Elastic Stack
    100% open source
    No enterprise edition
    All new versions with 5.0

    View Slide

  3. 3
    Elasticsearch
    Heart of the Elastic Stack
    Distributed, Scalable High-availability Multi-tenancy
    Developer Friendly Real-time, Full-text Search Aggregations

    View Slide

  4. 4
    Kibana
    Window into the Elastic Stack
    Visualize and analyze Geospatial Customize and Share
    Reports
    Graph Exploration UX to secure and manage
    the Elastic Stack
    Build Custom Apps

    View Slide

  5. 5

    View Slide

  6. 6

    View Slide

  7. 7
    Beats
    Lightweight data shippers
    Ship data from the source
    Ship and centralize in
    Elasticsearch
    Ship to Logstash for
    transformation and parsing
    Ship to Elastic Cloud Libbeat: API framework to
    build custom beats 30+ community Beats

    View Slide

  8. 8
    FILEBEAT
    Log Files
    METRICBEAT
    Metrics
    PACKETBEAT
    Network Data
    WINGLOGBEAT
    Window Events
    More than 30 community Beats
    and growing …
    Apachebeat, dockbeat, httpbeat,
    mysqlbeat, nginxbeat, redis beats,
    twitterbeat, and more

    View Slide

  9. 9
    Logstash
    Data processing pipeline
    Ingest data of all shapes,
    sizes, and sources
    Parse and dynamically
    transform data
    Transport data to any
    output
    Secure and encrypt data
    inputs
    Build your own pipeline More than 200+ plugins

    View Slide

  10. 10
    X-Pack
    Extensions for the Elastic Stack
    Security Alerting Monitoring
    Reporting Graph Analytics Single Install, included in
    Elastic Subscription

    View Slide

  11. 11
    X-Pack
    Security
    • Username and password
    • Integrate with authentication systems
    • Create a custom realm to authenticate users
    AUTHENTICATION
    • Manage users and roles
    • Assign permissions and privileges
    AUTHORITIZATION
    • SSL/TLS encryption
    • IP filtering
    • Field and document level security
    • Audit logging
    ADDITIONAL CONTROLS

    View Slide

  12. 12
    X-Pack
    Alerting
    • Create Watches to detect changes in
    your data
    • Trigger automatic notifications
    • Setup nested alerts
    • Store and track alert history
    SETUP ALERTS
    NOTIFY AND INTEGRATE
    • Email
    • Slack
    • Pagerduty
    • Hipchat or JIRA
    • Other monitoring systems

    View Slide

  13. 13
    X-Pack
    Monitoring
    • Prebuilt Kibana dashboards to monitor
    the performance of the Elastic Stack
    • Get vital statistics at various levels --
    cluster, node, and indices
    MONITOR CLUSTER HEALTH
    OPTIMIZE CLUSTER PERFORMANCE
    • Multicluster support to compare health and
    performance of multiple clusters
    • Analyze historical or real-time data for root
    cause analyses
    • Utilize analyses to proactively optimize and
    improve cluster performance
    • Configure data retention policy

    View Slide

  14. 14
    X-Pack
    Reporting
    • Email recurring status updates daily, weekly,
    monthly, etc.
    • Combine reporting with X-Pack alerting
    capabilities to trigger conditional reports
    AUTOMATE SCHEDULING
    SHARE AND COLLABORATE
    • Export any Kibana visualization or
    dashboard
    • Print-optimized and PDF formatted
    • Download and share past reports

    View Slide

  15. 15
    X-Pack
    Graph
    • Uses relevance capabilities of Elasticsearch
    • Discover linkages and connections
    • Leverage API and UI-drive tool
    A NEW WAY TO EXPLORE DATA
    EXTEND TO NEW USE CASES
    • Fraud discovery
    • Recommendations
    • Cyber security
    • Behavioral analyses

    View Slide

  16. 16

    View Slide

  17. σʔλ͔Β༗ҙٛͳ৘ใΛݟ͚ͭΔํ๏
    17
    Search Aggregations Visualization
    Prelert
    Machine Learning

    View Slide

  18. 18
    Prelert
    Behavioral analytics and
    unsupervised machine learning
    • Automatically detect anomalies
    • Advanced correlation and categorization
    • Identify root cause(s)
    • Expose early warning signs
    UNSUPERVISED MACHINE LEARNING
    ENABLE NEW USE CASES
    • Analyze time series data
    • Expand security, IT Ops, fraud, finance, and
    many more use cases
    • Currently beta; building a more native
    integration into the Elastic Stack

    View Slide

  19. *5ΦϖϨʔγϣϯ
    • ࣗ෼ͷγεςϜ͸ਖ਼ৗʹՔಇ͍ͯ͠Δ?
    • ͲͷΑ͏ʹᮢ஋Λ൑அ͢Δ?
    • ໰୊͕ൃੜͨ࣌͠ʹɺͲͷΑ͏ʹݪҼΛݟ͚ͭΔ?
    19

    View Slide

  20. *5ηΩϡϦςΟ
    • Ϛϧ΢ΣΞʹ৵ೖ͞Ε͍ͯΔγεςϜ͸ແ͍͔?
    • Ϛϧ΢ΣΞ͕ͲͷΑ͏ʹײછΛ޿͔͛ͨ?
    • જࡏతʹڴҖͱͳΔ૊৫಺෦ͷϢʔβʔ͸୭͔?
    20

    View Slide

  21. ͦͷଞ
    • ͲͷΑ͏ʹɺଟ͘ͷछྨͷ࣌ܥྻσʔλͱ޲͖߹͏͔?
    • ޻৔͸ਖ਼ৗʹՔಇ͍ͯ͠Δ?
    • Ͳͷަ௨ࣄނ͕࠷΋ौ଺ΛҾ͖ى͍ͯ͜͠Δ͔?
    21

    View Slide

  22. Demo
    IT Operation, Security

    View Slide

  23. View Slide

  24. View Slide

  25. View Slide

  26. View Slide

  27. View Slide

  28. View Slide

  29. View Slide

  30. View Slide

  31. View Slide

  32. View Slide

  33. &MBTUJDͷ.BDIJOF-FBSOJOHςΫϊϩδʔ
    33
    σʔλʹજΉߦಈϞσϧΛ

    ࣗಈతʹڭࢣͳֶ͠श
    ݱࡏͷߦಈ͕༧ଌϞσϧͱ

    ݦஶʹҟͳΔ৔߹ʹ௨஌

    View Slide

  34. ϩʔυϚοϓ
    • ϕʔλ൛Λఏڙத (prelert.com)
    • Elastic StackͱͷΠϯςάϨʔγϣϯ͕ਐߦத
    • 2017೥্൒ظͷϦϦʔεΛ໨ඪ
    34

    View Slide