Elasticsearch: you know, for more than search

Talk from NDC Sydney on capabilities in the Elastic stack that can be used for a variety of use cases, including Recommendations, Ad-hoc analysis, Reactive monitoring and Anomaly Detection

Transcript

1. 1 Russ Cam Software Engineer at Elastic @forloop Elasticsearch: you know, for more than search
2. 2 The Elastic Stack Elastic Cloud Security X-Pack Kibana User Interface Elasticsearch Store, Index, & Analyze Ingest Logstash Beats + Alerting Monitoring Graph ML
3. 3 3 MANY PROBLEMS CAN BE VIEWED AS SEARCH PROBLEMS
4. 4 Recommendations
5. 5 Etiology of Disease Outbreaks
6. 6 Security Analytics
7. 7 Anomalous Behaviour
8. 8 Fraud Detection
9. 9 Recommendations Inferring relationships from co-occurrence
10. 10 Co-occurrence Users who like the movie "Pi" also like the movie "Requiem for a Dream"
11. 11 Documents in Elasticsearch { "id": 1, "likes": ["Pi", "Dark City", "Requiem For a Dream"] } { "id": 2, "likes": ["Pi", "Mullholland Drive"] } { "id": 3, "likes": ["Pi", "The Shawshank Redemption", "Fight Club"] } { "id": 4, "likes": ["Pi", "Pulp Fiction"] }
12. 12 Graphing Relationships Wisdom of Crowds Pi Requiem for A Dream Dark City Mullholland Drive The Shawshank Redemption Fight Club Pulp Fiction
13. 13 13 DEMO
14. 14 Graphing Relationships Super Connected Entities Pi Requiem for A Dream Dark City Mullholland Drive The Shawshank Redemption Fight Club Pulp Fiction
15. 15 Graphing Relationships Super Connected Entities Data Document type Vertices Super connected Entities Twitter Tweet Hashtags #YOLO, #MAGA Movies User Favourite Movie The Shawshank Redemption, Fight Club Music User Favourite Band The Beatles, Coldplay, Radiohead Wikipedia Article Linked article United States, Living people Phone Statements Call Phone number Taxi firms, Centrelink Bank Statements Transaction Paid to account Amazon, Paypal, Energy Company
16. 16 Frequency distributions Zipf's Law
17. 17 How Graph Databases typically work • Specify connections between vertices • Apply a weight to each connection • Calculate a threshold for connections • Use threshold to determine interesting connections Limiting the effect of super connected entities
18. 18 How Elasticsearch Graph API works Combining graph algorithms with search Calculate term frequencies when indexing documents 1 2 3 4 Use terms co- occurrence across documents to infer relationships Calculate statistical significance of inferred relationships Only show significant (interesting) relationships
19. 19 Significant Terms Aggregation Interesting or unusual occurrences of terms in a set All user's favourite movies Users who favourite movies include the movie "Pi" Requiem for a Dream Requiem for a Dream
20. 20 Ad-hoc forensic analysis Graph exploration to determine root cause
21. 21 Etiology of events Using graph for ad-hoc exploration User A Port 22 10.0.0.4 10.1.1.53 Port 80 User B Virus
22. 22 Finding Patient Zero • Every connection could be potentially significant • Certainty in implied relationships • Traverse relationships and understand linkages Identifying the source
23. 23 23 DEMO
24. 24 Graph Recap REAL TIME & SCALABLE EXISTING DATA MODEL RELEVANCE BASED TRAVERSAL
25. 25 Reactive Monitoring Taking action when events happen
26. 26 Typical search queries Query in, documents out Cluster Node 1 Node 2 Node 3 Query JSON Matching Documents
27. 27 Percolate queries "Reverse search" Cluster Node 1 Node 2 Node 3 JSON Document Matching Queries
28. 28 Percolate use cases Simple Alerting Query Targeting Feedback Classification • Price Monitoring • News Alerts • Stock price Alerts • Weather Alerts • Advertisements • Marketplaces • Automatic tagging • Geo tagging • Language Detection
29. 29 Alerting • Temporal patterns are important • Avoiding alert fatigue [Action] when [input] is [condition]. Check it [trigger]. Context is key Cluster Node 1 Node 2 Node 3 Alerting Trigger & Input Condition & Action HTTP INDEX LOG
30. 30 30 DEMO
31. 31 Anomaly Detection Unsupervised Machine Learning
32. 32 Anomaly detection • Sometimes, rule based alerts are insufficient • Defining normal static thresholds is hard For time series data ? ?
33. 33 Anomaly detection • What is normal / abnormal? • Learning what is normal / abnormal For time series data Probability E-mails per day
34. 34 Types of anomalies Univariate and multivariate time series Unusual traffic on website Unusual traffic on website by country China France UK USA
35. 35 Types of anomalies Population analysis for outlier detection
36. 36 Types of anomalies Unusual or rare events via log categorization
37. 37 37 DEMO
38. 38 Summary
39. 39 Thanks! Come find me at the Elastic booth Ping me on twitter @forloop