Using Elasticsearch to Power Analytics at AppDynamics

Learn how AppDynamics utilizes Elasticsearch to power the three core uses of AppDynamics Analytics – advanced performance analytics, business impact analytics, and user analytics – as well as lessons learned along the way in deploying and managing large Elasticsearch clusters.

1. 1 Arjun Iyer, Senior Engineering Director, AppDynamics 2/17/2015 Using Elasticsearch to power Analytics across Billions of Transactions every day at AppDynamics
2. 2 About Me •  Lead the “Analytics” BI @ AppDynamics as Senior Engineering Director. •  Manage a team of Big Data engineers and Data Scientists. •  Passionate about Distributed Systems and Machine Learning.
3. 3 Agenda •  Brief Overview of AppDynamics •  Why Elasticsearch ? •  Data/Index Model •  RCA using Significant Terms •  Scaling Elasticsearch •  Monitoring and Index Management •  Future Direction
4. 4 Brief Overview of AppDynamics
5. 5 To watch every line of code and empower organizations to innovate, compete and Win in the Digital age Company Vision
6. 6 Application Intelligence from AppDynamics Market leadership 100% growth 2014 Net promoter score 1,800+ customers Our Platform 8 Smart services On-Premises, SaaS, or Cloud Rapid time to value Low cost of ownership Enterprise adoption Scalable – 18/20 largest APM deployments Website download 85
7. 7 Copyright © 2014 AppDynamics. All rights reserved. 7 Application complexity is exploding SOA NOSQL Cloud Agile Micro-services IoT ESB/MQ WEBLOGIC SERVICE MONGODB TOMCAT SERVICE Login Flight Status Search Flight Purchase Web Mobile NETWORK ORACLE APACHE JBOSS SERVICE ESB/MQ NETWORK WEBLOGIC SERVICE MONGODB ORACLE TOMCAT SERVICE JBOSS SERVICE
8. 8 What if you knew in real-time? Top Product Categories Customers by Tier Average Response Time $232,390 Revenue impact of poor performance Top products generating highest revenue Revenue by cities Most of customers experiencing issues are Platinum Total Revenues Top Cities Performance timestamp indicating trend towards problem
9. 9 Why Elasticsearch ?
10. 10 Analytics Requirements Dynamic Schemas Near real-time Aggregations Free Text Search Horizontal Scalability
11. 11 Data/Index Model
12. 12 Schemas Mobile Browser Biz Txn Logs Custom •  Free text fields •  Dynamic fields •  Nested structures
13. 13 Index Model •  Always use doc values ! •  Index per account per type * (roll indices by size) ‒  We chose this finally to get the granularity & isolation we need ‒  Need to be more careful about cluster state as the #indices increase •  One index (fields for account and type) ‒  Runs into the “sparse” index problem ‒  Purging is much harder ‒  Not much isolation (all eggs in 1 basket/index!) •  Index per type (field for account) ‒  Again isolation is lower and also suffers from the purge problem
14. 14 Mapping – Storage Requirements _source   storage   LZ4 / DEFLATE analyzed   fields   Reverse   Index   Terms dictionary compression non_analyzed   fields   Doc_values (columnar)   Doc values compression
15. 15 RCA using Significant Terms
16. 16 Significant Terms •  An aggregation that returns interesting or unusual occurrences of terms in a set. •  These are the terms that have undergone a significant change in popularity measured between a foreground and background set.
17. 17 NOTE: USE THIS LAYOUT FOR PLACING ONE FULL BLEED SCREENSHOT
18. 18 Our enhancements •  Dynamic baseline ‒  Default is the whole index ‒  We calculate the baseline based on given query and it’s time range •  Score Normalization ‒  We normalize the scores [0-100] using logistic function
19. 19 Scaling Elasticsearch
20. 20 Calls/day 150M 2B 1 Statistics Events/day TB/day
21. 21 Multi-Cluster Model Cluster  Router  
22. 22 Clusters 12 5-20 6K Our ES Footprint Nodes/Cluster Indices
23. 23 Core Architecture
24. 24 AWS – EC2 instance types •  EBS backed instances are another option but needs further testing
25. 25 Things to keep in mind •  Watch out for cluster state. Keep minimum #shards to achieve performance target. •  Choose #replicas carefully. For high write throughput and lower costs, replica=1 works best. •  Keep shard size within HEAP. •  Use tiered storage for lower costs ( https://www.elastic.co/blog/hot-warm-architecture) •  Optimize settings for SSDs (E.g disable merge throttling etc)
26. 26 Monitoring and Cluster Management
27. 27 Flowmap view using AppDynamics
28. 28 IOPS on Elasticsearch data nodes
29. 29 Monitoring E.g Instrumented the org.elasticsearch.search.query.QueryPhase.execute method
30. 30 Index Management •  Roll Indices by size or time ‒  Allows you to change mapping and/or #shards ‒  Easy to purge older data (index deletion is fast!) •  Purge older data by deleting indices (fast) ‒  Deleting data using scan/scroll is slow •  Aliases are your friend! Use them liberally J ‒  1 write index ‒  Multiple read indices •  Automate Backups using snapshots
31. 31 Future Direction
32. 32 We’re interested in… •  Query streaming •  Job management (3.0?) –  https://github.com/elastic/elasticsearch/issues/15117 •  Leveraging more statistical aggregations like moving average, Holt- Winters etc. •  Potential use of Mesos to manage ES clusters
33. 33 We’re Hiring! https://www.appdynamics.com/company/careers/ Thank You!