Elastic at Datadog

Datadog is a SaaS-based infrastructure monitoring company that processes billions of data points every day, including metrics (CPU utilization, database keys, and queue lengths) and events (completed Chef job notifications, GitHub commits, and Docker container status). Storing this information and being able to make use of it in their Stream and Dashboards is challenging. They started with Postgres, but as their needs grew, they moved to Elasticsearch, which is now a core component of their infrastructure, indexing vast numbers of events every second.

In this talk, you will see how Datadog uses an Elasticsearch cluster to create a fast and efficient environment for thousands of customers.

Isaac Sadaqah l Software Engineer l Datadog

Transcript

1. Datadog 03/09/2017 isaac@datadoghq.com Elasticsearch at Datadog Isaac Sadaqah, Software Engineer
2. • SaaS based infrastructure and application monitoring • Built for dynamic infrastructure and applications • 150+ integrations • Process trillions of metrics per day • Process millions of events per day • APM This is a sample image Datadog
3. Agenda 1 Use case: Events 2 Use case: Alerting management 3 How do we monitor elasticsearch
4. 4 Events
5. Examples
6. Examples
7. Examples
8. Examples
9. 9 Examples
10. 10 Events
11. 11 Events
12. 12 Events
13. 13 Events
14. 14 Events sources:nagios,chef status:error redis_* AND down
15. 15 Events
16. 16 Events
17. 17 Event Alerts
18. 18 Event Alerts
19. 19 Event Alerts
20. 20 Event Alerts
21. 21 Event Alerts
22. 22 Event Alerts
23. • First implementation used Postgres • Then we moved to ES • Evolved from a small 3 node cluster to a large cluster with dedicated master/data/client nodes • 1 index per month (additional indices are created semi-automatically if needed) • 40 shards and 2 replicas • 1bn documents per index Events Cluster
24. Architecture Events Processing users Web, Agent, API, Email Elasticsearch Master Nodes Client Nodes Data Nodes Alerting Web App
25. Architecture Events Processing users Web, Agent, API, Email Elasticsearch Master Nodes Client Nodes Data Nodes Alerting Web App
26. Architecture Events Processing users Web, Agent, API, Email Elasticsearch Master Nodes Client Nodes Data Nodes Alerting Web App
27. Architecture Events Processing users Web, Agent, API, Email Elasticsearch Master Nodes Client Nodes Data Nodes Alerting Web App
28. Architecture Events Processing users Web, Agent, API, Email Elasticsearch Master Nodes Client Nodes Data Nodes Alerting Web App
29. Architecture Events Processing users Web, Agent, API, Email Elasticsearch Master Nodes Client Nodes Data Nodes Alerting Web App
30. Architecture Events Processing users Web, Agent, API, Email Elasticsearch Master Nodes Client Nodes Data Nodes Alerting Web App
31. Architecture Events Processing users Web, Agent, API, Email Elasticsearch Master Nodes Client Nodes Data Nodes Alerting Web App
32. 32 Manage Monitors
33. 33 Manage Monitors
34. 34 Manage Monitors
35. 35 Manage Monitors
36. 36 Manage Monitors
37. 37 Manage Monitors
38. • PostgreSQL is used for backend • Loads in <1 second for most customers • Very slow for customers who have several thousand monitors or more • Advanced filtering, grouping and search are too slow to implement Manage Monitors
39. • Search should take under 1 second for all customers, 200ms on average • Data freshness: • under 2 seconds for machine generated data • immediate for user generated action • Consistency New Manage Monitors
40. Architecture Postgres Indexer Elasticsearch Master Nodes Client Nodes Data Nodes Zookeeper Web/API
41. Architecture Postgres Indexer Elasticsearch Master Nodes Client Nodes Data Nodes Zookeeper Web/API
42. Architecture Postgres Indexer Elasticsearch Master Nodes Client Nodes Data Nodes Zookeeper Web/API
43. Architecture Postgres Indexer Elasticsearch Master Nodes Client Nodes Data Nodes Zookeeper Web/API
44. Architecture Postgres Indexer Elasticsearch Master Nodes Client Nodes Data Nodes Zookeeper Web/API
45. Architecture Postgres Indexer Elasticsearch Master Nodes Client Nodes Data Nodes Zookeeper Web/API
46. Architecture Postgres Indexer Elasticsearch Master Nodes Client Nodes Data Nodes Zookeeper Web/API
47. 47 New Manage Monitors
48. How we monitor Elasticsearch? 1 Metrics 2 Events 3 Service checks 4 APM
52. APM
53. APM
54. APM
55. APM
56. APM
57. APM
58. APM
59. APM
60. APM
61. APM
62. Example Elasticsearch Alerts
63. http://dtdg.co/ddelastic
64. We are hiring!
65. Questions?