FutureStack 15: New Relic at Scale

In a typical day, New Relic receives half a trillion data points from our customers. All of these need to be processed, stored and made available for charts, queries and alerts as soon as possible. Getting from humble roots to querying more than a billion events a second has meant radically reinventing and re-architecting the whole platform multiple times.

In this session, Matthew Flaming and Nic Benders from New Relic will take you on a deep dive into some of the lessons we've learned along the way and where we go from here. They'll touch on designing scalable services, optimizing for predictability, creating software architecture through team structure, and what it takes to (continuously) rebuild a system that can never stop.

Transcript

1. Confidential ©2008-15 New Relic, Inc. All rights reserved. Building a System that Never Stops: New Relic at Scale 1 Matthew Flaming Nic Benders
2. Confidential ©2008-15 New Relic, Inc. All rights reserved.
4. EVERY MINUTE requests accepts
 over 16M stores
 over analytic
 events 2M aggregates
 over 600M metrics 3B queries
 over data
 points ▪
5. ▪
6. different
 services contains
 over 200 maintained/
 built by 25+ engineering
 teams ▪ 2.5 more
 than SSD
 storage PETABYTES
7. 1. NOTHING Lasts Forever 2. Run EXPERIMENTS 3. SYNCHRONOUS CALLS are going to be a problem 4. Master the ROLLOUT 5. NEW TECH = NEW CHALLENGES 6. Use the right WORKLOAD DISTRIBUTION 7. Technology enables CULTURE 8. Software architecture: THE BIG PICTURE The New Relic lesson plan
8. Collector New Relic App Agents
9. Confidential ©2008-15 New Relic, Inc. All rights reserved. ▪ Double-click to edit
10. Collector New Relic App Agents Browser R.U.M.
 Beacon New Relic Browser Agents Mobile Proxy New Relic Mobile Agents
11. New Relic App Agents New Relic Browser Agents New Relic Mobile Agents Zzz Collector Browser R.U.M.
 Beacon Mobile Proxy
12. Confidential ©2008-15 New Relic, Inc. All rights reserved.
13. NOTHING Lasts Forever Lesson 1
14. Run EXPERIMENTS Lesson 2
15. Confidential ©2008-15 New Relic, Inc. All rights reserved.
16. Confidential ©2008-15 New Relic, Inc. All rights reserved.
17. EXPERIMENTS Don’t always work Lesson 2(a)
18. SYNCHRONOUS CALLS are going to be a problem Lesson 3
19. ▪ Kafka Kafka Browser Data Consumer Browser Data Router New Relic App Agents New Relic Browser Agents Collector
20. ▪ Kafka Kafka Browser Data Consumer New Relic App Agents Collector Zzz Browser Data Router New Relic Browser Agents
21. Confidential ©2008-15 New Relic, Inc. All rights reserved.
22. Data Routers Kafka Topics Data Consumers Data Stores/DBs Retrieval Services Ingest Queue Process/Aggregate Store Query
23. Data Routers Kafka Topics Data Consumers Data Stores/DBs Retrieval Services
24. Master the ROLLOUT Lesson 4
25. Rollout
 Techniques Kafka Load Balancer Cluster Kafka Browser Data Consumer Browser Data Router Browser Data
 Consumer V2 Browser R.U.M.
 Beacon %- %+ New Relic Browser Agents Collector Old New
26. NEW TECH NEW CHALLENGES Lesson 5 =
27. Monitoring Tooling Operations Runbooks
28. Confidential ©2008-15 New Relic, Inc. All rights reserved.
29. Lesson 6 Use the right WORKLOAD DISTRIBUTION
30. ©2008-15 New Relic, Inc. All rights reserved. ▪ Workload Distribution A C B D D D D D B C B B B B D D A A A A A D D Active Management Random C B B B D D A A D C B B D B C A A D D C C A B
31. Technology enables CULTURE Lesson 7
32. Thanks for the pic! https://www.flickr.com/photos/stephenyeargin/7466608166
33. Thanks for the pic! https://www.flickr.com/photos/stephenyeargin/7466608166 �����
34. Software Architecture: THE BIG PICTURE Lesson 8
35. Software Architecture: The Big Picture ▪ ▪ ▪ ▪ ▪ Organic growth Paradigm shift Breaking point
36. New Paradigm Collector New Relic App Agents
37. Organic Growth Collector New Relic App Agents Browser R.U.M.
 Beacon New Relic Browser Agents Mobile Proxy New Relic Mobile Agents
38. Breaking Point Collector Browser R.U.M.
 Beacon Mobile Proxy Zzz New Relic App Agents New Relic Browser Agents New Relic Mobile Agents
39. ▪ Kafka Kafka Browser Data Consumer Browser Data Router New Relic App Agents New Relic Browser Agents Collector
40. Confidential ©2008-15 New Relic, Inc. All rights reserved. ▪ Double-click to edit
41. Confidential ©2008-15 New Relic, Inc. All rights reserved. ▪ Double-click to edit
43. Confidential ©2008-15 New Relic, Inc. All rights reserved. 43 Thank you Matthew Flaming Nic Benders