Scaling a System (that never stops)

In a typical day, New Relic receives half a trillion data points from our customers. All of these need to be processed, stored and made available for charts, queried and alerted on in near real-time. Going from humble roots to querying more than a billion events a second has meant radically reinventing and re-architecting the whole platform multiple times, and doing it all in public view.

The evolution of New Relic's service illustrates the opportunities and challenges implementing modern technologies. There are lessons on designing scalable services, optimizing for predictability, creating software architecture through team structure, and what it takes to (continuously) rebuild a system that can never stop.

Transcript

1. Confidential ©2008-15 New Relic, Inc. All rights reserved. ©2008-16 New Relic, Inc. All rights reserved. SCALING A SYSTEM (that never stops) Nic Benders - Chief Architect, New Relic
2. ©2008-16 New Relic, Inc. All rights reserved. @nicbenders http://nicbenders.com/presentations/scaling-a-system/
3. ©2008-16 New Relic, Inc. All rights reserved. Safe Harbor This document and the information herein (including any information that may be incorporated by reference) is provided for informational purposes only and should not be construed as an offer, commitment, promise or obligation on behalf of New Relic, Inc. (“New Relic”) to sell securities or deliver any product, material, code, functionality, or other feature. Any information provided hereby is proprietary to New Relic and may not be replicated or disclosed without New Relic’s express written permission. Such information may contain forward-looking statements within the meaning of federal securities laws. Any statement that is not a historical fact or refers to expectations, projections, future plans, objectives, estimates, goals, or other characterizations of future events is a forward-looking statement. These forward-looking statements can often be identified as such because the context of the statement will include words such as “believes,” “anticipates,” “expects” or words of similar import. Actual results may differ materially from those expressed in these forward-looking statements, which speak only as of the date hereof, and are subject to change at any time without notice. Existing and prospective investors, customers and other third parties transacting business with New Relic are cautioned not to place undue reliance on this forward-looking information. The achievement or success of the matters covered by such forward-looking statements are based on New Relic’s current assumptions, expectations, and beliefs and are subject to substantial risks, uncertainties, assumptions, and changes in circumstances that may cause the actual results, performance, or achievements to differ materially from those expressed or implied in any forward-looking statement. Further information on factors that could affect such forward-looking statements is included in the filings we make with the SEC from time to time. Copies of these documents may be obtained by visiting New Relic’s Investor Relations website at ir.newrelic.com or the SEC’s website at www.sec.gov. New Relic assumes no obligation and does not intend to update these forward-looking statements, except as required by law. New Relic makes no warranties, expressed or implied, in this document or otherwise, with respect to the information provided
5. EVERY MINUTE requests accepts
 over 16M stores
 over analytic
 events 2M aggregates
 over 600M metrics 3B queries
 over data
 points ▪
6. ▪
7. different
 services contains
 over 200 maintained/
 built by 25+ engineering
 teams ▪ 2.5 more
 than SSD
 storage PETABYTES
8. ©2008-16 New Relic, Inc. All rights reserved. 1. NOTHING Lasts Forever 2. Run EXPERIMENTS 3. Experiments DON’T ALWAYS WORK 4. SYNCHRONOUS is going to be a problem 5. Master the ROLLOUT 6. UNDERSTAND your technology 7. NEW TECH = NEW CHALLENGES 8. MAGIC is the enemy 9. WORKLOAD DISTRIBUTION matters 10.Technology enables CULTURE 11.Software architecture: THE BIG PICTURE 11 lessons from our data pipeline
9. ©2008-16 New Relic, Inc. All rights reserved. Collector New Relic App Agents
10. Confidential ©2008-15 New Relic, Inc. All rights reserved. ▪ Double-click to edit
11. ©2008-16 New Relic, Inc. All rights reserved. Collector New Relic App Agents Browser R.U.M.
 Beacon New Relic Browser Agents Mobile Proxy New Relic Mobile Agents
12. ©2008-16 New Relic, Inc. All rights reserved. New Relic App Agents New Relic Browser Agents New Relic Mobile Agents Zzz Collector Browser R.U.M.
 Beacon Mobile Proxy
13. Confidential ©2008-15 New Relic, Inc. All rights reserved.
14. ©2008-16 New Relic, Inc. All rights reserved. NOTHING Lasts Forever Lesson 1
15. Run EXPERIMENTS Lesson 2 ©2008-16 New Relic, Inc. All rights reserved.
16. Confidential ©2008-15 New Relic, Inc. All rights reserved.
17. Confidential ©2008-15 New Relic, Inc. All rights reserved.
18. ©2008-16 New Relic, Inc. All rights reserved. EXPERIMENTS Don’t always work Lesson 3
19. SYNCHRONOUS is going to be a problem Lesson 4 ©2008-16 New Relic, Inc. All rights reserved.
20. ©2008-16 New Relic, Inc. All rights reserved. Kafka Kafka Browser Data Consumer Browser Data Router New Relic App Agents New Relic Browser Agents Collector
21. ©2008-16 New Relic, Inc. All rights reserved. Kafka Kafka Browser Data Consumer New Relic App Agents Collector Zzz Browser Data Router New Relic Browser Agents
22. Confidential ©2008-15 New Relic, Inc. All rights reserved.
23. Master the ROLLOUT Lesson 5 ©2008-16 New Relic, Inc. All rights reserved.
24. ©2008-16 New Relic, Inc. All rights reserved. Rollout
 Techniques Kafka Load Balancer Cluster Kafka Browser Data Consumer Browser Data Router Browser Data
 Consumer V2 Browser R.U.M.
 Beacon %- %+ New Relic Browser Agents Collector Old New
25. Lesson 6 ©2008-16 New Relic, Inc. All rights reserved. UNDERSTAND your technology
26. Confidential ©2008-15 New Relic, Inc. All rights reserved. It’s a TRANSPORT Kafka is not a DATABASE
27. ©2008-16 New Relic, Inc. All rights reserved. Enterprise Service Bus Metric Ingest Edge Raw Ingest Kafka Cluster Local Broker Local Broker Broker Mirror Browser Consumer Kafka Cluster Local Broker Local Broker Broker BAM Consumer Dirac Event Transformer Alerts Kafka Cluster Local Broker Local Broker Broker Alert Consumers Collector Alert Consumers Alert Consumers Vortex Local Broker Local Broker Local Broker Mirror Vortex Vortex Vortex Router Mirror Mirror Clean Ingest Kafka Cluster Local Broker Local Broker Broker Raw Ingest Kafka Cluster Local Broker Local Broker Broker Deduplication Timeslice Kafka Cluster Local Broker Local Broker Broker Mirror Event Kafka Cluster Local Broker Local Broker Broker Mirror Dirac Insert Worker Mirror It’s a TRANSPORT
28. NEW TECH NEW CHALLENGES Lesson 7 ©2008-16 New Relic, Inc. All rights reserved. =
29. Confidential ©2008-15 New Relic, Inc. All rights reserved. Monitoring Tooling Operations Runbooks
30. Confidential ©2008-15 New Relic, Inc. All rights reserved.
31. ©2008-16 New Relic, Inc. All rights reserved. Lesson 8 MAGIC is the enemy
32. ©2008-16 New Relic, Inc. All rights reserved. Brokers “mysteriously” dropping out
33. ©2008-16 New Relic, Inc. All rights reserved. Transparent Huge Pages (/sys/kernel/mm/redhat_transparent_hugepage/enabled) always never
34. Lesson 9 ©2008-16 New Relic, Inc. All rights reserved. Use the right WORKLOAD DISTRIBUTION
35. ©2008-16 New Relic, Inc. All rights reserved. Workload Distribution D D D D D C B B B B A A A A A C B D A B D D D Active Management Random C B B B D D A A D C B B D B C A A D D C C A B
36. ©2008-16 New Relic, Inc. All rights reserved. D D D D D C B B B B A A A A A C B D A B D D D C B B B D D A A D C B B D B C A A D D C C A B Active Management Random Use Both!
37. Technology enables CULTURE Lesson 10 ©2008-16 New Relic, Inc. All rights reserved.
38. Thanks for the pic! https://www.flickr.com/photos/stephenyeargin/7466608166
39. Thanks for the pic! https://www.flickr.com/photos/stephenyeargin/7466608166 �����
40. Software Architecture: THE BIG PICTURE Lesson 11 ©2008-16 New Relic, Inc. All rights reserved.
41. Software Architecture: The Big Picture ▪ ▪ ▪ ▪ ▪ Organic growth Paradigm shift Breaking point
42. ©2008-16 New Relic, Inc. All rights reserved. New Paradigm Collector New Relic App Agents
43. ©2008-16 New Relic, Inc. All rights reserved. Organic Growth Collector New Relic App Agents Browser R.U.M.
 Beacon New Relic Browser Agents Mobile Proxy New Relic Mobile Agents
44. ©2008-16 New Relic, Inc. All rights reserved. Breaking Point Collector Browser R.U.M.
 Beacon Mobile Proxy Zzz New Relic App Agents New Relic Browser Agents New Relic Mobile Agents
45. ©2008-16 New Relic, Inc. All rights reserved. Kafka Kafka Browser Data Consumer Browser Data Router New Relic App Agents New Relic Browser Agents Collector
46. Confidential ©2008-15 New Relic, Inc. All rights reserved. ▪ Double-click to edit
47. Confidential ©2008-15 New Relic, Inc. All rights reserved. ▪ Double-click to edit
49. ©2008-16 New Relic, Inc. All rights reserved. Thanks! ©2008-16 New Relic, Inc. All rights reserved. @nicbenders http://nicbenders.com/presentations/scaling-a-system/