Scaling a System (that never stops)

B5b0f56c5edce4c54f1fea72e31d8378?s=47 Nic Benders
February 13, 2016
300

Scaling a System (that never stops)

In a typical day, New Relic receives half a trillion data points from our customers. All of these need to be processed, stored and made available for charts, queried and alerted on in near real-time. Going from humble roots to querying more than a billion events a second has meant radically reinventing and re-architecting the whole platform multiple times, and doing it all in public view.

The evolution of New Relic's service illustrates the opportunities and challenges implementing modern technologies. There are lessons on designing scalable services, optimizing for predictability, creating software architecture through team structure, and what it takes to (continuously) rebuild a system that can never stop.

B5b0f56c5edce4c54f1fea72e31d8378?s=128

Nic Benders

February 13, 2016
Tweet

Transcript

  1. Confidential ©2008-15 New Relic, Inc. All rights reserved. ©2008-16 New

    Relic, Inc. All rights reserved. SCALING A SYSTEM (that never stops) Nic Benders - Chief Architect, New Relic
  2. ©2008-16 New Relic, Inc. All rights reserved. @nicbenders http://nicbenders.com/presentations/scaling-a-system/

  3. ©2008-16 New Relic, Inc. All rights reserved. Safe Harbor This

    document and the information herein (including any information that may be incorporated by reference) is provided for informational purposes only and should not be construed as an offer, commitment, promise or obligation on behalf of New Relic, Inc. (“New Relic”) to sell securities or deliver any product, material, code, functionality, or other feature. Any information provided hereby is proprietary to New Relic and may not be replicated or disclosed without New Relic’s express written permission. Such information may contain forward-looking statements within the meaning of federal securities laws. Any statement that is not a historical fact or refers to expectations, projections, future plans, objectives, estimates, goals, or other characterizations of future events is a forward-looking statement. These forward-looking statements can often be identified as such because the context of the statement will include words such as “believes,” “anticipates,” “expects” or words of similar import. Actual results may differ materially from those expressed in these forward-looking statements, which speak only as of the date hereof, and are subject to change at any time without notice. Existing and prospective investors, customers and other third parties transacting business with New Relic are cautioned not to place undue reliance on this forward-looking information. The achievement or success of the matters covered by such forward-looking statements are based on New Relic’s current assumptions, expectations, and beliefs and are subject to substantial risks, uncertainties, assumptions, and changes in circumstances that may cause the actual results, performance, or achievements to differ materially from those expressed or implied in any forward-looking statement. Further information on factors that could affect such forward-looking statements is included in the filings we make with the SEC from time to time. Copies of these documents may be obtained by visiting New Relic’s Investor Relations website at ir.newrelic.com or the SEC’s website at www.sec.gov. New Relic assumes no obligation and does not intend to update these forward-looking statements, except as required by law. New Relic makes no warranties, expressed or implied, in this document or otherwise, with respect to the information provided
  4. None
  5. EVERY MINUTE requests accepts
 over 16M stores
 over analytic
 events

    2M aggregates
 over 600M metrics 3B queries
 over data
 points ▪
  6. different
 services contains
 over 200 maintained/
 built by 25+ engineering


    teams ▪ 2.5 more
 than SSD
 storage PETABYTES
  7. ©2008-16 New Relic, Inc. All rights reserved. 1. NOTHING Lasts

    Forever 2. Run EXPERIMENTS 3. Experiments DON’T ALWAYS WORK 4. SYNCHRONOUS is going to be a problem 5. Master the ROLLOUT 6. UNDERSTAND your technology 7. NEW TECH = NEW CHALLENGES 8. MAGIC is the enemy 9. WORKLOAD DISTRIBUTION matters 10.Technology enables CULTURE 11.Software architecture: THE BIG PICTURE 11 lessons from our data pipeline
  8. ©2008-16 New Relic, Inc. All rights reserved. Collector New Relic

    App Agents
  9. Confidential ©2008-15 New Relic, Inc. All rights reserved. ▪ Double-click

    to edit
  10. ©2008-16 New Relic, Inc. All rights reserved. Collector New Relic

    App Agents Browser R.U.M.
 Beacon New Relic Browser Agents Mobile Proxy New Relic Mobile Agents
  11. ©2008-16 New Relic, Inc. All rights reserved. New Relic App

    Agents New Relic Browser Agents New Relic Mobile Agents Zzz Collector Browser R.U.M.
 Beacon Mobile Proxy
  12. Confidential ©2008-15 New Relic, Inc. All rights reserved.

  13. ©2008-16 New Relic, Inc. All rights reserved. NOTHING Lasts Forever

    Lesson 1
  14. Run EXPERIMENTS Lesson 2 ©2008-16 New Relic, Inc. All rights

    reserved.
  15. Confidential ©2008-15 New Relic, Inc. All rights reserved.

  16. Confidential ©2008-15 New Relic, Inc. All rights reserved.

  17. ©2008-16 New Relic, Inc. All rights reserved. EXPERIMENTS Don’t always

    work Lesson 3
  18. SYNCHRONOUS is going to be a problem Lesson 4 ©2008-16

    New Relic, Inc. All rights reserved.
  19. ©2008-16 New Relic, Inc. All rights reserved. Kafka Kafka Browser

    Data Consumer Browser Data Router New Relic App Agents New Relic Browser Agents Collector
  20. ©2008-16 New Relic, Inc. All rights reserved. Kafka Kafka Browser

    Data Consumer New Relic App Agents Collector Zzz Browser Data Router New Relic Browser Agents
  21. Confidential ©2008-15 New Relic, Inc. All rights reserved.

  22. Master the ROLLOUT Lesson 5 ©2008-16 New Relic, Inc. All

    rights reserved.
  23. ©2008-16 New Relic, Inc. All rights reserved. Rollout
 Techniques Kafka

    Load Balancer Cluster Kafka Browser Data Consumer Browser Data Router Browser Data
 Consumer V2 Browser R.U.M.
 Beacon %- %+ New Relic Browser Agents Collector Old New
  24. Lesson 6 ©2008-16 New Relic, Inc. All rights reserved. UNDERSTAND

    your technology
  25. Confidential ©2008-15 New Relic, Inc. All rights reserved. It’s a

    TRANSPORT Kafka is not a DATABASE
  26. ©2008-16 New Relic, Inc. All rights reserved. Enterprise Service Bus

    Metric Ingest Edge Raw Ingest Kafka Cluster Local Broker Local Broker Broker Mirror Browser Consumer Kafka Cluster Local Broker Local Broker Broker BAM Consumer Dirac Event Transformer Alerts Kafka Cluster Local Broker Local Broker Broker Alert Consumers Collector Alert Consumers Alert Consumers Vortex Local Broker Local Broker Local Broker Mirror Vortex Vortex Vortex Router Mirror Mirror Clean Ingest Kafka Cluster Local Broker Local Broker Broker Raw Ingest Kafka Cluster Local Broker Local Broker Broker Deduplication Timeslice Kafka Cluster Local Broker Local Broker Broker Mirror Event Kafka Cluster Local Broker Local Broker Broker Mirror Dirac Insert Worker Mirror It’s a TRANSPORT
  27. NEW TECH NEW CHALLENGES Lesson 7 ©2008-16 New Relic, Inc.

    All rights reserved. =
  28. Confidential ©2008-15 New Relic, Inc. All rights reserved. Monitoring Tooling

    Operations Runbooks
  29. Confidential ©2008-15 New Relic, Inc. All rights reserved.

  30. ©2008-16 New Relic, Inc. All rights reserved. Lesson 8 MAGIC

    is the enemy
  31. ©2008-16 New Relic, Inc. All rights reserved. Brokers “mysteriously” dropping

    out
  32. ©2008-16 New Relic, Inc. All rights reserved. Transparent Huge Pages

    (/sys/kernel/mm/redhat_transparent_hugepage/enabled) always never
  33. Lesson 9 ©2008-16 New Relic, Inc. All rights reserved. Use

    the right WORKLOAD DISTRIBUTION
  34. ©2008-16 New Relic, Inc. All rights reserved. Workload Distribution D

    D D D D C B B B B A A A A A C B D A B D D D Active Management Random C B B B D D A A D C B B D B C A A D D C C A B
  35. ©2008-16 New Relic, Inc. All rights reserved. D D D

    D D C B B B B A A A A A C B D A B D D D C B B B D D A A D C B B D B C A A D D C C A B Active Management Random Use Both!
  36. Technology enables CULTURE Lesson 10 ©2008-16 New Relic, Inc. All

    rights reserved.
  37. Thanks for the pic! https://www.flickr.com/photos/stephenyeargin/7466608166

  38. Thanks for the pic! https://www.flickr.com/photos/stephenyeargin/7466608166 �����

  39. Software Architecture: THE BIG PICTURE Lesson 11 ©2008-16 New Relic,

    Inc. All rights reserved.
  40. Software Architecture: The Big Picture ▪ ▪ ▪ ▪ ▪

    Organic growth Paradigm shift Breaking point
  41. ©2008-16 New Relic, Inc. All rights reserved. New Paradigm Collector

    New Relic App Agents
  42. ©2008-16 New Relic, Inc. All rights reserved. Organic Growth Collector

    New Relic App Agents Browser R.U.M.
 Beacon New Relic Browser Agents Mobile Proxy New Relic Mobile Agents
  43. ©2008-16 New Relic, Inc. All rights reserved. Breaking Point Collector

    Browser R.U.M.
 Beacon Mobile Proxy Zzz New Relic App Agents New Relic Browser Agents New Relic Mobile Agents
  44. ©2008-16 New Relic, Inc. All rights reserved. Kafka Kafka Browser

    Data Consumer Browser Data Router New Relic App Agents New Relic Browser Agents Collector
  45. Confidential ©2008-15 New Relic, Inc. All rights reserved. ▪ Double-click

    to edit
  46. Confidential ©2008-15 New Relic, Inc. All rights reserved. ▪ Double-click

    to edit
  47. None
  48. ©2008-16 New Relic, Inc. All rights reserved. Thanks! ©2008-16 New

    Relic, Inc. All rights reserved. @nicbenders http://nicbenders.com/presentations/scaling-a-system/