FutureStack 15: New Relic at Scale

B5b0f56c5edce4c54f1fea72e31d8378?s=47 Nic Benders
November 11, 2015

FutureStack 15: New Relic at Scale

In a typical day, New Relic receives half a trillion data points from our customers. All of these need to be processed, stored and made available for charts, queries and alerts as soon as possible. Getting from humble roots to querying more than a billion events a second has meant radically reinventing and re-architecting the whole platform multiple times.

In this session, Matthew Flaming and Nic Benders from New Relic will take you on a deep dive into some of the lessons we've learned along the way and where we go from here. They'll touch on designing scalable services, optimizing for predictability, creating software architecture through team structure, and what it takes to (continuously) rebuild a system that can never stop.

B5b0f56c5edce4c54f1fea72e31d8378?s=128

Nic Benders

November 11, 2015
Tweet

Transcript

  1. Confidential ©2008-15 New Relic, Inc. All rights reserved. Building a

    System that Never Stops: New Relic at Scale 1 Matthew Flaming Nic Benders
  2. Confidential ©2008-15 New Relic, Inc. All rights reserved.

  3. None
  4. EVERY MINUTE requests accepts
 over 16M stores
 over analytic
 events

    2M aggregates
 over 600M metrics 3B queries
 over data
 points ▪
  5. different
 services contains
 over 200 maintained/
 built by 25+ engineering


    teams ▪ 2.5 more
 than SSD
 storage PETABYTES
  6. 1. NOTHING Lasts Forever 2. Run EXPERIMENTS 3. SYNCHRONOUS CALLS

    are going to be a problem 4. Master the ROLLOUT 5. NEW TECH = NEW CHALLENGES 6. Use the right WORKLOAD DISTRIBUTION 7. Technology enables CULTURE 8. Software architecture: THE BIG PICTURE The New Relic lesson plan
  7. Collector New Relic App Agents

  8. Confidential ©2008-15 New Relic, Inc. All rights reserved. ▪ Double-click

    to edit
  9. Collector New Relic App Agents Browser R.U.M.
 Beacon New Relic

    Browser Agents Mobile Proxy New Relic Mobile Agents
  10. New Relic App Agents New Relic Browser Agents New Relic

    Mobile Agents Zzz Collector Browser R.U.M.
 Beacon Mobile Proxy
  11. Confidential ©2008-15 New Relic, Inc. All rights reserved.

  12. NOTHING Lasts Forever Lesson 1

  13. Run EXPERIMENTS Lesson 2

  14. Confidential ©2008-15 New Relic, Inc. All rights reserved.

  15. Confidential ©2008-15 New Relic, Inc. All rights reserved.

  16. EXPERIMENTS Don’t always work Lesson 2(a)

  17. SYNCHRONOUS CALLS are going to be a problem Lesson 3

  18. ▪ Kafka Kafka Browser Data Consumer Browser Data Router New

    Relic App Agents New Relic Browser Agents Collector
  19. ▪ Kafka Kafka Browser Data Consumer New Relic App Agents

    Collector Zzz Browser Data Router New Relic Browser Agents
  20. Confidential ©2008-15 New Relic, Inc. All rights reserved.

  21. Data Routers Kafka Topics Data Consumers Data Stores/DBs Retrieval Services

    Ingest Queue Process/Aggregate Store Query
  22. Data Routers Kafka Topics Data Consumers Data Stores/DBs Retrieval Services

  23. Master the ROLLOUT Lesson 4

  24. Rollout
 Techniques Kafka Load Balancer Cluster Kafka Browser Data Consumer

    Browser Data Router Browser Data
 Consumer V2 Browser R.U.M.
 Beacon %- %+ New Relic Browser Agents Collector Old New
  25. NEW TECH NEW CHALLENGES Lesson 5 =

  26. Monitoring Tooling Operations Runbooks

  27. Confidential ©2008-15 New Relic, Inc. All rights reserved.

  28. Lesson 6 Use the right WORKLOAD DISTRIBUTION

  29. ©2008-15 New Relic, Inc. All rights reserved. ▪ Workload Distribution

    A C B D D D D D B C B B B B D D A A A A A D D Active Management Random C B B B D D A A D C B B D B C A A D D C C A B
  30. Technology enables CULTURE Lesson 7

  31. Thanks for the pic! https://www.flickr.com/photos/stephenyeargin/7466608166

  32. Thanks for the pic! https://www.flickr.com/photos/stephenyeargin/7466608166 �����

  33. Software Architecture: THE BIG PICTURE Lesson 8

  34. Software Architecture: The Big Picture ▪ ▪ ▪ ▪ ▪

    Organic growth Paradigm shift Breaking point
  35. New Paradigm Collector New Relic App Agents

  36. Organic Growth Collector New Relic App Agents Browser R.U.M.
 Beacon

    New Relic Browser Agents Mobile Proxy New Relic Mobile Agents
  37. Breaking Point Collector Browser R.U.M.
 Beacon Mobile Proxy Zzz New

    Relic App Agents New Relic Browser Agents New Relic Mobile Agents
  38. ▪ Kafka Kafka Browser Data Consumer Browser Data Router New

    Relic App Agents New Relic Browser Agents Collector
  39. Confidential ©2008-15 New Relic, Inc. All rights reserved. ▪ Double-click

    to edit
  40. Confidential ©2008-15 New Relic, Inc. All rights reserved. ▪ Double-click

    to edit
  41. None
  42. Confidential ©2008-15 New Relic, Inc. All rights reserved. 43 Thank

    you Matthew Flaming Nic Benders