Setting Up New Relic's Infrastructure

B5b0f56c5edce4c54f1fea72e31d8378?s=47 Nic Benders
October 25, 2013

Setting Up New Relic's Infrastructure

Benefit from our mistakes and hindsight! There's a lot that goes into running a SaaS company - from continuous deployment to DNS to who gets to play with production. We've been running New Relic for over five years now, but if we had to do it over again there are a few things we'd do differently. But there's also a couple things we'd repeat.

B5b0f56c5edce4c54f1fea72e31d8378?s=128

Nic Benders

October 25, 2013
Tweet

Transcript

  1. Setting up New Relic's Infrastructure: A bunch of things we

    did wrong and a few we did right. DAVE PETERSON & NIC BENDERS 2013-10-25
  2. David Peterson (dave@newrelic.com) Nic Benders (nic@newrelic.com)

  3. [“star” graph of code base at launch]

  4. None
  5. 100,000req/sec 2Gbps 200TB

  6. Who Does What?

  7. Site Operations != IT

  8. You are not Batman Do your work during the day

  9. Lowest hour is still 60% of peak hour!

  10. Put your servers on the moon

  11. Never open the box

  12. Domain names are cheap

  13. 10.0.0.0/31 10.0.0.0-10.0.0.1 255.255.255.254 2 10.0.0.0/30 10.0.0.0-10.0.0.3 255.255.255.252 4 10.0.0.0/29 10.0.0.0-10.0.0.7

    255.255.255.248 8 10.0.0.0/28 10.0.0.0-10.0.0.15 255.255.255.240 16 10.0.0.0/27 10.0.0.0-10.0.0.31 255.255.255.224 32 10.0.0.0/26 10.0.0.0-10.0.0.63 255.255.255.192 64 10.0.0.0/25 10.0.0.0-10.0.0.127 255.255.255.128 128 10.0.0.0/24 10.0.0.0-10.0.0.255 255.255.255.0 256 10.0.0.0/23 10.0.0.0-10.0.1.255 255.255.254.0 512 10.0.0.0/22 10.0.0.0-10.0.3.255 255.255.252.0 1024 10.0.0.0/21 10.0.0.0-10.0.7.255 255.255.248.0 2048 10.0.0.0/20 10.0.0.0-10.0.15.255 255.255.240.0 4096 10.0.0.0/19 10.0.0.0-10.0.31.255 255.255.224.0 8192 10.0.0.0/18 10.0.0.0-10.0.63.255 255.255.192.0 16384 10.0.0.0/17 10.0.0.0-10.0.127.255 255.255.128.0 32768 10.0.0.0/16 10.0.0.0-10.0.255.255 255.255.0.0 65536 Private IPs are FREE
  14. Hardware

  15. System Management

  16. X6550 E5-2450 E5-2643 E5-2690 Average response time 137 ms 167

    ms 158 ms 189 ms
  17. 25% performance improvement 2x 6-Core X6550 @ 2.67 GHz 2x

    8-Core E5-2690 @ 2.90 GHz
  18. Storage √

  19. Databases

  20. 32-bits isn't enough class CreateTribbles < ActiveRecord::Migration def change create_table

    :tribbles do |t| t.boolean :hungry t.string :color t.references :parent, index: true end end end
  21. 32-bits isn't enough `id` int(11) NOT NULL AUTO_INCREMENT `id` bigint(20)

    unsigned NOT NULL AUTO_INCREMENT `id` SERIAL
  22. Your database is not a queue

  23. Long running queries will kill you

  24. And now, a thanks to the people who are out

    there every day fixing the mistakes that we made:
  25. Jonathan Owens Jonathan Thurman Aaron Bento Sean Kane Paul Showalter

    Karl Matthias